23

Suppose there is a test for Disease A that is correct 90% of the time. You had this test done, and it came out positive. I understand that the chance that this test is right is 90%, but I thought this would mean the chance that you have a disease should be 90% too. However, according to Bayes' rule, your chance of disease depends on the percentage of the population that has this disease. It sounds absurd: If the test is correct then you have it, and if it's not then you don't, 90% of the time- so there should be 90% chance that the results are right for you...

But on other hand, say 100% of population has it. Then regardless of the chance the test says you have it, let it be 90% or 30%, your chance is still 100%... now all of a sudden it doesn't sound absurd.

Please avoid using weird symbols as I'm not statistics expert. It just deludes things for me and buries the insight.

Asker
  • 1,327
Muhammad Umer
  • 685
  • 1
    I wrote an article on this problem contrasting several different tests: the test where you always say "negative", the test where you flip a coin, and a test that is 99% accurate. You might find it helpful in trying to understand why it is that a 99% accurate test is 50% accurate when the result is positive and 99.99% accurate when the result is negative. http://blogs.msdn.com/b/ericlippert/archive/2010/07/01/murky-research.aspx – Eric Lippert Jun 02 '15 at 14:30
  • 3
    This is known as the "base rate fallacy". The Wikipedia article on the topic is enlightening. – Stephan Kolassa Jun 02 '15 at 15:29
  • 4
    I have a test for any disease that is right 50% of the time. The test is: I flip a coin. Suddenly you have a 50% chance of having every disease known or unknown to man. –  Jun 02 '15 at 19:54
  • Back in the real world, most medical tests don't have a simple "right 90% of the time" state: initial screening tests are chosen to have a very low false negative rate, for example, while they may have a higher false positive rate. This preserves resources, as the more expensive and precise tests are used only on those who were found positive in the initial screening. Furthermore, you don't care about the incidence of the condition in the population at large: you care about the incidence in the population being tested. – TRiG Aug 28 '15 at 12:07
  • Related: https://math.stackexchange.com/questions/32933/describing-bayesian-probability – Henry Oct 08 '20 at 09:36

12 Answers12

37

Look at it this way. There is a $10$% chance that any given instance of the test is wrong. It can be wrong in either of two ways: it can be positive when you don’t have the disease (a false positive), or it can be negative when you do have the disease (a false negative).

  • If the disease is very common, most of the people being tested will have the disease, so most of the errors will be made on those people and will therefore be false negatives. In that case few of the errors will be false positives, so if you test positive, you probably have the disease; a negative test, on the other hand, could easily be one of the false negatives. You already pointed out the extreme case of this, in which the entire population has the disease, and every error is a false negative.

  • If the disease is very rare, however, most of the people being tested will not have the disease, and most of the errors will therefore be false positives; if you test negative, you probably don’t have the disease, but a positive test could easily be one of the false positives. The extreme case would be when no one has the disease, and every error is a false positive.

As the incidence of the disease shifts from $100$% to $0$%, the probability that an error is a false positive increases from $0$ to $1$.

Brian M. Scott
  • 616,228
13

Your intuition is relying on the test always being accurate. However, this is not the case. There are four conditions that we have to account for:

  1. The test is positive and you have the disease;
  2. The test is positive and you don't have the disease;
  3. The test is negative and you have the disease;
  4. The test is negative and you don't have the disease.

If the test were perfect, only results #1 and #4 would happen. But this is not the case in the real world. As a consequence, we have to condition ourselves on the probability that the test makes a mistake--or, looking at it another way, we have to evaluate the results on the test based on the as-yet unknown reality as to whether we have the disease or not. A test has a given accuracy independent of whether the disease is present or not.

Put another way, suppose you notice a cookie is missing, and you ask a child. That child will either tell the truth, or not. Regardless of what the child tells you, there are two possible truths: the cookie is gone, or you miscounted your number of cookies.

Emily
  • 35,688
  • 6
  • 93
  • 141
  • why person tested positive isn't 90% likely to have this disease. if test is 90% times correct. – Muhammad Umer Jun 02 '15 at 03:34
  • 10
    @MuhammadUmer Suppose there are two tests for the disease. One test was designed by a Nobel Laureate. One test was designed by someone who failed out of college. The first test is 90% accurate. The second test is 60% accurate. Both tests come up positive. Does the person have a 90% likelihood of having the disease, or a 60% likelihood? – Emily Jun 02 '15 at 03:36
  • valid point, but then what is the likelihood. Say some test is 100% accurate.. and some test is 50% and both come positive, won't we say known likelihood is dependent on highest one. – Muhammad Umer Jun 02 '15 at 03:41
  • 3
    @MuhammadUmer The likelihood depends on the accuracy of the test. Instead of asking "what is the likelihood that this person has a disease," we ask, "given that a test was positive, what is the likelihood that this person has a disease?" Asking the latter question gives us a more accurate, but not entirely accurate picture. The conditional probability answers the question: "what is the likelihood that a person both has a disease and tests positive for the disease," which is the question that we actually want answered. – Emily Jun 02 '15 at 03:45
  • 4
    @anonymous user suggesting an edit: while replacing "and" with "but" in bullet points #2 and #3 would be more grammatically correct, I deliberately chose to use "and" because of the mathematical connotation it carries when linking intuition to Bayes Theorem. Remember, we have things like "probability of A and Not B", etc. in the statement of the theorem. We don't typically use "but". – Emily Jun 02 '15 at 13:56
  • so 4 things: 1) what is probability person has a disease? 2) what is probability test is accurate? 3) if test is positive what is probability person has disease? 4) what is the probability that person has disease and tests positive? ....I understand Q1 depends on population ratio of disease. Q2 depends on test's accuracy. However, Q3 is *unclear. Q4*, makes sense that what are the chances for population that has disease will be detected by this test. – Muhammad Umer Jun 02 '15 at 16:50
  • how q3 and q4 are different, and how would someone measure q3. My thoughts are that q3 is asking for chance of disease given person came positive. q4 is asking for chance of person being positive and test being accurate. – Muhammad Umer Jun 02 '15 at 17:02
  • 1
    @MuhammadUmer As you have phrased them, Q3 and Q4 are the same. But your four questions are not my four questions. You are focusing on the question "what is the probability the test is accurate." But this question encompasses two possible outcomes: 1.) the test is negative and the person has no disease; 2.) the test is positive and the person has the disease. We have to explore both of these conditions. – Emily Jun 02 '15 at 17:27
11

Part of the difficulty in understanding this is that we automatically involve other aspects of the imagined scenario in our thinking. The Bayesian approach takes a simplistic view. Here's an example:

In a population of 1000 people, suppose 10 have a disease (ignore how we know that 10 people do....) Suppose we have a test which is correct 90% of the time it claims the disease is present, and also correct 90% of the time it claims the disease is absent. (These two numbers need not be the same, but let's suppose they are.)

Story #1: A new health initiative leads to the entire population being tested. Out of the 10 people with the disease, 9 are correctly identified by the test, while 1 is missed. Out of the 990 people without the disease, 891 (90%) are cleared as healthy, while 99 (10%) are mistakenly labeled as diseased.

Out of the 99 + 9 = 108 people who were tagged as having the disease, only 9 really do. (about 8%). So if we take one of the positive test results at random, that person who tested positive has only an 8% chance of having the disease.

Because the number of healthy people WHO WERE TESTED is so high, more false positives than true positives occurred.

Story #2: The test is expensive and rarely done. Only people who have symptoms suggesting the disease bother to have the test done. So out of our population of 1000, only 30 have the test done, including all 10 who really have the disease. Now, 9 of the 10 people with the disease get positive test results, and 2 of the 20 people WHO WERE TESTED but don't have the disease get positive test results. The chance of having the disease, given that you WERE TESTED and tested positive for the disease, is 9 / 11, or 82%.

The Bayesian analysis describes Story #1 above, but real life is usually more like Story #2. That contributes to the result seeming so counter to intuition.

To more directly address the OP's question: you had the test, and it came out positive. Your question is now "was that positive test result one of the correct positive results, or one of the false positives?" In a situation like Story #1 there are many more false positives than true positives, so you are likely to have gotten a false positive result.

Mike Kantor
  • 231
  • It might also be interesting to think about what happens if, in Story #2, some portion of the initially tested population is then retested because they continue to display symptoms or seek help from a different medical group. (This also happens in the real world.) – jpmc26 Jun 02 '15 at 23:30
  • 1
    The Bayesian approach is only simplistic when you apply it simplistically. In Story#2, to correctly apply the Bayesian approach you should also plug in the numbers of those "WHO WERE TESTED" who have the disease (again, there is the same issue you mentioned about how you might know that!). That will then give the correct answer. – psmears Jun 03 '15 at 10:40
  • Psmears is entirely right. The point in the Bayesian approach is that it's consistent - in Story#2, you're doing two tests - the first is the symptoms, the second is the confirmation test. Unlike frequentist statistics, bayesians don't ignore these - they're part of the final solution. Bayesian statistics work even better for more complicated scenarios like #2. That doesn't mean #1 scenarios aren't real, though! The greatest example being female breast cancer screening. And that's exactly where people misunderstand results of the test the most (exactly because it's "counterintuitive"). – Luaan Jun 03 '15 at 12:46
4

Consider a disease that only affects 2% of the population (and there are tests for rarer diseases than that). If you have a 10% chance of being assessed as having the disease even if you don't, then even if the test always identifies those affected by the disease, the number of positive results will be five times more than the number of people with the disease.

So in that case the chance of having the disease, given a positive test result (and no other information) would stand at about 17%, not 90%.

Joffan
  • 39,627
3

Consider the following table which shows the probabilities all the possibilities of presence of disease and the result of the test. One can see that the proportion of people in the population with the disease, $x$, will affect the final outcomes.

$$ \begin{array}{c|c|c|} & \text{Has disease} & \text{No disease} \\ \hline \text{Test positive} & 0.9 \times x & 0.1 \times (1-x) \\ \hline \text{Test negative} & 0.1 \times x & 0.9 \times (1-x) \end{array} $$

Indeed, if $x$ is equal to the failure rate of the test, then a positive result will only have a 50% chance of being correct!

James
  • 328
3

Good question, and lots of good answers here. I recommend Steven Strogatz's New York Times column on this very point.

http://opinionator.blogs.nytimes.com/2010/04/25/chances-are/

He reports on doctors faced with this problem:

The probability that one of these women has breast cancer is 0.8 percent. If a woman has breast cancer, the probability is 90 percent that she will have a positive mammogram. If a woman does not have breast cancer, the probability is 7 percent that she will still have a positive mammogram. Imagine a woman who has a positive mammogram. What is the probability that she actually has breast cancer?

Later on:

When Gigerenzer asked 24 other German doctors the same question, their estimates whipsawed from 1 percent to 90 percent. Eight of them thought the chances were 10 percent or less, 8 more said 90 percent, and the remaining 8 guessed somewhere between 50 and 80 percent. Imagine how upsetting it would be as a patient to hear such divergent opinions.

As for the American doctors, 95 out of 100 estimated the woman’s probability of having breast cancer to be somewhere around 75 percent.

The right answer is 9 percent.

Then Strogatz shows you why the problem confuses people, and how to think about it correctly. No formulas.

Ethan Bolker
  • 95,224
  • 7
  • 108
  • 199
3

Suppose there is a test for Disease A that is correct 90% of the time. You had this test done, and it came out positive. I understand that the chance that this test is right is 90%, but I thought this would mean the chance that you have a disease should be 90% too.

Let's change that 90% to 50%:

I have developed a software that analyzes your eye movements and detects if you have brain cancer. Because it's actually a difficult thing to do and I was lazy, I programmed the software just to flip a coin and tell YES or NO, with 50% chance each.

So, we have a test for Disease A that is correct 50% of the time. I had this test done, and it came out positive. Is the chance that the test was right 50%?

JiK
  • 5,815
  • 19
  • 39
2

Put in real values and it's easier. Test is 90% accurate for both false positives and false negatives so if you don't have the disease it will falsely say you do 10% of the time. If you do have he disease it will falsely say you don't 10% of the time.

There is a population of 1000 and 1% actually have the disease. So 10 people out of 1000. Apply the test to the whole population and you get 100 false positives and 1 false negative .For any individual then who tests positive what's the likelihood the test is wrong? The odds are 10:1 it's wrong, because 100 people will get positive results but only 10 will actually have the disease. So even though the test is 90% accurate, there's only a 1 in 10 chance that a positive result means you really have the disease.

Change the percentage of the population that actually has the disease to say 50% and you'll see this changes the situation dramatically.

Mark Hoheisel
  • 21
2

A visual for the issue at hand - say the population has 10% sick people and a test that's 90% accurate. Of the 10 sick people in 100, it identifies 9 (the red X). But of the 90% who are not sick 10% are bad results identifying as sick (the blue X). A "sick" result, therefore, has a 50/50 chance of correctly identifying a sick person or falsely identifying a well person.

enter image description here

The other answers are good ones. I just prefer the visual approach to clarify.

JTP - Apologise to Monica
  • 3,102
0

Just to add to the previous answers:

Think of it this way: If, say, 60% of the tested population is known to have the disease, is the test really detecting a disease, or is it just a coincidence that 90% of those who tested positive actually have the disease?

Let's say 60% of a population has influenza. 90% of those who have the common cold also have influenza. We develop a test, thinking it will help us identify those who have influenza. This test returns a positive result whenever we test someone who has a cold and influenza. So, how do we know how accurate the test is? We can include in our test a) those who have only influenza and b) those who are known to be perfectly healthy, and c) those who have only a cold. We can then uses Bayes's theorem to figure out the probability that the test has correctly identified or ruled out influenza.

moonman239
  • 553
0

There is a very good tutorial on Bayes Theorem at http://www.yudkowsky.net/rational/bayes/

As others have mentioned, the issue is that you must consider the base rate (the occurence of the disease). The link above deals with this as the very first example.

TaoOfGaming
  • 11
0

Let's say 1/1000 people have a disease and we test 1000 people with a 10% fail rate on a test.

enter image description here

In this instance (1000 people) I haven't shown any false negatives. If we were showing 10,000 people we could include one.

Derek Tomes
  • 117
  • Down votes without comments; how constructive! – Derek Tomes Jun 03 '15 at 01:56
  • I'm upvoting in compensation because pictures are often helpful, and I don't think a downvote is deserved. However, I point out that this picture highlights false positives at the expense of false negatives (indeed, it's not clear from the picture whether the one actual true diseased person tested positive or not). – Brian Tung Jun 03 '15 at 02:03
  • 1
    @BrianTung: But in the example false negatives are so rare (the question is not clear, but one may suppose it is a small fraction of those actually having the disease, so maybe $1/10,000$ of the whole population) that they have a very small effect on the conditional probability of having the disease if testing positive (which the question is about); they do determine the (very small) probability of having the disease while testing negative. – Marc van Leeuwen Jun 03 '15 at 06:58
  • 1
    "I'm not allowed to not post text or I'd let the picture speak for itself." Is necessary and sufficient cause for downvotes. If the question were simply "does anyone have a graphic related to Baye's theorem and disease testing?" this would be adequate. But that is not the question here. And there's not exactly 1000 people in that image, either, so your lead-in is misleading and incorrect. It's 30x40 at a quick inspection, which is 1200. – zibadawa timmy Jun 03 '15 at 07:55
  • Thanks zibadawa -- down votes with comments are constructive (I've corrected the image) -- down votes without comments are just pointless. – Derek Tomes Jun 03 '15 at 20:46