Skip to main content

"95% Confidence interval" does not imply "a 95% probability that the mean is contained in the CI"

I was damned confused! I think I understand it now, so I'd like to share how to resolve this "paradox".

First, to recap, suppose you have a process / experiment that produces a confidence interval [x, y], for which you say, if we repeat this experiment a lot of time, 95% of the cases we'll see that the [x, y] covers something that we are estimating, e.g. the true mean. We call this 95% Confidence Interval (CI) for the true mean.

But then, your instructor, or the author of the book you are reading, will immediately follow up by the statement "Be careful! It's wrong to say that the CI you computed has 95% chance of containing the true mean".

Confusion ensues. You lose sleep, you don't eat as much. Your parents, friends, and loved ones start to get worried because you don't look so well. You, on the other hand, has lost all confidence in your understanding of statistics and basics of probabilities.

WHY on earth?? If 95% of the CIs produced by the repeated experiments contain the true mean, then wouldn't it make perfect sense to say that the probability of the mean being contained in the CI is 0.95??

And you read about various articles, and thought you got an "aha!" moment, only to discover that the insight you thought you had vaporizes by the time you read the next articles.

Yeah, I know, I know. That all happened to me too. Good news is, I think I've read enough to come up with a simple way to finally put this to rest.

First, let me write this down: although 95% of all CIs produced by our experiments contain the parameter p, it does not mean that the probability of a specific CI to contain p is 95%.

Here's the illustration to bring this point home!

Let's say we have a uniform distribution U of 3 integers [p-1, p, p+1], where p is the mean of the distribution. Now, suppose that we don't know what the actual value of p is, but we somehow know that p is either 2 or 3, with equal probability. We want to come up with a procedure to compute a confidence interval which hopefully can help us estimate p. Here's how.

We sample 2 values from U, call that x and y. If e.g. p = 2, then all possible (x,y) are (1,1), (1,2), (1,3), (2,1), (2,2), (2,3), (3,1), (3,2), (3,3), each with equal probability 1/9. From that, construct a confidence interval defined as [min(x,y), max(x,y)]. Then you and I know that 7 out of 9 of the CIs constructed contains p. We can say that our procedure produces a 77.8% CI for p.

Now, is it true that a CI has 77.8% chance of containing p?

Let's take the CI (1,1). We are pretty darn sure that it does not contain any possible values of p. Hence P(p in CI (1,1)) = 0, not 77.8%!

Well, that's not fair. Let's try another. Is the probability of CI (2,3) to contain p is 77.8%? Nope, since p is either 2 or 3, that CI actually is guaranteed to contain p. So P(p in CI (2,3)) = 1, not 77.8%!

Last example, take CI (2,2). What is the probability of CI (2, 2) to contain p? That's P(p in CI (2,2)) = P(p in CI (2,2) | p = 2) * P(p = 2) + P(p in CI (2,2) | p = 3) * P(p = 3) = 1 * 1/2 + 0 * 1/2 = 1/2. Again, not 77.8%!

Hence, it's crystal clear in this example, that although 77.8% of the time our procedure produces a CI that contains p, the probability of a specific CI to contain p is not 77.8%!

Now, you ask me, since we 95% of all CIs contain the true mean, what's the probability of a specific CI to contain the true mean? Then my answer is, it's either 0 or 1, since the CI either contains the true mean or it doesn't.

Confused much? 😁

Comments

Post a Comment

Popular posts from this blog

641 | 2^32 + 1

Problem: Show that \( 641 | 2^{32} + 1 \). Solution: (From Problem Solving Strategies, Arthur Engel) \( 641 = 625 + 16 = 5^4 + 2^4 \). So \( 641 | 2^{32} + 2^{28} \cdot 5^4 \). Also, \( 641 = 640 + 1 = 2^7 \cdot 5 + 1\). So \( 641 | (2^7 \cdot 5)^4 - 1 = 2^{28}\cdot 5^4 - 1 \). Hence \( 641 | 2^{32} + 2^{28} \cdot 5^4 -(2^{28}\cdot 5^4 - 1) \). QED
Post a Comment
Read more

Derive e

Prove that \( \left( 1 + \frac{1}{n} \right)^n \) converges. Solution Let \( G(n) =  \left( 1 + \frac{1}{n} \right)^n \). Binomial expansion, \( G(n) = \sum_{k = 0}^{n} \frac{ \binom{n}{k} }{n^k} \). By simple inequality argument, \( \frac{ \binom{n}{k} }{n^k} < \frac{1}{k!} \). So if we let \( F(n) = \sum_{k = 0}^{n} \frac{1}{k!} \), we have \( G(n) < F(n) \). By simple inequality argument, \( F(n) < 1 + \sum_{k = 1}^{n} \frac{1}{2^{k-1}} < 3 \). So G and F are bounded above. By AM-GM, \( \frac{G(n-1)}{G(n)} = \left( \frac{ n^2 }{ n^2 - 1} \right)^{(n-1)} \frac{ n}{n+1} < \left(  \frac{ \frac{ n^2 }{ n^2 - 1} (n-1) + \frac{ n}{n+1} }{n} \right)^n = 1 \). So \( G(n-1) < G(n) \). Since G is monotonic increasing and bounded above, G converges. Similarly F converges. Let's also prove that G and F converges to the same limit. By studying individual terms in \( F(n) - G(n) \) and making simple inequality arguments, we can show that for each \( \epsilon > 0 \) w...
Post a Comment
Read more

Random walk by the cliff

You are one step away from falling down a cliff. In each step, you have 1/3 chance of moving towards the cliff, 2/3 chance of moving away from the cliff. What is the probability you fall down the cliff? Solution It is 1/2, and there are 3 ways to approach this. Let's say we are at position 0, and the cliff is at position -1. We go +1 with probability p, and go -1 with probability 1-p. Approach 1. Catalan number: All paths that end with falling down the cliff can be broken down to two parts: 0 -> ... -> 0 and 0 -> -1. The path 0 -> ... -> 0 is Catalan; if we fix the path length to 2n, then the number of paths would be C_n. What we want to compute is the total probability = (1-p) sum { C_n (p(1-p))^n }. We can use the generating function for Catalan to arrive at: if p <= 1/2, probability = 1; if p > 1/2, probability = 1/p - 1. Approach 2. Let P(i) be the probability of reaching -1 from position i.  P(0) = 1-p + pP(1) Also, P(1) can be derived as follows: can we r...
Post a Comment
Read more