"95% Confidence interval" does not imply "a 95% probability that the mean is contained in the CI"

I was damned confused! I think I understand it now, so I'd like to share how to resolve this "paradox".

First, to recap, suppose you have a process / experiment that produces a confidence interval [x, y], for which you say, if we repeat this experiment a lot of time, 95% of the cases we'll see that the [x, y] covers something that we are estimating, e.g. the true mean. We call this 95% Confidence Interval (CI) for the true mean.

But then, your instructor, or the author of the book you are reading, will immediately follow up by the statement "Be careful! It's wrong to say that the CI you computed has 95% chance of containing the true mean".

Confusion ensues. You lose sleep, you don't eat as much. Your parents, friends, and loved ones start to get worried because you don't look so well. You, on the other hand, has lost all confidence in your understanding of statistics and basics of probabilities.

WHY on earth?? If 95% of the CIs produced by the repeated experiments contain the true mean, then wouldn't it make perfect sense to say that the probability of the mean being contained in the CI is 0.95??

And you read about various articles, and thought you got an "aha!" moment, only to discover that the insight you thought you had vaporizes by the time you read the next articles.

Yeah, I know, I know. That all happened to me too. Good news is, I think I've read enough to come up with a simple way to finally put this to rest.

First, let me write this down: although 95% of all CIs produced by our experiments contain the parameter p, it does not mean that the probability of a specific CI to contain p is 95%.

Here's the illustration to bring this point home!

Let's say we have a uniform distribution U of 3 integers [p-1, p, p+1], where p is the mean of the distribution. Now, suppose that we don't know what the actual value of p is, but we somehow know that p is either 2 or 3, with equal probability. We want to come up with a procedure to compute a confidence interval which hopefully can help us estimate p. Here's how.

We sample 2 values from U, call that x and y. If e.g. p = 2, then all possible (x,y) are (1,1), (1,2), (1,3), (2,1), (2,2), (2,3), (3,1), (3,2), (3,3), each with equal probability 1/9. From that, construct a confidence interval defined as [min(x,y), max(x,y)]. Then you and I know that 7 out of 9 of the CIs constructed contains p. We can say that our procedure produces a 77.8% CI for p.

Now, is it true that a CI has 77.8% chance of containing p?

Let's take the CI (1,1). We are pretty darn sure that it does not contain any possible values of p. Hence P(p in CI (1,1)) = 0, not 77.8%!

Well, that's not fair. Let's try another. Is the probability of CI (2,3) to contain p is 77.8%? Nope, since p is either 2 or 3, that CI actually is guaranteed to contain p. So P(p in CI (2,3)) = 1, not 77.8%!

Last example, take CI (2,2). What is the probability of CI (2, 2) to contain p? That's P(p in CI (2,2)) = P(p in CI (2,2) | p = 2) * P(p = 2) + P(p in CI (2,2) | p = 3) * P(p = 3) = 1 * 1/2 + 0 * 1/2 = 1/2. Again, not 77.8%!

Hence, it's crystal clear in this example, that although 77.8% of the time our procedure produces a CI that contains p, the probability of a specific CI to contain p is not 77.8%!

Now, you ask me, since we 95% of all CIs contain the true mean, what's the probability of a specific CI to contain the true mean? Then my answer is, it's either 0 or 1, since the CI either contains the true mean or it doesn't.

Confused much? 😁