COMMON MISTEAKS MISTAKES IN USING STATISTICS: Spotting and Avoiding Them

Introduction        Types of Mistakes        Suggestions        Resources        Table of Contents         About


Overview of Frequentist Confidence Intervals

The General Situation:
The first goal is usually easier than the second.

Example: If the parameter we are interested in estimating is the mean of the random variable, then we can estimate it using a sample mean.1

Important note on terminology:
The idea for the second goal: Although we typically have just one sample at hand when we do statistics, the reasoning used in frequentist (classical) inference depends on thinking about all possible suitable samples of the same size n. Which samples are considered "suitable" will depend on the particular statistical procedure to be used. Each statistical procedure has model assumptions that are needed to ensure that the reasoning behind the procedure is sound. The model assumptions determine which samples are "suitable." (Cf. Overview of Frequentist Hypothesis Testing)

Example: The parameter we are interested in estimating is the population mean µ = E(Y) of the random variable Y.
(*)    The probability that Ȳn lies between µ - a and µ + a is 0.95

Caution: It is important to get the reference category straight here. This amounts to keeping in mind what is a random variable and what is a constant. Here, 
Ȳn is the random variable (that is, the sample is varying), whereas  µ and a are constant.3
(**)   The probability that µ  lies between Ȳn - a and Ȳn + a is 0.95

Caution: It is again important to get the reference category correct here. It hasn't changed: it is still the sample that is varying, not
µ.  So the probability refers to Ȳn,  not to µ. Thinking that the probability refers to µ is a common mistake in interpreting confidence intervals. It may help to restate (**) as

(***)    The probability that the interval from
 Ȳn - a to Ȳn + a contains µ  is .95.

A demo such as those posted by bioconsultingR. Webster or W. H. Freeman can help reinforce the correct interpretation.4 The Rice Virtual Lab in Statistics' Confidence Interval Simulation shows both 95% and 99% confidence intervals.
1) The sample we have taken is one of the 95% for which the interval from  Ȳn - a to Ȳn + a contains µ.
2) Our sample is one of the  5% for which 
the interval from  Ȳn - a to Ȳn + a does not contain µ.
Unfortunately, we can't know which of these two possibilities is true.

In general:
We can follow a similar procedure for many other situations to obtain confidence intervals for parameters.

Notes:
1. The sample will need to be a simple random sample in order for the sample mean to be a reasonable estimate of the population mean; see also biased sampling.

2. The inference procedure behind this example is known as the "large-sample z-procedure for the mean." It is only an approximate procedure, but it is quite good for large sample sizes. Since it is simpler than the t-procedure that is an "exact" procedure for inference for a mean, it is a good example for illustrating the basic idea of sampling distribution, especially if we include a model assumption that Y is normally distributed. This, plus the other model assumptions, will imply that
Ȳn also has a normal distribution. The only problem is then that we don't know the standard deviation of the sampling distribution -- but for n large enough, the sample standard deviation is very close, close enough to get a good approximation to a.

3. Stating (*) as above may help keep the reference category straight. Restating as

"The probability that µ - a <  Ȳn µ + a is 0.95"
may prompt the common mistake of thinking that µ is the variable. This highlights an important difference between frequentist and Bayesian statistics: In frequentist statistics, parameters are assumed to be constant but unknown, whereas in Bayesian inference for parameters, the parameter may be assumed to be a random variable, with different values corresponding to different "states of nature".

4. Still another way to rephrase this: Define two new random variables L
nȲn - a and  Rn = Ȳn + a. Then the probability that a simple random sample for Y will have the property that the interval from  Ln to Rn contains µ is .95.