This
site is under construction. Please check back every few weeks for
updates
COMMON MISTEAKS
MISTAKES IN
USING STATISTICS: Spotting and Avoiding Them
Over-Interpretation
Authors and readers alike often succumb to the temptation to
over-interpret the results of statistical studies. Some common
mistakes
of this form include:
Extrapolation to a larger population than the one studied
Examples:
- The classic example: running a psychology experiment
with undergraduates enrolled in psychology classes and drawing a
conclusion
about people in general.
- Extrapolation to a larger population occurs when the
sampling method is biased. For
example, results from a
voluntary sample for an observational study cannot justifiably be
extended to people who do not volunteer to participate.
Extrapolation beyond the range of data
This is similar to extrapolating to a larger population,
but concerns the values of the variables rather than the
individuals. More
Ignoring Ecological Validity
This is also similar to extrapolating to a larger population,
but
the extrapolation may involve the setting
(i.e., the "ecology") rather than the individuals studied, or it may
involve extrapolation to a population having characteristics very
different from the population that is relevant for application.
Examples:
1. An
experiment designed to study whether an abstract or concrete
approach works better for teaching abstract concepts1 used
computer-delivered instruction. This was done to avoid confounding
variables such as the effect of the teacher. However, the study then
lacked ecological
validity for most real-life classroom instruction.
2. A review of articles on molecular diagnostic tests2 found
that "Of 108 articles included in the study, 82 (76%) used a design
that used healthy controls or alternative-diagnosis controls, only 15
(11%) addressed a clinically relevant population similar to that in
which the test might be applied in practice." The use of a control
group not from a clinically relevant population is an instance of lack
of ecological validity.
Using overly strong language in stating results
Statistical procedures do not prove results. They only give us
information on whether or not the data support or are consistent with a
particular conclusion. There is always uncertainty involved. Thus
results need to be phrased in ways that acknowledge this uncertainty.
Considering statistical significance but not practical
significance
Example:
Suppose that a well-designed, well-carried out, and carefully analyzed
study shows that there is a statistically significant difference in
life span between people engaging in a certain exercise regime at least
five hours a week for at least two years and those not following the
exercise regime. If the difference in average life span between the two
groups is three days, ... well, so what?
1. J.A. Kaminski, V.M.
Sloutsky, & A.F. Heckler. 2008. The Advantage of Learning Abstract
Examples in Learning Math. Science. Vol. 320: 454-455. This study also
presents other problems in interpretation and applicability. For
example, teh subjects were college students, so applying the results to
elementary school students would be a serious extrapolation.
2. B. Lumbreras et al (2009), Overinterpretation
of clinical applicability in molecular diagnostic research, Clin
Chem. 2009 Apr;55(4):786-94. The quote is from the abstract. The
authors also point out other types of overinterpretation.