COMMON MISTEAKS MISTAKES IN USING STATISTICS: Spotting and Avoiding Them

Conditional distribution: A probability distribution obtained by restricting certain variables to certain values. This then gives a conditional distribution of the remaining variables. For example, we could talk about the conditional distribution of people's heights given that the people are of age greater than 80.

Conditional probability: A probability with some restriction imposed; e.g., the conditional probability of dying of a heart attack in the next five years for people over age 65 is a conditional probability, since only people over age 65 are being considered. A conditional probability can also be considered as a probability restricting to a subpopulation.

Data snooping: Inference someone decides to do after looking at the data (contrasted with pre-planned inference). Data snooping can be done ethically, unethically due to ignorance, or unethically by intent.

Experiment: A study where the researchers deliberately do something (an "intervention" or "manipulation" or "assignment to treatment") to affect at least some of the data collected.

Extrapolation: Extrapolation is a broad word referring to various ways of going beyond the data. For example, one can extrapolate beyond the range of the explanatory variables or beyond the population from which the data are drawn. The further from the data the extrapolation, the less reliable the results.

File drawer problem: The tendency not to publish results that are not statistically significant (or have a result that is not what is hoped for), giving a misrepresnetaion in the literature. Also called publication bias.

Family-wise Error Rate: The probability that a randomly chosen sample (of the given size, satisfying the appropriate model assumptions) will give a Type I error for at least one of the hypothesis tests performed on the sample. Used when more than one hypothesis test is performed on the same data. Also known as overall Type I error rate, or joint significance level, or simultaneous significance level, or joint Type I error rate, or experiment-wise error rate, etc. Also abbreviated FWER.

Fixed effect factor: Data has been gathered from all the levels of the factor that are of interest

Intent-to-treat analysis: In a controlled comparison (for example, a clinical trial comparing two drugs), an intent-to-treat analysis compares the results of all subjects randomized to each treatment. This preserves the randomization (which may be needed to legitimize the statistical analysis) and also is typically what is of interest to the consumer.

Model assumptions: Most frequentist techniques for statistical inference assume certain properties of the data collection, of the random variables studied, etc. If the assumptions are not valid for the application at hand, the results of the inference may not be true. However, some techniques are robust to some assumptions -- that is, they are still approximately true if the model assumption is close to true.

Multiple inference: Performing more than one hypothesis test on the same data. Also known as joint inference, or simultaneous inference, or multiple testing, or multiple comparisons, or the problem of multiplicity.

Overfitting: Obtaining a regression model that may fit the data very well, but at the expense of not fitting the population from which the data were obtained.

Parameter: A number associated with a random variable or its distribution that helps characterize the distribution. For example, the mean and standard deviation are parameters. If we know that a distribution is normal, then knowing the values of these two parameters tells us exactly which normal distribution it is.

- Population: A particular group of subjects being studied. For example, we might talk about the population of all adults over age 50 who have high blood pressure, or the population of all adults over age 50, or the population of all adults over 60, or the population of all adults who have high blood pressure, etc.

Pre-planned inference: Inference planned as part of the design of a study, before looking at the data.

Probability distribution: A way of describing how the probabilities of values of a random variable vary.

Pseudoreplication: Using data that does not have true replication for each experimental condition.

- Random: Refers to the method by which a sample is chosen, not to properties of the resulting sample.

Random Variable: Can usually be thought of as a variable whose value depends on a random process.

Replication: May refer to having more than one experimental (or observational) unit with the same treatment, or may refer to repeating a study to see if the same conclusion is obtained with different data.

Robust: A statistical technique is said to be robust to departures from a model assumption if the technique is still approximately valid if that model assumption is not true.

Sample: A collection of individuals from a particular population that are chosen for study, with the intent of saying something about the overall population.

Significance level: The probability of falsely rejecting a true null hypthesis when repeatedly using a specific hypothesis test on different samples.

Skewed distribution: A distribution of a random variable whose values are not distributed symmetrically, and values on one end of the range are more frequent than those on the other end. The "tail" is the end where values are less frequent.

Type I error: Falsely rejecting a true null hypothesis.

Type II error: Failing to reject the null hypothesis when the null hypothesis is not true.

Uncertainty: Used in various (related) ways by various speakers and authors, but sometimes distinguished from variability.

Variability: Used in various (related) ways by various speakers and authors, but sometimes distinguished from uncertainty.

Last updated February 4, 2013