Introduction        Types of Mistakes        Suggestions        Resources        Table of Contents         About    Glossary

Suggestions for Researchers

The most common error in statistics is to assume that statistical procedures can take the place of sustained effort.

Good and Hardin (2006) Common Errors in Statistics, p. 186

The hard part, and the one where training is so poor, is the a priori thinking about the science of the matter before data analysis -- even before data colleciton. It has been too easy to collect data on a large number of variables in the hope that a fast computer and sophisticated software will sort out the important things -- the "significant" ones (the "just the numbers" approach). Instead, a major effort shoud be mounted to understand the nature of the problem by critical examination of the literature, talking with others working on the general problem, and thinking deeply about alternative hypotheses. Rather than "test" dozens of trivial matters ... there must be a more concerted effort to provide evidence on meaningful questions that are important to a discipline. This is the critical point: the common failure to address important science questions in a fully competent fashion.

Burnham and Anderson (2002) Model Selection and Multimodel Inference, pp. 144 - 145

: Look for, take into account, and report sources of uncertainty.

Specific suggestions for planning research:
Specific suggestions for analyzing data:
Specific suggestions for writing up research:

Critics may complain that we advocate interpreting reports not merely with a grain of salt but with an entire shaker; so be it. ... Neither society nor we can afford to be led down false pathways.

Good and Hardin (2006), Common Errors in Statistics, p. 119

Until a happier future arrives, imperfections in models require further thought, and routine disclosure of imperfections would be helpful.

David Freedman (2008, p. 61)


K. P. Burnham and D. R. Anderson (2002), Model selection and Multimodel Inference: A Practical Information-Theoretic Approach, 2nd ed., Springer

D. Freedman (2008), Editorial: Oasis or Mirage?, Chance v. 21 No 1, pp. 59 -61

P. Good and J. Hardin (2006), Common Errors in Statistics (and How to Avoid Them), Wiley

Harris, A. H. S., R. Reeder and J. K. Hyun (2009), Common statistical and research design problems in manuscripts submitted to high-impact psychiatry journals: What editors and reviewers want authors to know, Journal of Psychiatric Research, vol 43 no15, 1231 -1234

Miller, Jane (2004), The Chicago Guide to Writing about Numbers: The Effective Presentation of Quantitative Information, University of Chicago Press

Robbins, N. (2004), Creating More Effective Graphs, Wiley

Strasak, A. M. et al (2007), The Use of Statistics in Medical Research, The American Statistician, February 1, 2007, 61(1): 47-55

van Belle, G. (2008) Statistical Rules of Thumb, Wiley

1. For more discussion, see:
2.  Nobel Laureate in Physics Richard Feynman offers good advice:

"The only way to have real success in science ... is to describe the evidence very carefully without regard to the way you feel it should be. If you have a theory, you must try to explain what's good and what's bad about it equally. In science, you learn a kind of standard integrity and honesty.
        What Do You Care What Other People Think? (1988) p. 217

There is one feature I notice that is generally missing in "cargo cult science"... It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty — a kind of leaning over backwards... For example, if you're doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it... Details that could throw doubt on your interpretation must be given, if you know them. ... If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. ... In summary, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another. ... The first principle is that you must not fool yourself -- and you are the easiest person to fool. So you have to be very careful about that.
    "Cargo Cult Science", adapted from a commencement address given at Caltech (1974)"

Last updated September 8, 2012