Hypothesis testing - disambiguation

Hypothesis testing - disambiguation

There are different approaches used to test hypotheses in statistics, although confusion about them is common, even to the point of bringing the whole procedure to the realm of pseudoscience. Below is a disambiguation of the main approaches to hypotheses testing:

Hypothesis testing procedures

Test Object Expression Author
Test of significance3 probability of data given Ho p(D|Ho) Fisher
Hypotheses testing probability of data given H1 & H2 p(D|H1)/p(D|H2) Neyman-Pearson
Bayes' theorem probability of Hs given data p(H|D) Bayes
Null hypothesis significance testing (pseudoscientific approach that mixes up all three procedures)2,1

Hypothesis testing according to authors

Author Test Object Expression
Fisher test of significance3 probability data given Ho p(D|Ho)
Neyman-Pearson hypotheses testing probability data given H1 & H2 p(D|H1)/p(D|H2)
Bayes Bayes' theorem probability Hs given data p(H1|D)

Brief explanatory notes

  • Tests of significance are normally mistaken for a procedure for testing a null hypothesis. However, the test is run under the assumption of an imaginary infinite distribution were the null hypothesis is true. Thus, the test only calculates the probability of the observed data against this imaginary distribution. It was formulated by Fisher as an inductive approach, were the probability of the data may flag "significant" results that could be used as evidence against the null hypothesis. Thus, the test neither is a proper test of a null hypothesis nor can be used as evidence to support alternative ones.
  • Hypotheses testing procedures are used to assess the probability of a (null) hypothesis against one or more alternative hypotheses. Although the test itself can only assess the probability of the data, Neyman and Pearson assumed all hypotheses to be based on, thus deduced from real populations. Therefore, data can actually help make decisions, acting as if they proved or disproved hypotheses (even if the researchers believed otherwise). Unfortunately, Neyman and Pearson called their initial hypothesis as the null hypothesis (probably following on the steps of Fisher) against which one or more alternative hypotheses would be tested. They also used a similar procedure than Fisher's significance testing for estimating their type I error (of wrongly rejecting the null hypothesis). Both actions, among others, simply seeded future confusion (the NHST).
  • Bayes' theorem is a procedure for actually testing hypotheses (assuming the data are correct). It requires prior knowledge of the probability of events and, if appropriate, of the instruments with which those events are observed.
  • Null hypothesis significance testing (NHST) is a pseudoscientific approach to hypotheses testing born from a mix up of above three approaches (and perpetuated in published research and teaching materials). Namely, researchers confuse Neyman-Pearson's null hypothesis with Fisher's, and assume both procedures are one and the same. Thus, research is simplified because only one hypothesis (the null) is specified and measured, an alternative hypothesis is assumed (which is the negation of the null, rather than a proper hypothesis about a different population), the significant test (p) is used to both reject the null hypothesis and accept the alternative one (thus, using 'p' both as evidence against the null hypothesis, at a conventional level such as p < 0.05, and as a probability of making a type I error when accepting the alternative hypothesis, at a level such as α < 0.05). Finally, a decision is made unknowingly following a Bayesian procedure (more than a Neyman-Pearson's one), where hypotheses are proved or disproved (sometimes, accepted or rejected) by using 'p' levels as the probability of correctness or truth of the hypotheses.
1. COHEN Jacob (1994). The Earth is round (p < .05). American Psychologist (ISSN 0003-066X), 1994, volume 49, number 12, pages 997-1003.
2. GIGERENZER Gerd, Stefan KRAUSS & Oliver VITOUCH (2004). The null ritual: what you always wanted to know about significance testing but were afraid to ask. Chapter 21 in David KAPLAN [ed] (2004), The SAGE handbook of quantitative methodology for the social sciences. SAGE (California, USA), 2004. ISBN 9780761923596.
+++ Footnotes +++
3. Also referred to as Null hypothesis testing, even if this is not quite correct.

Want to know more?

Null hypothesis testing
This page deals with Fisher's significance testing procedure, based on simply testing data against one hypothesis (the null hypothesis).

Contributors to this page

Authors / Editors


Other interesting sites
Journal KAI
Wiki of Science
The Balanced Nutrition Index
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License