Hypothesis testing - disambiguation
There are different approaches used to test hypotheses in statistics, although confusion about them is common, even to the point of bringing the whole procedure to the realm of pseudoscience. Below is a disambiguation of the main approaches to hypotheses testing:
Hypothesis testing procedures
|Test of significance3||probability of data given Ho||p(D|Ho)||Fisher|
|Hypotheses testing||probability of data given H1 & H2||p(D|H1)/p(D|H2)||Neyman-Pearson|
|Bayes' theorem||probability of Hs given data||p(H|D)||Bayes|
|Null hypothesis significance testing||(pseudoscientific approach that mixes up all three procedures)2,1|
Hypothesis testing according to authors
|Fisher||test of significance3||probability data given Ho||p(D|Ho)|
|Neyman-Pearson||hypotheses testing||probability data given H1 & H2||p(D|H1)/p(D|H2)|
|Bayes||Bayes' theorem||probability Hs given data||p(H1|D)|
Brief explanatory notes
- Tests of significance are normally mistaken for a procedure for testing a null hypothesis. However, the test is run under the assumption of an imaginary infinite distribution were the null hypothesis is true. Thus, the test only calculates the probability of the observed data against this imaginary distribution. It was formulated by Fisher as an inductive approach, were the probability of the data may flag "significant" results that could be used as evidence against the null hypothesis. Thus, the test neither is a proper test of a null hypothesis nor can be used as evidence to support alternative ones.
- Hypotheses testing procedures are used to assess the probability of a (null) hypothesis against one or more alternative hypotheses. Although the test itself can only assess the probability of the data, Neyman and Pearson assumed all hypotheses to be based on, thus deduced from real populations. Therefore, data can actually help make decisions, acting as if they proved or disproved hypotheses (even if the researchers believed otherwise). Unfortunately, Neyman and Pearson called their initial hypothesis as the null hypothesis (probably following on the steps of Fisher) against which one or more alternative hypotheses would be tested. They also used a similar procedure than Fisher's significance testing for estimating their type I error (of wrongly rejecting the null hypothesis). Both actions, among others, simply seeded future confusion (the NHST).
- Bayes' theorem is a procedure for actually testing hypotheses (assuming the data are correct). It requires prior knowledge of the probability of events and, if appropriate, of the instruments with which those events are observed.
- Null hypothesis significance testing (NHST) is a pseudoscientific approach to hypotheses testing born from a mix up of above three approaches (and perpetuated in published research and teaching materials). Namely, researchers confuse Neyman-Pearson's null hypothesis with Fisher's, and assume both procedures are one and the same. Thus, research is simplified because only one hypothesis (the null) is specified and measured, an alternative hypothesis is assumed (which is the negation of the null, rather than a proper hypothesis about a different population), the significant test (p) is used to both reject the null hypothesis and accept the alternative one (thus, using 'p' both as evidence against the null hypothesis, at a conventional level such as p < 0.05, and as a probability of making a type I error when accepting the alternative hypothesis, at a level such as α < 0.05). Finally, a decision is made unknowingly following a Bayesian procedure (more than a Neyman-Pearson's one), where hypotheses are proved or disproved (sometimes, accepted or rejected) by using 'p' levels as the probability of correctness or truth of the hypotheses.
Want to know more?
- Null hypothesis testing
- This page deals with Fisher's significance testing procedure, based on simply testing data against one hypothesis (the null hypothesis).
Contributors to this page
Authors / Editors
|Other interesting sites|
Wiki of Science
The Balanced Nutrition Index