[PEREZGONZALEZ Jose D (2014). Misinterpretation of the p-value and of the level of significance (2000). Knowledge (ISSN 1177-4576), 2014, pages 24-25.] [DOI] [Printer friendly]
Misinterpretation of 'sig' and 'p'
Oakes (19863) was the first in exploring the extent to which British academic users misinterpreted inferential statistics, namely 'statistical significance' and the 'p-value'. His research was later replicated by Falk & Greenbaum in Israel (19951), and Haller & Krauss (20002) in Germany. This article offers the results of a meta-analysis done on the collated data.
Results (see illustration 1) show that, overall, about 91% of academic users of statistics (including researchers, teachers and students) misunderstood the concepts of probability and statistical significance in the context of the so-called "null hypothesis testing".
|Illustration 1. Collated frequencies and percentages of misinterpretations regarding tests of significance|
|Statistical significance proves the alternative hypothesis||21||8.9%||(236)|
|Statistical significance disproves the null hypothesis||27||11.4%||(236)|
|The p-value informs of the probability of the null hypothesis||96||40.7%||(236)|
|The p-value informs of the probability of the alternative hypothesis||97||41.1%||(236)|
|The p-value informs of the probability of the results if replicated||90||49.2%||(183)|
|The p-value informs of the probability of making a wrong decision when rejecting the null||138||75.4%||(183)|
|(Participants who answered that all of above were false)||20||8.5%||(236)|
|f = frequencies, or times a misinterpretation was chosen; n = collated sample size|
As far as misinterpretations go, misunderstandings regarding the meaning of the p-value appeared more pronounced than those regarding proving or disproving hypotheses. About 9% of academic users thought that achieving statistical significance proved the alternative hypothesis, while a slightly larger percentage (11%) thought achieving statistical significance disproved the null hypothesis.
As per the p-value (or the probability of data given that the null hypothesis is true), 75% of academic users misinterpreted it as the probability of making the so-called 'Type I error' (rejecting the null hypothesis when it is true), and almost half of them misinterpreted it as the probability of obtaining similar results if the study were to be replicated. To a lesser degree, they misinterpreted the p-value as the probability of the null hypothesis being true (41%) or as the probability of the alternative hypothesis being false (41%).
Jose D PEREZGONZALEZ (2014). Massey University, Turitea Campus, Private Bag 11-222, Palmerston North 4442, New Zealand. (JDPerezgonzalez).
Want to know more?
- WikiofScience - Null hypothesis significance testing procedure
- This WikiofScience page offers an introduction to the pseudoscientific use of the so-called 'null hypothesis testing'.
- WikiofScience - Related studies
- You can find more information on the studies used for this meta-analysis on WikiofScience: the study done by Oakes in 1986; the replication done by Falk & Greenbaum in 1995; and the replication done by Haller & Krauss in 2000.
- WikiofScience - Tests of significance
- This page offers a correct procedure for testing the research data against a null hypothesis.