|[Data]||[<Normal page] [PEREZGONZALEZ Jose D (2014). Misinterpretation of the p-value and of the level of significance (2000). Knowledge (ISSN 1177-4576), 2014, pages 24-25.] [DOI]|
Misinterpretation of 'sig' and 'p'
Oakes (19863) was the first in exploring the extent to which British academic users misinterpreted inferential statistics, namely 'statistical significance' and the 'p-value'. His research was later replicated by Falk & Greenbaum in Israel (19951), and Haller & Krauss (20002) in Germany. This article offers the results of a meta-analysis done on the collated data.
Results (see illustration 1) show that, overall, about 91% of academic users of statistics (including researchers, teachers and students) misunderstood the concepts of probability and statistical significance in the context of the so-called "null hypothesis testing".
|Illustration 1. Collated frequencies and percentages of misinterpretations regarding tests of significance|
|Statistical significance proves the alternative hypothesis||21||8.9%||(236)|
|Statistical significance disproves the null hypothesis||27||11.4%||(236)|
|The p-value informs of the probability of the null hypothesis||96||40.7%||(236)|
|The p-value informs of the probability of the alternative hypothesis||97||41.1%||(236)|
|The p-value informs of the probability of the results if replicated||90||49.2%||(183)|
|The p-value informs of the probability of making a wrong decision when rejecting the null||138||75.4%||(183)|
|(Participants who answered that all of above were false)||20||8.5%||(236)|
|f = frequencies, or times a misinterpretation was chosen; n = collated sample size|
As far as misinterpretations go, misunderstandings regarding the meaning of the p-value appeared more pronounced than those regarding proving or disproving hypotheses. About 9% of academic users thought that achieving statistical significance proved the alternative hypothesis, while a slightly larger percentage (11%) thought achieving statistical significance disproved the null hypothesis.
As per the p-value (or the probability of data given that the null hypothesis is true), 75% of academic users misinterpreted it as the probability of making the so-called 'Type I error' (rejecting the null hypothesis when it is true), and almost half of them misinterpreted it as the probability of obtaining similar results if the study were to be replicated. To a lesser degree, they misinterpreted the p-value as the probability of the null hypothesis being true (41%) or as the probability of the alternative hypothesis being false (41%).
Meta-analysis done on data collated from three exploratory studies.
Frequency of responses from 236 users of statistics for six of the items, and from 183 users for the remaining two items4. The users were a mix of university lecturers, researchers and students from three countries: the UK (Oakes, 19863), Israel (Falk & Greenbaum, 19951), and Germany (Haller & Krauss, 20002).
Six statements reflecting common misinterpretations of the p-value and tests of significance.
Want to know more?
- WikiofScience - Null hypothesis significance testing procedure
- This WikiofScience page offers an introduction to the pseudoscientific use of the so-called 'null hypothesis testing'.
- WikiofScience - Related studies
- You can find more information on the studies used for this meta-analysis on WikiofScience: the study done by Oakes in 1986; the replication done by Falk & Greenbaum in 1995; and the replication done by Haller & Krauss in 2000.
- WikiofScience - Tests of significance
- This page offers a correct procedure for testing the research data against a null hypothesis.
Jose D PEREZGONZALEZ (2014). Massey University, Turitea Campus, Private Bag 11-222, Palmerston North 4442, New Zealand. (JDPerezgonzalez).