20141210 - Misinterpretation of the p-value and of the level of significance (2000)

[Data] [<Normal page] [PEREZGONZALEZ Jose D (2014). Misinterpretation of the p-value and of the level of significance (2000). Knowledge (ISSN 1177-4576), 2014, pages 24-25.] [DOI]

Misinterpretation of 'sig' and 'p'

Oakes (19863) was the first in exploring the extent to which British academic users misinterpreted inferential statistics, namely 'statistical significance' and the 'p-value'. His research was later replicated by Falk & Greenbaum in Israel (19951), and Haller & Krauss (20002) in Germany. This article offers the results of a meta-analysis done on the collated data.

Results (see illustration 1) show that, overall, about 91% of academic users of statistics (including researchers, teachers and students) misunderstood the concepts of probability and statistical significance in the context of the so-called "null hypothesis testing".

Illustration 1. Collated frequencies and percentages of misinterpretations regarding tests of significance
Common misinterpretations f % (n)
Statistical significance proves the alternative hypothesis 21 8.9% (236)
Statistical significance disproves the null hypothesis 27 11.4% (236)
The p-value informs of the probability of the null hypothesis 96 40.7% (236)
The p-value informs of the probability of the alternative hypothesis 97 41.1% (236)
The p-value informs of the probability of the results if replicated 90 49.2% (183)
The p-value informs of the probability of making a wrong decision when rejecting the null 138 75.4% (183)
(Participants who answered that all of above were false) 20 8.5% (236)
f = frequencies, or times a misinterpretation was chosen; n = collated sample size

As far as misinterpretations go, misunderstandings regarding the meaning of the p-value appeared more pronounced than those regarding proving or disproving hypotheses. About 9% of academic users thought that achieving statistical significance proved the alternative hypothesis, while a slightly larger percentage (11%) thought achieving statistical significance disproved the null hypothesis.

As per the p-value (or the probability of data given that the null hypothesis is true), 75% of academic users misinterpreted it as the probability of making the so-called 'Type I error' (rejecting the null hypothesis when it is true), and almost half of them misinterpreted it as the probability of obtaining similar results if the study were to be replicated. To a lesser degree, they misinterpreted the p-value as the probability of the null hypothesis being true (41%) or as the probability of the alternative hypothesis being false (41%).


Methods

Research approach

Meta-analysis done on data collated from three exploratory studies.

Sample

Frequency of responses from 236 users of statistics for six of the items, and from 183 users for the remaining two items4. The users were a mix of university lecturers, researchers and students from three countries: the UK (Oakes, 19863), Israel (Falk & Greenbaum, 19951), and Germany (Haller & Krauss, 20002).

Materials

Six statements reflecting common misinterpretations of the p-value and tests of significance.

Analysis

Percentages.

References
1. FALK Ruma & Charles W GREENBAUM (1995). Significance tests die hard: the amazing persistence of a probabilistic misconception. Theory & Psychology, 1995, volume 5, number 1, pages 75-98. DOI 10.1177/0959354395051004.
2. HALLER Heiko & Stefan KRAUSS (2000). Misinterpretations of significance. A problem students share with their teachers. Methods of Psychological Research Online, 2002, volume 7, number 1, pages 1-20.
3. OAKES Michael (1986). Statistical inference: a commentary for the social and behavioral sciences. John Wiley & Sons (Chichester, UK), 1986.
+++ Notes +++
4. Those two items were not used by Falk & Greenbaum (19951).

Want to know more?

WikiofScience - Null hypothesis significance testing procedure
This WikiofScience page offers an introduction to the pseudoscientific use of the so-called 'null hypothesis testing'.
WikiofScience - Related studies
You can find more information on the studies used for this meta-analysis on WikiofScience: the study done by Oakes in 1986; the replication done by Falk & Greenbaum in 1995; and the replication done by Haller & Krauss in 2000.
WikiofScience - Tests of significance
This page offers a correct procedure for testing the research data against a null hypothesis.

Author

Jose D PEREZGONZALEZ (2014). Massey University, Turitea Campus, Private Bag 11-222, Palmerston North 4442, New Zealand. (JDPerezgonzalezJDPerezgonzalez).


Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License