20141210 - Misinterpretation of the p-value and of the level of significance (2000)

[Data]

[<Normal page] [PEREZGONZALEZ Jose D (2014). Misinterpretation of the p-value and of the level of significance (2000). Knowledge (ISSN 1177-4576), 2014, pages 24-25.] [DOI]

Table of Contents

Misinterpretation of 'sig' and 'p'

Misinterpretation of 'sig' and 'p'

Oakes (1986³) was the first in exploring the extent to which British academic users misinterpreted inferential statistics, namely 'statistical significance' and the 'p-value'. His research was later replicated by Falk & Greenbaum in Israel (1995¹), and Haller & Krauss (2000²) in Germany. This article offers the results of a meta-analysis done on the collated data.

Results (see illustration 1) show that, overall, about 91% of academic users of statistics (including researchers, teachers and students) misunderstood the concepts of probability and statistical significance in the context of the so-called "null hypothesis testing".

Illustration 1. Collated frequencies and percentages of misinterpretations regarding tests of significance
Common misinterpretations	f	%	(n)
Statistical significance proves the alternative hypothesis	21	8.9%	(236)
Statistical significance disproves the null hypothesis	27	11.4%	(236)

The p-value informs of the probability of the null hypothesis	96	40.7%	(236)
The p-value informs of the probability of the alternative hypothesis	97	41.1%	(236)
The p-value informs of the probability of the results if replicated	90	49.2%	(183)
The p-value informs of the probability of making a wrong decision when rejecting the null	138	75.4%	(183)

(Participants who answered that all of above were false)	20	8.5%	(236)
f = frequencies, or times a misinterpretation was chosen; n = collated sample size

As far as misinterpretations go, misunderstandings regarding the meaning of the p-value appeared more pronounced than those regarding proving or disproving hypotheses. About 9% of academic users thought that achieving statistical significance proved the alternative hypothesis, while a slightly larger percentage (11%) thought achieving statistical significance disproved the null hypothesis.

As per the p-value (or the probability of data given that the null hypothesis is true), 75% of academic users misinterpreted it as the probability of making the so-called 'Type I error' (rejecting the null hypothesis when it is true), and almost half of them misinterpreted it as the probability of obtaining similar results if the study were to be replicated. To a lesser degree, they misinterpreted the p-value as the probability of the null hypothesis being true (41%) or as the probability of the alternative hypothesis being false (41%).

Methods

Research approach

Meta-analysis done on data collated from three exploratory studies.

Sample

Frequency of responses from 236 users of statistics for six of the items, and from 183 users for the remaining two items⁴. The users were a mix of university lecturers, researchers and students from three countries: the UK (Oakes, 1986³), Israel (Falk & Greenbaum, 1995¹), and Germany (Haller & Krauss, 2000²).

Materials

Six statements reflecting common misinterpretations of the p-value and tests of significance.

Analysis

Percentages.

References

1. FALK Ruma & Charles W GREENBAUM (1995). Significance tests die hard: the amazing persistence of a probabilistic misconception. Theory & Psychology, 1995, volume 5, number 1, pages 75-98. DOI 10.1177/0959354395051004.

2. HALLER Heiko & Stefan KRAUSS (2000). Misinterpretations of significance. A problem students share with their teachers. Methods of Psychological Research Online, 2002, volume 7, number 1, pages 1-20.

3. OAKES Michael (1986). Statistical inference: a commentary for the social and behavioral sciences. John Wiley & Sons (Chichester, UK), 1986.

+++ Notes +++

4. Those two items were not used by Falk & Greenbaum (1995¹).

Want to know more?

WikiofScience - Null hypothesis significance testing procedure: This WikiofScience page offers an introduction to the pseudoscientific use of the so-called 'null hypothesis testing'.
WikiofScience - Related studies: You can find more information on the studies used for this meta-analysis on WikiofScience: the study done by Oakes in 1986; the replication done by Falk & Greenbaum in 1995; and the replication done by Haller & Krauss in 2000.
WikiofScience - Tests of significance: This page offers a correct procedure for testing the research data against a null hypothesis.