20110907 - Misinterpretation of 'p' (1995)

[Data]

(Note: A more conceptually sound edition of this article exists⁴)
[<Normal page] [PEREZGONZALEZ Jose D [ed] (2011). Misinterpretation of 'p' (1995). Journal of Knowledge Advancement & Integration (ISSN 1177-4576), 2011, pages 107-109.]

Table of Contents

Misinterpretation of 'p'

Generalization potential

Misinterpretation of 'p'

Falk and Greenbaum (1995¹) carried out a study on common misinterpretations of the level of significance (p) among Israeli psychology students, which partly replicates one by Oakes (1986³). Typically, most of these misinterpretations confuse the level of significance (ie, the probability of the data assuming that the null hypothesis is correct) with the probability of proving or disproving hypotheses (be this the null hypothesis or an alternative hypothesis).

Falk and Greenbaum found that almost 87% of the students held at least one misinterpretation out of the four presented (see table 1). Most of the students misinterpreted statistical significance as the probability of the null hypothesis being true.

Table 1. Frequencies and percentages of misinterpretations of statistical significance
Common misinterpretations⁵	f	%
Significance disproves the null hypothesis	2	3.8%
Significance informs of the probability of the null hypothesis	42	79.2%
Significance proves the alternative hypothesis	0	0.0%
Significance informs of the probability of the alternative hypothesis	2	3.8%

(Participants who answered that all of above were false)	7	13.2%

Methods

Research approach

Not much detail. It appears to have been a confirmatory study with a hint of 'quasi-experimental' assumption (the quasi-experiment being that students should had being familiar with Bakan's, 1966, paper, as it had been one of the readings for their Experimental Psychology course).

Sample

A convenient sample of 53 psychology students from the Hebrew University of Jerusalem. The participants had taken two courses in statistics and one course in experimental psychology.

Materials

Not much detail about the materials used. Plausibly a tool consisting of either a verbal or written scenario regarding the results of a test with a predetermined level of significance (p), and a one-item questionnaire with five multiple-choice options. These options presented several interpretations of the results, and the participants could choose as many options as they thought correct. (Unknown to the participants, four statements were false, representing four common misinterpretations of the level of significance. The last statement negated all others.)

Analysis

Descriptive results

Generalization potential

This particular research appears to be limited to population of (undergraduate) psychology students at the Hebrew University of Jerusalem. Yet the results might, at least, serve as a working hypothesis for generalizing to other populations such as the following (in order of decreasing generalization scope):

Israeli psychology academics and graduands from that university.
Israeli psychology students, academics, researchers and graduands, in general.
Professional psychologists trained in Israeli universities.
(See also Oakes, 1986³, original study in Britain, and a partial replication of that study by Haller and Krauss, (2000², in Germany, for potential generalization beyond Israel).

References

1. FALK Ruma & Charles W GREENBAUM (1995). Significance tests die hard: the amazing persistence of a probabilistic misconception. Theory & Psychology, 1995, volume 5, number 1, pages 75-98. DOI 10.1177/0959354395051004.

2. HALLER Heiko & Stefan KRAUSS (2000). Misinterpretations of significance. A problem students share with their teachers. Methods of Psychological Research Online, 2002, volume 7, number 1, pages 1-20.

3. OAKES Michael (1986). Statistical inference: a commentary for the social and behavioral sciences. John Wiley & Sons (Chichester, UK), 1986.

+++ Notes +++

4. The newer edition makes a clearer distinction between p-values and statistical significance.

5. The original research statements have been rephrased here.

Want to know more?

Wiki of Science - Hypotheses testing (disambiguation): This Wiki of Science page lists alternative methods for testing the probability of data or hypotheses.
Wiki of Science - Null hypothesis significance testing: This Wiki of Science page reflects on the pseudoscientific bases of the null hypothesis significance testing (NHST) procedure typically used in the social sciences and medicine.
Wiki of Science - Related studies: You can find more information on two related studies in Wiki of Science. One was the original study done by Oakes in 1986; the other study was a replication done by Haller and Krauss in 2000.