Significance level - conventional level of significance

Table of Contents

# Conventional levels of significance

The probability of a statistical test (normally referred to as 'p' or 'p-value') measures the likelihood of obtaining similar or more extreme results if the null hypothesis were true. This probability runs between '0' and '1', with lower values representing a smaller probability of obtaining those results if the null hypothesis were true (ie, a very rare event).

In this context, a predetermined significance level can be used as a "cutoff point" for deciding whether the results of a statistical test are improbable enough for the null hypothesis to remain valid. A significance level of '0.05' is conventionally used in the social sciences, although probabilities as high as '0.10' as well as lower probabilities may also be used. Probabilities greater than '0.10' are rarely used.

A significance level of '0.05', for example, indicates that the result is extreme enough as to have a 95% probability of appearing if the null hypothesis were true. A significance level of '0.01' indicates a 99% probability of appearing if the null hypothesis were true. Thus, smaller probabilities can be used as indirect evidence to disprove or reject the null hypothesis.

Fisher introduced levels of significance as a convention in 1935, although he rejected them years later in favor of exact levels of significance (according to Gigerenzer, 2004). The former can be interpreted as a 'special case' of the latter, though, as an exact level of significance (ie, a 'p-value') needs to be obtained before a conventional level of significance is used for making the decision whether to reject or not the null hypothesis.

In any case, conventional levels of significance may be better understood within Fisher's theoretical positioning within statistics:

• Inference is based on a frequentist approach to statistics, thus random sampling and controlled experiments are a necessary prerequisite to reduce non-random error. Given these conditions then, the level of significance is a property of the data themselves.
• Only the null hypothesis is tested. Thus, non-significant results are ignored, while significant results may be considered for rejecting the null hypothesis. Significant results, however, can say nothing about an alternative hypothesis, other than as opposition to the null hypothesis (eg, we may be able to reject the null hypothesis that there is no correlation between a pair of variables, but we can say nothing regarding whether the correlation is real nor whether it is due to the particular variables being correlated).
• The null hypothesis cannot be proved.

# Properties

• The significance level is a probability figure between '0' and '1' associated to some statistical tests.
• This figure is used as a cut-off point for rejecting a null hypothesis (eg, that there is no difference between groups), but not for drawing any conclusions about unstated or untested "alternative hypotheses".
• Cut-off points are set up by convention. In the social sciences, conventional cut-off points are '0.05' or '0.01', representing a 95% and 99% probability that the results are extreme enough for supporting the null hypothesis if it were true.
• All things being equal, standard errors will be larger in smaller data sets, so it may make sense to choose '0.1' [as significance level] in a smaller data set. Similarly, in large data sets (hundreds of thousands of observations or more), it is not uncommon for nearly every test to be significant at the […] '0.05' level; therefore the more stringent level of '0.01' is often used (or even '0.001' in some instances) (Noymer, undated1).
• A result is said to be significant when its "p-value" is equal of lower than the cut-off point. You should see it expressed as, for example, p < .05, or p < .01).
References
1. NOYMER Andrew (undated). Alpha, significance level of test. In, Paul J LAVRAKAS (undated). Encuclopedia of survey research methods. ISBN 9781412918084.

# Contributors to this page

## Authors / Editors

 Other interesting sites
page revision: 8, last edited: 20 Aug 2011 23:13
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License