Alpha (Type I error)

Alpha (α) is the probability of making a Type I error while testing two hypotheses.

Alpha represents an area were two population distributions may coincide. Data that fall within this area may pertain either to one or the other population. Thus, deciding whether the data are representative of one or the other is subjected to two types of error:

A Type I error is made when we decide that the data is representative of one population (typically phrased as the alternative hypothesis) and not the other (typically phrased as the null hypothesis) when the data is, indeed, representative of the latter. Said otherwise, we make a Type I error when we reject the null hypothesis (in favor of the alternative one) when the null hypothesis is correct.
- The alpha level (α) is the probability we want to have, thus determined beforehand, of making such error. It is conventionally set at 5% (ie, α = 0.05), indicating a 5% chance of making a Type I error.
- The alpha level also informs us of the specificity (= 1 - α) of a test (ie, the probability of retaining the null hypothesis when it is, indeed, correct).

A Type II error is made when we decide that the data is representative of one population (typically phrased as the null hypothesis) and not the other (typically phrased as the alternative hypothesis) when the data is, indeed, representative of the latter. Said otherwise, we make a Type II error when we fail to reject the null hypothesis (in favor of the alternative one) when the alternative hypothesis is correct.
- The beta level (β) is the probability we want to have, thus determined beforehand, of making such error. It is conventionally set at 10% (ie, α = 0.10), indicating a 10% chance of making a Type II error.
- The beta level also informs us of the power (= 1 - β) of a test (ie, the probability of accepting the alternative hypothesis when it is, indeed, correct).

Neyman and Pearson used the concept of level of significance as a proxy for the alpha level. This level of significance, always set beforehand, represents the probability of making a Type I error in the long run, ie after repeated experimentation under control conditions. Thus, an alpha / significance level of 0.05 indicates a 5% chance of making such error in the long run (quoted by Gigerenzer, 2004). In any case, the alpha level is better understood within Neyman-Pearson's theoretical positioning within statistics:

Inference is based on a frequentist approach with repeated measuring, thus random sampling, controlled experiments and repeated experimentation, are a necessary prerequisite to reduce non-random error. Given these conditions then, the level of significance is a property of the test (not of the data).
Two hypotheses are tested at once. Thus, we may be able to prove or disprove the null hypothesis, as well as to prove or disprove the alternative one.
Because we are testing two hypotheses, we can make two errors with the same test: a Type I error (rejecting the null hypothesis when the null hypothesis is correct), or a Type II error (rejecting the alternative hypothesis when the alternative hypothesis is correct). Thus, we need to decide beforehand acceptable levels for both errors, α and β, as well as acceptable power for the test (1-β), which depends on the sample size.

Properties

The alpha level is a probability figure between '0' and '1'.
This figure is used to decide whether to reject the null hypothesis and, thus, accept the alternative one.

References

1. NOYMER Andrew (undated). Alpha, significance level of test. In, Paul J LAVRAKAS (undated). Encyclopedia of survey research methods. ISBN 9781412918084.