What’s the Alpha Level? | Statistics

In frequentist statistics, the alpha level (also known as the significance level) is the probability of rejecting the null hypothesis when it is true. In the context of physiotherapy research, the null hypothesis might be that there is no difference in pain reduction between two different physiotherapy interventions. The alpha level is typically set at 0.05, which means that there is a 5% chance of incorrectly rejecting the null hypothesis (i.e., concluding that there is a difference in pain reduction when there actually isn’t) in the long term.

It is especially important to consider this as a long-term result. If 100 similar studies are conducted, 5 of them, on average, will show a false positive result if there is no effect.

Explained with an example

Let’s say a study compares two physiotherapy interventions for lower back pain, and the results show that the mean pain reduction for Intervention A is 6 points on a pain scale, and the mean pain reduction for Intervention B is 8 points on a pain scale. With an alpha level of 0.05, the researchers would reject the null hypothesis and conclude that there is a statistically significant difference in pain reduction between the two interventions because the difference in means is greater than what would be expected by chance.

All hail p<0.05?

It’s important to note that setting an alpha level of 0.05 is a convention and not a rule. The choice of alpha level depends on the context of the study and the potential consequences of a false positive or false negative result. For example, if the consequences of a false positive result (i.e., concluding that a treatment is effective when it is not) are more severe, researchers might choose to use a lower alpha level (e.g. 0.01) to decrease the probability of a false positive result.

Long term view

We want to stress again the importance of a long-term view. You cannot simply say there’s a 5% chance that the paper has become a false positive result. When the research is conducted, it simply is a false positive, or it is not. The 5% speaks of long-term results. Doing this test in multiple studies with similar conditions will result in about 5% of the papers having a false positive result.

A physiotherapy intervention may appear to be very effective for reducing lower back pain, with a small p-value (indicating a statistically significant difference) and a large effect size. However, if this single study is not replicated in other studies, it’s difficult to determine if the results are due to chance or a real effect.

A long-term view considers the results of multiple studies over time to provide a more comprehensive understanding of the effectiveness of an intervention. This approach is particularly important in physiotherapy research, where the results of a single study may not generalize to other populations or settings.

Misconceptions

There are several common misconceptions surrounding the p-value:

• P-value is a measure of the strength of evidence: The p-value does not measure the strength of evidence against the null hypothesis, but rather the probability of observing a test statistic as extreme or more extreme than the one observed, assuming the null hypothesis is true
• Small p-value means strong evidence against the null hypothesis: A small p-value only indicates that the data are not consistent with the null hypothesis, but it does not provide evidence for the alternative hypothesis. Furthermore, a small p-value does not imply that the effect is large or important
• P-value of 0.05 is a hard threshold for significance: The 0.05 threshold is arbitrary and has been adopted as a conventional cutoff for statistical significance, but it does not mean that results with p-values greater than 0.05 are automatically not significant. The interpretation of the p-value should depend on the context and the research question being studied
• P-value is the probability of the null hypothesis being true: The p-value is not the probability that the null hypothesis is true, but rather the probability of observing the data if the null hypothesis is true
• P-value can be used to make a causal inference: The p-value only provides evidence for or against a null hypothesis and does not necessarily imply causality. Causal inference requires additional information, such as a well-designed study with appropriate controls for confounding factors

For more information around the P-value. Check out our post on it here!

Upshur, R. E. (2001). The ethics of alpha: reflections on statistics, evidence and values in medicine. Theoretical Medicine and Bioethics, 22(6), 565-576.

Berger, V. W., & Hsieh, G. (2005). Rethinking statistics: basing efficacy alpha levels on safety data in randomized trials. Israeli Journal of Emergency Medicine, 5(3), 55-60.

Neyman, J. and Pearson, E.S. (1928) On the Use and Interpretation of Certain Test Criteria for Purposes of Statistical Inference. Biometrika, 20A, 175-240.

Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European journal of epidemiology, 31(4), 337–350.