What’s the Alpha Level? | Statistics
In frequentist statistics, the alpha level (also known as the significance level) is the probability of rejecting the null hypothesis when it is true. In the context of physiotherapy research, the null hypothesis might be that there is no difference in pain reduction between two different physiotherapy interventions. The alpha level is typically set at 0.05, which means that there is a 5% chance of incorrectly rejecting the null hypothesis (i.e., concluding that there is a difference in pain reduction when there actually isn’t) in the long term.
It is especially important to consider this as a long-term result. If 100 similar studies are conducted, 5 of them, on average, will show a false positive result if there is no effect.
Explained with an example
Let’s say a study compares two physiotherapy interventions for lower back pain, and the results show that the mean pain reduction for Intervention A is 6 points on a pain scale, and the mean pain reduction for Intervention B is 8 points on a pain scale. With an alpha level of 0.05, the researchers would reject the null hypothesis and conclude that there is a statistically significant difference in pain reduction between the two interventions because the difference in means is greater than what would be expected by chance.
All hail p<0.05?
It’s important to note that setting an alpha level of 0.05 is a convention and not a rule. The choice of alpha level depends on the context of the study and the potential consequences of a false positive or false negative result. For example, if the consequences of a false positive result (i.e., concluding that a treatment is effective when it is not) are more severe, researchers might choose to use a lower alpha level (e.g. 0.01) to decrease the probability of a false positive result.
Long term view
We want to stress again the importance of a long-term view. You cannot simply say there’s a 5% chance that the paper has become a false positive result. When the research is conducted, it simply is a false positive, or it is not. The 5% speaks of long-term results. Doing this test in multiple studies with similar conditions will result in about 5% of the papers having a false positive result.
A physiotherapy intervention may appear to be very effective for reducing lower back pain, with a small p-value (indicating a statistically significant difference) and a large effect size. However, if this single study is not replicated in other studies, it’s difficult to determine if the results are due to chance or a real effect.
A long-term view considers the results of multiple studies over time to provide a more comprehensive understanding of the effectiveness of an intervention. This approach is particularly important in physiotherapy research, where the results of a single study may not generalize to other populations or settings.
There are several common misconceptions surrounding the p-value:
- P-value is a measure of the strength of evidence: The p-value does not measure the strength of evidence against the null hypothesis, but rather the probability of observing a test statistic as extreme or more extreme than the one observed, assuming the null hypothesis is true
- Small p-value means strong evidence against the null hypothesis: A small p-value only indicates that the data are not consistent with the null hypothesis, but it does not provide evidence for the alternative hypothesis. Furthermore, a small p-value does not imply that the effect is large or important
- P-value of 0.05 is a hard threshold for significance: The 0.05 threshold is arbitrary and has been adopted as a conventional cutoff for statistical significance, but it does not mean that results with p-values greater than 0.05 are automatically not significant. The interpretation of the p-value should depend on the context and the research question being studied
- P-value is the probability of the null hypothesis being true: The p-value is not the probability that the null hypothesis is true, but rather the probability of observing the data if the null hypothesis is true
- P-value can be used to make a causal inference: The p-value only provides evidence for or against a null hypothesis and does not necessarily imply causality. Causal inference requires additional information, such as a well-designed study with appropriate controls for confounding factors
For more information around the P-value. Check out our post on it here!
Upshur, R. E. (2001). The ethics of alpha: reflections on statistics, evidence and values in medicine. Theoretical Medicine and Bioethics, 22(6), 565-576.
Berger, V. W., & Hsieh, G. (2005). Rethinking statistics: basing efficacy alpha levels on safety data in randomized trials. Israeli Journal of Emergency Medicine, 5(3), 55-60.
Neyman, J. and Pearson, E.S. (1928) On the Use and Interpretation of Certain Test Criteria for Purposes of Statistical Inference. Biometrika, 20A, 175-240.
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European journal of epidemiology, 31(4), 337–350.
What customers have to say about the Assessment E-Book
- Vince199225/04/20The Assessment E-Book This book helped me in my studying for my exam and in assessing my first patients. Awesome! Also for beginners!Simon Pagitz06/04/20The Assessment E-Book It’s an amazing Compilation! Congrats to all the work you have put in there. You’ll propably find all the test’s you’ve been looking for with propper explaination and source to doublecheck for you self. definetly a must have for every student, but it will also help an experienced practioner. Im looking forward to the lifelong updates on the topics.
Great work, guys
- Jordi Burrut27/10/19The Assessment E-Book A must-have for all physiotherapists, osteopaths and manual therapists. The authors conducted an extensive research on assessment tests in manual therapy. I find it very easy to read. The more I read the more I learn. Thank you!Josh07/07/19The Assessment E-Book This book is great! It is very structured and detailed. It works extremely well on my Macbook and iPad.
- Polo_soa15/02/19The Assessment E-Book The best way to spend 80euros. Totally worth it. The amount of work you put behind this must have been absolutely huge. Every physical or physiotherapist should own it.
Congrats guys you’ve done an incredible job.
I’ve learnd a lot of new things and my approach to therapy in general have totally changed.
In one word: amazing. Keep going guys ! Best wishes from france.