Testing multiple variables inflates the type 1 error rate or the false positive rate. This is called the multiple comparison problem. Correcting for this alpha-inflation is not hard. There are two main ways, namely the Bonferroni correction and the Holm correction.
The Bonferroni correction is simple but quite conservative. You divide your alpha level by the number of tests you are about to perform. This will be the new significance level. So in this case:
ɑ / n
ɑ: alpha or significance level
n: number of tests
0.05 / 10 = 0.005
You can thus do this quite easily yourself when reading a paper. If five variables are tested, you know the alpha level should be about 0.01 instead of 0.05 (0.05 / 5). This is under the assumption that the researchers did not perform a boatload of tests “behind the scenes” while not reporting them. This is called data-dredging or p-hacking.
Another way is by simply multiplying the p-value in the paper by the number of tests.
P-value = 0.03
0.03 * 10 = 0.3
This means that the previously significant p-value now became insignificant if 10 variables were tested.
Bonferroni correction limitations
The Bonferroni correction is a widely used method to adjust the significance level for multiple comparisons in order to control the overall Type I error rate. However, it has several limitations.
One of the main issues is that it can be overly stringent, which may lead to a loss of statistical power. Additionally, it assumes that all comparisons are independent, which may not be the case in real-world data, potentially leading to higher Type II error rates.
Another limitation of the Bonferroni correction is that it increases the chance of false negatives or Type II errors, meaning that there is a higher chance of missing a true effect.
Finally, the Bonferroni correction is most appropriate for situations where the number of comparisons is relatively small, as it may not be as effective when the number of comparisons is very large. Therefore, researchers should carefully consider the appropriateness of the Bonferroni correction for their research question and data set, and be aware of its limitations.
A second way to correct the alpha inflation is the Holm correction. Let’s say the researchers did five tests and thus became five p-values. For the Holm correction to work, they should be ranked from lowest to highest.
The Holm formula is as follows:
p-value * (m + 1 – k)
m = number of p-values
k = the rank of the p-value
So for the third p-value we get…
0,0172 * (5 + 1 – 3) = 0,0516
… making the results insignificant.
Holm Correction Limitations
One limitation is that Holm’s correction assumes that all tests are independent, meaning that the results of one test do not affect the results of another. However, in some cases, the tests may be dependent, such as when testing multiple outcomes from the same sample or when testing different time points from the same intervention. In such cases, Holm’s correction may be too conservative or too liberal, leading to incorrect conclusions. Another limitation of Holm’s correction is that it does not take into account the correlation between the tests, which can affect the false positive rate. For example, if multiple tests are related to the same underlying construct, the probability of detecting a significant effect increases, and Holm’s correction may not adequately account for this. While Holm’s correction is a useful method for adjusting p-values in multiple comparison testing, it is important to consider its limitations, particularly when tests are dependent or correlated. Other methods such as False Discovery Rate control or Bayesian methods may be more appropriate in some cases.
What customers have to say about the Assessment E-Book
The Assessment E-Book This book helped me in my studying for my exam and in assessing my first patients. Awesome! Also for beginners!
The Assessment E-Book It’s an amazing Compilation! Congrats to all the work you have put in there. You’ll propably find all the test’s you’ve been looking for with propper explaination and source to doublecheck for you self. definetly a must have for every student, but it will also help an experienced practioner. Im looking forward to the lifelong updates on the topics.
Great work, guys
The Assessment E-Book A must-have for all physiotherapists, osteopaths and manual therapists. The authors conducted an extensive research on assessment tests in manual therapy. I find it very easy to read. The more I read the more I learn. Thank you!
The Assessment E-Book This book is great! It is very structured and detailed. It works extremely well on my Macbook and iPad.
The Assessment E-Book The best way to spend 80euros. Totally worth it. The amount of work you put behind this must have been absolutely huge. Every physical or physiotherapist should own it.
Congrats guys you’ve done an incredible job.
I’ve learnd a lot of new things and my approach to therapy in general have totally changed.
In one word: amazing. Keep going guys ! Best wishes from france.
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.