Stats & Methodology

Type 1 Error Rate | Statistics

Check our shop
P value wiki

Type 1 Error Rate | Statistics

Testing multiple variables inflates the type 1 error rate or the false positive rate. This is called the multiple comparison problem. Correcting for this alpha-inflation is not hard. There are two main ways, namely the Bonferroni correction and the Holm correction.

The Bonferroni correction is simple but quite conservative. You divide your alpha level by the number of tests you are about to perform. This will be the new significance level. So in this case:

ɑ / n

ɑ: alpha or significance level

n: number of tests

0.05 / 10 = 0.005

You can thus do this quite easily yourself when reading a paper. If five variables are tested, you know the alpha level should be about 0.01 instead of 0.05 (0.05 / 5). This is under the assumption that the researchers did not perform a boatload of tests “behind the scenes” while not reporting them. This is called data-dredging or p-hacking.

Another way is by simply multiplying the p-value in the paper by the number of tests.

Eg.

P-value = 0.03

0.03 * 10 = 0.3

This means that the previously significant p-value now became insignificant.

A second way to correct the alpha-inflation is the Holm correction. Let’s say the researchers did five tests and thus became five p-values. For the Holm correction to work, they should be ranked from lowest to highest.

Eg.

  • 0,0004
  • 0,0130
  • 0,0172
  • 0,0460
  • 0,0600

The Holm formula is as follows: 

p-value * (m + 1 – k)

m = number of p-values

k = the rank of the p-value

So for the third p-value we get…

0,0172 * (5 + 1 – 3) = 0,0516

… making the results insignificant.

References

Lakens, D., Type 1 error control by Daniel Lakens, youtube

John Ludbrook (1998). Multiple comparison procedures updated. , 25(12), 1032–1037. doi:10.1111/j.1440-1681.1998.tb02179.x 

Giacalone, M., Agata, Z., Cozzucoli, P. C., & Alibrandi, A. (2018). Bonferroni-Holm and permutation tests to compare health data: methodological and applicative issues. BMC Medical Research Methodology, 18(1). doi:10.1186/s12874-018-0540-8

Like what you're learning?

Use the assessment app

  • Over 300 orthopedic physical assessment tests
  • Statistics, basic assessments, and screening tests included
  • Direct links to PubMed references
  • Concise test descriptions
  • Video demonstration
  • Easy search & favorites function
E-Book

ALL ORTHOPEDIC TESTS IN ONE PLACE

SEE ALL PRODUCTS
Assessment app banner
Assessment E-book
Reviews

What customers have to say about the Assessment E-Book

Wait before you go!GET ACCESS TO A FREE HEADACHE REHAB PROGRAM

Subscribe now and download a free home exercise program for headaches designed by our headache expert René Castien