The power is the long-term probability of a series of identical studies to detect a statistically significant effect (eg. p<0.05) if there is any. The probability of a type 2 error in a series of identical studies is one minus the power (1-ß, often 20%).
One hundred studies are conducted within the same population with the same treatment A vs treatment B structure. The true treatment difference in real life between A and B is a 30% higher chance of full recovery in treatment A. When the stats are performed on these one hundred studies (same population, same variance, same standard deviation), on average about 20 studies will not show a statistically significant effect. This is the type 2 error rate, or false negatives— directly related to the statistical power (1-ß).
So to put it simply, an inadequately powered study will less often show a statistically significant effect, while there actually is a difference.
This influences power
Power is influenced by a few factors, just like with p-values.
Sample size: bigger sample = more power (clearer differences between groups, fewer data noise)
Variance: smaller variance = more power
Effect sizes: bigger effect sizes = more power (easier to spot by a test)
Type of statistical test: some tests yield more power in exchange for more assumptions (there are no free lunches in stats)
It is crucial to understand, though, that the statistical power (eg. 80%) is there for one measurement tool, for one point in time, for one effect size.
Low power = unreliable study
So an underpowered study increases the risk of type 2 errors (false negatives), but, it increases the risk of type 1 errors as well (false positives), with inflated effects. This is called ‘the winner’s curse’. This is why you simply cannot throw multiple outcome measures at a sample size and measure at multiple points in time without letting your statistical power crash. Good researchers and clinicians know that secondary outcome measures are merely suggestive because the study is not powered for that amount of measures. You need new studies to confirm those suggestions. The problem described above is referred to as the multiple comparison problem.
I can imagine this sounds a bit counterintuitive. Let’s look at an example.
You are lecturing a group of 200 students and decide to split them up into two groups. The aim of your study is to see if there are gender differences like more females in one group compared to the other. There’s no difference. You then look at eye color, hair color, length of their index finger, benchpress PR, QOL, age, amount of siblings, etc. Chances are you will encounter a statistically significant result somewhere. This is the multiple comparison problem.
To avoid underpowered studies and the risk of false positives or false negatives, researchers must plan their studies with adequate power. This requires consideration of factors such as sample size, effect size, variance, and the statistical test used. Multiple testing also poses a risk of false positives, which can be addressed through methods such as adjusting the significance level or using False Discovery Rate control. By understanding the concept of statistical power and its importance in hypothesis testing, researchers can design studies that produce reliable and meaningful results.
What customers have to say about the Assessment E-Book
The Assessment E-Book This book helped me in my studying for my exam and in assessing my first patients. Awesome! Also for beginners!
The Assessment E-Book It’s an amazing Compilation! Congrats to all the work you have put in there. You’ll propably find all the test’s you’ve been looking for with propper explaination and source to doublecheck for you self. definetly a must have for every student, but it will also help an experienced practioner. Im looking forward to the lifelong updates on the topics.
Great work, guys
The Assessment E-Book A must-have for all physiotherapists, osteopaths and manual therapists. The authors conducted an extensive research on assessment tests in manual therapy. I find it very easy to read. The more I read the more I learn. Thank you!
The Assessment E-Book This book is great! It is very structured and detailed. It works extremely well on my Macbook and iPad.
The Assessment E-Book The best way to spend 80euros. Totally worth it. The amount of work you put behind this must have been absolutely huge. Every physical or physiotherapist should own it.
Congrats guys you’ve done an incredible job.
I’ve learnd a lot of new things and my approach to therapy in general have totally changed.
In one word: amazing. Keep going guys ! Best wishes from france.
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.