What’s P-Hacking? | Statistics

P-hacking is a term used to describe various techniques that researchers can use to increase the chances of finding statistically significant results in their study, even if the results are not actually meaningful. This is a form of data manipulation that can lead to the publication of false positive results.

Eg.

In a study that compares the effectiveness of two different physiotherapy interventions for rotator cuff-related shoulder pain, a researcher may conduct multiple analyses of the data, only reporting the ones that show a statistically significant difference between the two interventions. This is p-hacking because the researcher is selectively reporting only the results that support their hypothesis while ignoring the ones that do not.

Multiple Unadjusted Comparisons

Another example of p-hacking in physiotherapy research is when a researcher conducts multiple comparisons between different interventions but does not adjust for these multiple comparisons. This can lead to an increased chance of finding statistically significant results by chance, even if the interventions are not truly different.

By selectively removing some of the data, the researcher is effectively “p-hacking” the results, as it increases the probability of finding a statistically significant difference between the two interventions, even if there isn’t any real difference. The researcher might then present the data as if it were a fair comparison, which can lead to inaccurate conclusions and mislead future research and treatment. This is just one of the ways to do this.

Unintentional

It’s important to note that P-hacking can occur unintentionally and can be the result of a researcher’s lack of understanding of statistics or the pressure to publish positive results. But it can also be a deliberate act, driven by a desire to achieve a specific outcome. To avoid p-hacking, researchers should pre-register their study design and analysis plan, report all the results, and use appropriate statistical methods to adjust for multiple comparisons. Also, the use of replication studies and independent replication of findings can help to increase confidence in the results.

More on Pre-Trial Registration

Pre-trial registration is the process of registering a clinical trial before it begins, which involves providing detailed information about the study design, interventions, outcomes, and analysis plan. This practice is considered essential for promoting transparency and reducing bias in clinical research. By publicly registering a study before it is conducted, researchers commit to a set of objectives and methods, which can help to prevent selective reporting of outcomes and data manipulation. Pre-trial registration also helps to avoid duplication of research efforts and improves access to information about ongoing and completed trials. Registered trials are more likely to be published, which can reduce publication bias and ensure that the results of the study are available for use in clinical practice and policy-making. In summary, pre-trial registration is an important step toward ensuring that clinical research is conducted in a transparent and rigorous manner, leading to more reliable and useful results.

References

Gelman, A., & Loken, E. (2019). The garden of forking paths : Why multiple comparisons can be a problem , even when there is no “ fishing expedition ” or “ p-hacking ” and the research hypothesis was posited ahead of time ∗.

Daniele Fanelli (2012). Negative results are disappearing from most disciplines and countries. , 90(3), 891–904. doi:10.1007/s11192-011-0494-7

Lakens, D. (2019). The value of preregistration for psychological science: A conceptual analysis. 心理学評論, 62(3), 221-230.

Storey, J. D. (2003). The positive false discovery rate: a Bayesian interpretation and the q-value. The annals of statistics, 31(6), 2013-2035.

Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., & van Aert, R. C. (2016). M., & van Assen, MALM (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7.