{"id":14228,"date":"2023-01-25T09:01:19","date_gmt":"2023-01-25T08:01:19","guid":{"rendered":"https:\/\/www.physiotutors.com\/?post_type=wiki&#038;p=14228"},"modified":"2026-10-15T19:09:50","modified_gmt":"2026-10-15T17:09:50","slug":"multiple-comparison-problem-statistics","status":"publish","type":"wiki","link":"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/","title":{"rendered":"Multiple Comparison Problem"},"featured_media":15379,"template":"","meta":{"_acf_changed":false,"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":""},"class_list":["post-14228","wiki","type-wiki","status-publish","has-post-thumbnail","hentry"],"acf":{"free_access":"","app_link":"","sections":[{"acf_fc_layout":"page-header-detail-page","background":{"background_image":15379,"background_image_alt_text":"What's the Multiple Comparison Problem?"},"heading":"What's the Multiple Comparison Problem? | Statistics","subheading":"Stats & Methodology","content_editor":"","button_type":"internal_link","button_label":"Check our shop","button_style":"button-five","button_internal_link":4646,"button_anchor":"","button_url":"","button_file":null,"label":""},{"acf_fc_layout":"submenu","links":[{"button_type":"anchor","button_label":"Learn","button_internal_link":null,"button_anchor":"section-number-3","button_url":"","button_file":null},{"button_type":"anchor","button_label":"Free sample","button_internal_link":null,"button_anchor":"section-number-5","button_url":"","button_file":null},{"button_type":"anchor","button_label":"Assessment E-Book","button_internal_link":null,"button_anchor":"section-number-6","button_url":"","button_file":null}],"dropdown_label":"Go to","button_right":{"button_type":"internal_link","button_label":"Try Premium free","button_style":"button-three","button_internal_link":26802,"button_anchor":"","button_url":"","button_file":""},"button_description":""},{"acf_fc_layout":"content-content","section_options":{"vertical_alignment":"top","column_right_box":false,"column_left_width":"col-12 col-lg-4 offset"},"column_left":{"content_editor":"<h4>Learn<\/h4>"},"column_right":{"content_editor":"<h6>What's the Multiple Comparison Problem? | Statistics<\/h2>\nThe multiple comparison problem is the issue that arises when multiple tests on the same sample are performed. An example will illustrate this.\n\n<em>Eg.<\/em>\n\n<em>Let\u2019s say that a study looks at prospective risk factors for running injuries in 5000 novice runners. Different variables are tested, since we do not yet know which ones will increase the risk. Examples are: running volume, navicular drop, q-angle, quad and glute strength, heel vs forefoot strike pattern, minimalist vs maximalist shoe, and ankle dorsiflexion ROM.<\/em>\n\n&nbsp;\n<h6>False positives with multiple comparison<\/h3>\nMost researchers will accept a 5% false positive rate, the <a href=\"https:\/\/www.physiotutors.com\/wiki\/alpha-level\/\" target=\"_blank\" rel=\"noopener\">alpha or significance level<\/a>. This is for a given variable like quadriceps strength. It means that if this study is conducted one hundred times, about 5 studies will show a <a href=\"https:\/\/www.physiotutors.com\/wiki\/type-1-errors\/\" target=\"_blank\" rel=\"noopener\">false positive result<\/a>, when in fact, there is none.\n\nHowever, the researchers are looking at ten variables, not just quad strength; within the same sample. This poses a problem.\n\nThe researchers, unbeknownst to this problem, conduct the trial. Two years later the data comes in, showing a heel strike pattern and glute strength to be a risk factor for a running injury. Great! That\u2019s the conclusion and the paper gets published.\n\nAs noted before, the significance level at 5% does not mean there is a 5% false positive rate at this point due to the plethora of different variables that are being researched. So the researchers implicitly accepted a much greater risk of false positive results by conducting the trial, looking at ten variables.\n\nThe family-wise error rate demonstrates this. With a quite simple calculation, we can check the false positive rate, it is 40%! The formula is shown below.\n<h6>Solutions to the multiple comparison problem<\/h3>\nI think we can agree that this forms a problem. So what are we going to do about it? There is a solution. Researchers can make corrections to counteract this alpha-inflation by doing a Bonferroni or Holm correction. This is discussed in \u201c<a href=\"https:\/\/www.physiotutors.com\/wiki\/type-1-error-rate-control\/\" target=\"_blank\" rel=\"noopener\">Type 1 error rate control<\/a>\u201d.\n\nFamily-wise error rate formula:\n\n1 - (1 - \u0251)x\n\n\u0251: alpha or significance level in decimals\n\nx: number of tests\n<h6>Type II Errors<\/h3>\nHowever, adjusting the significance level of each individual test can increase the probability of making a <a href=\"https:\/\/www.physiotutors.com\/wiki\/type-2-errors\/\" target=\"_blank\" rel=\"noopener\">Type II error (false negative)<\/a> across all the tests. This is because the more stringent significance level reduces the power of each individual test to detect a true effect or relationship. Consequently, a significant effect may be missed in some tests, leading to <a href=\"https:\/\/www.physiotutors.com\/wiki\/type-2-errors\/\" target=\"_blank\" rel=\"noopener\">false negative results<\/a>. To avoid false negative results due to the multiple comparison problem, we can use techniques such as pre-registration of hypotheses, replication studies, or more powerful statistical methods such as Bayesian inference. Additionally, it is important to carefully design the study and the hypotheses being tested to minimize the number of tests conducted and ensure that they are meaningful and relevant to the research question.\n"}},{"acf_fc_layout":"content-content","section_options":{"vertical_alignment":"top","column_right_box":false,"column_left_width":"col-12 col-lg-4 offset"},"column_left":{"content_editor":"<h4>References<\/h4>"},"column_right":{"content_editor":"<a href=\"https:\/\/onlinelibrary.wiley.com\/doi\/full\/10.1002\/pst.2059\" target=\"_blank\" rel=\"noopener\">Wason, J. M. S., &amp; Robertson, D. S. (2021). Controlling type I error rates in multi-arm clinical trials: A case for the false discovery rate. Pharmaceutical statistics, 20(1), 109\u2013116.<\/a>\n\n<a href=\"https:\/\/www.degruyter.com\/document\/doi\/10.2202\/1544-6115.1040\/html\" target=\"_blank\" rel=\"noopener\">Dudoit, S., van der Laan, M. J., &amp; Pollard, K. S. (2004). Multiple testing. Part I. Single-step procedures for control of general type I error rates. Statistical applications in genetics and molecular biology, 3, Article13.<\/a>\n\n<a href=\"https:\/\/www.youtube.com\/watch?v=MbrocgM_4sI&amp;t=478s\" target=\"_blank\" rel=\"noopener\">Lakens, D., Type 1 error control by Daniel Lakens, youtube<\/a>\n\n<a href=\"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/j.1440-1681.1998.tb02179.x\" target=\"_blank\" rel=\"noopener\">John Ludbrook (1998). MULTIPLE COMPARISON PROCEDURES UPDATED. , 25(12), 1032\u20131037. doi:10.1111\/j.1440-1681.1998.tb02179.x<\/a>"}},{"acf_fc_layout":"cta-form-v2","content_editor":"<h2>Get the <strong>21 most useful<\/strong> orthopedic tests<\/h2>\r\n<p>Download our <strong>free<\/strong> e-book with the 21 most useful orthopedic tests per body region, to help you reach the right diagnosis.<\/p>","form":"39","image":29349,"image_alt_text":"Free e-book with 21 orthopedic tests"},{"acf_fc_layout":"cta-image-v2","layout":"","background":"transparent","heading":"","content_editor":"<p class=\"subheading\">Like what you\u2019re learning?<\/p>\n<p class=\"heading-two\">Get the full Physiotutors <strong>Assessment Book<\/strong><\/p>\n\n<ul>\n \t<li>600+ pages on orthopedic assessment<\/li>\n \t<li>Video demonstrations and direct links to the PubMed studies<\/li>\n \t<li>Statistical values for every special test, from the latest research<\/li>\n \t<li>Free lifetime updates<\/li>\n \t<li>Available in English, German, French, Spanish, Italian, Portuguese and Turkish<\/li>\n<\/ul>","button_type":"internal_link","button_label":"See the Assessment Book","button_style":"button-two","button_internal_link":4646,"button_anchor":"","button_url":"","button_file":null,"side_button":{"side_button_content_editor":""},"bottom_button":{"bottom_button_content_editor":""},"image":22705,"image_alt_text":""}]},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>The Multiple Comparison Problem Explained | Statistics<\/title>\n<meta name=\"description\" content=\"Researchers can&#039;t test whatever they want with their dataset. There are issues that arise when doing so. Learn here why.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The Multiple Comparison Problem Explained | Statistics\" \/>\n<meta property=\"og:description\" content=\"Researchers can&#039;t test whatever they want with their dataset. There are issues that arise when doing so. Learn here why.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/\" \/>\n<meta property=\"og:site_name\" content=\"Physiotutors\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/physiotutors\" \/>\n<meta property=\"article:modified_time\" content=\"2026-10-15T17:09:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.physiotutors.com\/wp-content\/uploads\/2023\/01\/statistics.png\" \/>\n\t<meta property=\"og:image:width\" content=\"560\" \/>\n\t<meta property=\"og:image:height\" content=\"466\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@physiotutors\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/wiki\\\/multiple-comparison-problem-statistics\\\/\",\"url\":\"https:\\\/\\\/www.physiotutors.com\\\/wiki\\\/multiple-comparison-problem-statistics\\\/\",\"name\":\"The Multiple Comparison Problem Explained | Statistics\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/wiki\\\/multiple-comparison-problem-statistics\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/wiki\\\/multiple-comparison-problem-statistics\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.physiotutors.com\\\/wp-content\\\/uploads\\\/2023\\\/01\\\/statistics.png\",\"datePublished\":\"2023-01-25T08:01:19+00:00\",\"dateModified\":\"2026-10-15T17:09:50+00:00\",\"description\":\"Researchers can't test whatever they want with their dataset. There are issues that arise when doing so. Learn here why.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/wiki\\\/multiple-comparison-problem-statistics\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.physiotutors.com\\\/wiki\\\/multiple-comparison-problem-statistics\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/wiki\\\/multiple-comparison-problem-statistics\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.physiotutors.com\\\/wp-content\\\/uploads\\\/2023\\\/01\\\/statistics.png\",\"contentUrl\":\"https:\\\/\\\/www.physiotutors.com\\\/wp-content\\\/uploads\\\/2023\\\/01\\\/statistics.png\",\"width\":560,\"height\":466,\"caption\":\"Statistics\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/wiki\\\/multiple-comparison-problem-statistics\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.physiotutors.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Multiple Comparison Problem\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/#website\",\"url\":\"https:\\\/\\\/www.physiotutors.com\\\/\",\"name\":\"Physiotutors\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.physiotutors.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/#organization\",\"name\":\"Physiotutors\",\"url\":\"https:\\\/\\\/www.physiotutors.com\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.physiotutors.com\\\/wp-content\\\/uploads\\\/2021\\\/06\\\/PHY_Logo-Purple-2.svg\",\"contentUrl\":\"https:\\\/\\\/www.physiotutors.com\\\/wp-content\\\/uploads\\\/2021\\\/06\\\/PHY_Logo-Purple-2.svg\",\"width\":315,\"height\":65,\"caption\":\"Physiotutors\"},\"image\":{\"@id\":\"https:\\\/\\\/www.physiotutors.com\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/facebook.com\\\/physiotutors\",\"https:\\\/\\\/x.com\\\/physiotutors\",\"https:\\\/\\\/instagram.com\\\/physiotutors\",\"https:\\\/\\\/youtube.com\\\/physiotutors\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The Multiple Comparison Problem Explained | Statistics","description":"Researchers can't test whatever they want with their dataset. There are issues that arise when doing so. Learn here why.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/","og_locale":"en_US","og_type":"article","og_title":"The Multiple Comparison Problem Explained | Statistics","og_description":"Researchers can't test whatever they want with their dataset. There are issues that arise when doing so. Learn here why.","og_url":"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/","og_site_name":"Physiotutors","article_publisher":"https:\/\/facebook.com\/physiotutors","article_modified_time":"2026-10-15T17:09:50+00:00","og_image":[{"width":560,"height":466,"url":"https:\/\/www.physiotutors.com\/wp-content\/uploads\/2023\/01\/statistics.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_site":"@physiotutors","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/","url":"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/","name":"The Multiple Comparison Problem Explained | Statistics","isPartOf":{"@id":"https:\/\/www.physiotutors.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/#primaryimage"},"image":{"@id":"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/#primaryimage"},"thumbnailUrl":"https:\/\/www.physiotutors.com\/wp-content\/uploads\/2023\/01\/statistics.png","datePublished":"2023-01-25T08:01:19+00:00","dateModified":"2026-10-15T17:09:50+00:00","description":"Researchers can't test whatever they want with their dataset. There are issues that arise when doing so. Learn here why.","breadcrumb":{"@id":"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/#primaryimage","url":"https:\/\/www.physiotutors.com\/wp-content\/uploads\/2023\/01\/statistics.png","contentUrl":"https:\/\/www.physiotutors.com\/wp-content\/uploads\/2023\/01\/statistics.png","width":560,"height":466,"caption":"Statistics"},{"@type":"BreadcrumbList","@id":"https:\/\/www.physiotutors.com\/wiki\/multiple-comparison-problem-statistics\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.physiotutors.com\/"},{"@type":"ListItem","position":2,"name":"Multiple Comparison Problem"}]},{"@type":"WebSite","@id":"https:\/\/www.physiotutors.com\/#website","url":"https:\/\/www.physiotutors.com\/","name":"Physiotutors","description":"","publisher":{"@id":"https:\/\/www.physiotutors.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.physiotutors.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.physiotutors.com\/#organization","name":"Physiotutors","url":"https:\/\/www.physiotutors.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.physiotutors.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.physiotutors.com\/wp-content\/uploads\/2021\/06\/PHY_Logo-Purple-2.svg","contentUrl":"https:\/\/www.physiotutors.com\/wp-content\/uploads\/2021\/06\/PHY_Logo-Purple-2.svg","width":315,"height":65,"caption":"Physiotutors"},"image":{"@id":"https:\/\/www.physiotutors.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/facebook.com\/physiotutors","https:\/\/x.com\/physiotutors","https:\/\/instagram.com\/physiotutors","https:\/\/youtube.com\/physiotutors"]}]}},"_links":{"self":[{"href":"https:\/\/www.physiotutors.com\/wp-json\/wp\/v2\/wiki\/14228","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.physiotutors.com\/wp-json\/wp\/v2\/wiki"}],"about":[{"href":"https:\/\/www.physiotutors.com\/wp-json\/wp\/v2\/types\/wiki"}],"version-history":[{"count":1,"href":"https:\/\/www.physiotutors.com\/wp-json\/wp\/v2\/wiki\/14228\/revisions"}],"predecessor-version":[{"id":29798,"href":"https:\/\/www.physiotutors.com\/wp-json\/wp\/v2\/wiki\/14228\/revisions\/29798"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.physiotutors.com\/wp-json\/wp\/v2\/media\/15379"}],"wp:attachment":[{"href":"https:\/\/www.physiotutors.com\/wp-json\/wp\/v2\/media?parent=14228"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}