The Ottawa ankle rules are the classic example of how great a simple clinical prediction rule can improve clinical practice. However, there are many challenges and barriers to why you should be careful using clinical prediction rules in practice. In this blog article, we’ll discuss which problems CPRs are facing:
Clinical prediction rules (CPRs) are mathematical tools that are intended to guide clinicians in their everyday decision-making. CPRs are created using multivariate statistical methods, are designed to examine the predictive ability of selected groupings of clinical variables. Clinical prediction rules may best be classified into three distinct groups: diagnostic, prognostic, and prescriptive. Studies that focus on predictive factors related to a specific diagnosis are known as diagnostic CPRs. Clinical prediction rules that are designed to predict an outcome such as success or failure are considered prognostic. Clinical prediction rules designed to target the most effective interventions are identified as prescriptive. Their advantage is that they can help clinicians to make quick decisions that may normally be subject to underlying biases.
An example of a predictive CPR, which we will refer to in this video is the CPR of Flynn et al. (2002) for the success of lumbar manipulation: If 3 or more of the following 5 items are present, the likelihood for success with manipulation is increased by a factor of 2.6, with 4+ or more with a factor of 24. These factors are: no symptoms distal to the knee, onset of symptoms shorter than 30 days, a FABQ score <19, hypomobility of the lumbar spine, and hip internal rotation of more than 35 degrees in at least one hip.
CPRs have to undergo 3 stages prior to full implementation in a clinical setting:
- Derivation: In this phase CPRs are derived using multivariate statistical methods to examine the predictive ability of selected groupings of clinical variables.
- Validation: The CPR is tested in a similar clinical setting (which is called internal validation), then the CPR is tested then different clinical setting (which is called external validation)
- Impact: Measurement of the usefulness of the rule in the clinical setting in terms of cost-benefit, patient satisfaction, time/resource allocation, etc. usually tested in randomized controlled trials
The last step would be the Implementation phase in which widespread acceptance and adoption of the rule in clinical practice are achieved.
Out of 434 clinical prediction rules only 54,8% have been validated and merely 2.8% had undergone impact analysis
Keogh et al. (2014) have found 434 individual rules up until the year 2014. Only 54,8% of them had been validated and merely 2.8% had undergone impact analysis. Most of the studies were conducted in the domain of cardiovascular and respiratory diseases followed by the musculoskeletal domain.
So the first caveat here is that although there are a lot of CPRs, many of them have not been validated, let alone have undergone impact studies and we can thus not say if using them will improve clinical practice. The CPR of Flynn is one of the few predictive CPRs we know that have been successfully validated two years later by Childs et al. (2004) in a randomized controlled trial. They found that the odds of a successful outcome of patients who were positive on the CPR with 4 out of 5 items compared to patients who were negative on the rule and received exercise was 60.8.
Like the CPR of Flynn and colleagues, most clinical prediction rules used in musculoskeletal practice are predictive CPRS. Those CPRs use baseline criteria called treatment effect modifiers which are gathered from a physical examination to inform the type of treatment that a patient should preferentially receive. Unfortunately, there are other potential pitfalls of clinical predictions rules, which Haskins and Cook (2016) pointed out in an editorial for the BJSM:
- A lot of simple, derivative modeling methods used by in many studies capture prognostic factors, rather than prescriptive factors. In other words, the rules identified patients who were going to improve anyway, regardless of the treatment they received. If we take the CPR of Flynn again, a duration of symptoms less than 30 days or no symptoms distal to the knee and a low level of fear avoidance are general positive prognostic factors who favor recovery independent of the treatment. In reality, the natural history associated with those signs and symptoms is very favorable, meaning that the improvement will not be associated with the care received, but with time.
- Many prescriptive CPRs contain non-modifiable factors such as age, gender or duration of symptoms that cannot be changed by treatment. To maximize the potential of the model the predictors should be mediating factors that can be influenced by treatment such as fear, catastrophizing, strength loss or flexibility
- Another important point is that factors included in the model should have high reliability. In case of the Flynn CPR, one factor in the model is “hypomobility of the lumbar spine”. A systematic review of van Trijffel et al. (2005) however, has shown that inter-rater reliability in the lumbar spine is only poor to fair. This will make it difficult for different raters using the CPR to use the same conclusion on this item.
- Most CPRs are underpowered due to insufficient sample sizes which lead to extremely wide confidence intervals indicating a lack of precision of a CPR’s predictive accuracy. In Flynn’s study we have a 95% confidence interval running from 4.63 to 139.41 in case of 4 or more positive items. So the effect of the manipulation in patients who score positive on the CPR can be moderate, but in can also be huge with and odds ratio of 139 in 95 out of 100 cases.
Alright, let’s summarize the reasons why we should not blindly rely on CPRs for clinical practice: Most of the CPRs have only been derived, but never (successfully) validated, let alone have reached clinical impact phase. The outcomes in one study and a particular setting cannot be simply transferred to your clinical setting. A lot of factors in a CPR are positive prognostic factors that are associated with a favorable natural course. So those patients were going to improve anyways. At last, it’s important that reliable and modifiable factors are included in a model to maximize its potential, while studies should increase their sample sizes in order to describe the effect of the CPR with higher precision.
The Manual Therapy App (iOS and Android)
The Manual Therapy App by Physiotutors houses over 150 mobilization and manipulation techniques for the musculoskeletal system. Furthermore you’ll find info on screening tests for the vascular system and ligamentous integrity.
If you want to stay up to date and get notified as soon as we are releasing a new blog article, make sure to subscribe to our RSS feed which will send you the new article notification right to your inbox.
References:
Adams ST, Leveson SH. Clinical prediction rules. Bmj. 2012 Jan 16;344:d8312.
Cook C. Potential pitfalls of clinical prediction rules.
Chad Cook’s Blog Article: https://relief.news/2016/09/05/rip-prescriptive-clinical-prediction-rules/
Wallace E, Johansen ME. Clinical prediction rules: challenges, barriers, and promise.
