This is a rather dangerous article for many reasons. Firstly, it’s published in a high-impact journal and received a fair bit of coverage in the news media. Secondly, it concludes its discussion by suggesting this ought to be adopted as a standardized rule for the evaluation of acute headache – this isn’t just a descriptive study on features of subarachnoid hemorrhage, it’s been given an official-sounding title, the “Ottawa SAH Rule”. Because of this, there’s significant potential for the rule described here to be adopted as widespread practice.
Therefore – it better be nearly perfect.
This is a prospective cohort from 10 university-affiliated Canadian hospitals. They looked at non-traumatic headaches reaching maximal intensity within 1 hour not part of a recurrent headache syndrome, and found 132 patients with SAH out of 2,131 assessed. They specifically gathered information on three previously-derived prediction rules and found none of them were 100% sensitive – so they chose the required elements from each to reach 100% (95% CI 97.2-100) sensitivity. The cost of this 100% sensitivity? Degeneration of specificity from the 28-35% in the three individual rules down to 15% (95% CI 13.8-16.9) in the final rule. The authors observed application of the derived rule would have decreased investigations for SAH from 84% of the enrolled cohort down to 74% of the enrolled cohort, and thusly their rule is superior to routine clinical practice by maintaining 100% sensitivity while decreasing resource utilization.
I think their inclusion criteria are fine – a rapid-onset, severe, atraumatic headache is the classical population of interest. Patients without this feature have such a low incidence of SAH that it’s unreasonable to evaluate for it. Their outcome measures, unfortunately, were a little softer. The positive diagnoses are reasonable – CT proven SAH or positive lumbar puncture with a source feature on cerebral angiography. However, only 82% underwent CT and 39% underwent LP, with a six month telephone follow-up – and a small number were lost to follow-up. Many would argue that CT alone is not sufficient for ruling out SAH, and 6-month survival is a limited proxy. This weakens its claim for 100% sensitivity.
Then, of course, a 15% specificity is awful. This isn’t necessarily a criticism of the authors, but more a recognition of the limitation of distilling diverse clinical data into concise decision instruments. 19 different patient features were significantly different between the SAH and no-SAH groups; reducing this to just 6 features discards so much information that an instrument designed for a complex clinical prediction is bound to fail. There were 1,694 false positives by the rule compared with 132 true positives. If this rule is applied without the strict exclusion criteria specified in the publication, there may be a huge number of inappropriate investigations.
Then, the rate of investigation comparison is also probably invalid. These institutions underwent specific 1-hour orientation to the study being performed and were actively involved in gathering clinical data for the study. I’m certain the 84% rate of investigation observed was conflated by the ongoing research at hand. The previous Perry study from 2011 had an evaluation rate of 57%, so it’s hard for me to believe the statistics from the current publication.
Finally, the kappa values for inter-observer agreement were rather mixed. Based on only sixty cases where two physicians evaluated the same patient, four of the six final rule elements had kappas between 0.44 and 0.59, representing only moderate agreement. This is a significant threat to the internal validity of the underlying data in support of their rule.
Overall, yes – the elements they identify through their observational cohort are likely to capture most cases of SAH. However, the limitations of this study and the poor specificity make me reluctant to buy in completely – and certainly not adopt it as a standardized “rule”.
“Clinical Decision Rules to Rule Out Subarachnoid Hemorrhage for Acute Headache”
http://www.ncbi.nlm.nih.gov/pubmed/24065011
Your interpretation is dangerous. The purpose of this clinical decision rule is to rule out, not rule in. You are correct, with such a low specificity, the posterior probability of a SAH with a positive screen based on the Ottawa SAH rule would be extraordinarily low. The purpose of this clinical prediction rule is to rapidly triage patients presenting with atraumatic headaches based on a targeted history and physical to identify patients where further diagnostic testing would be futile. With a reported 100% sensitivity, this rule would eliminate unnecessary CT scans, LPs, and longer ED visits. If the screen is positive, it would not mean to automatically obtain a CT (followed by LP if negative), but to use clinical judgment per usual care. You are correct to point out that the outcome was not ascertained completely in the study cohort. I would argue with you with adequate follow-up (which this study had), patients who "survived" the no-LP time period without morbidity/mortality did not have a clinically meaningful SAH. The modest kappa scores are of some concern for validity of reviewer ascertainment and this rule should be tested further in other clinical settings prior to widespread use. But this holds promise, and based on the criteria in the prediction rule, many clinicians unknowingly already incorporate these history & physical findings into their assessment (posttest probability) anyways. Cheers.
Your interpretation is dangerous. The purpose of this clinical decision rule is to rule out, not rule in. You are correct, with such a low specificity, the posterior probability of a SAH with a positive screen based on the Ottawa SAH rule would be extraordinarily low. The purpose of this clinical prediction rule is to rapidly triage patients presenting with atraumatic headaches based on a targeted history and physical to identify patients where further diagnostic testing would be futile. With a reported 100% sensitivity, this rule would eliminate unnecessary CT scans, LPs, and longer ED visits. If the screen is positive, it would not mean to automatically obtain a CT (followed by LP if negative), but to use clinical judgment per usual care. You are correct to point out that the outcome was not ascertained completely in the study cohort. I would argue with you with adequate follow-up (which this study had), patients who "survived" the no-LP time period without morbidity/mortality did not have a clinically meaningful SAH. The modest kappa scores are of some concern for validity of reviewer ascertainment and this rule should be tested further in other clinical settings prior to widespread use. But this holds promise, and based on the criteria in the prediction rule, many clinicians unknowingly already incorporate these history & physical findings into their assessment (posttest probability) anyways. Cheers.
I appreciate the post! I most of what you say is also a very reasonable interpretation of the study, but one I'm not entirely ready for. The 100% sensitivity is certainly a laudable goal – but not if more patients are harmed by testing than saved by the rule-out. My fear, as has been demonstrated with other rules, is clinicians will automatically use rule-in as CT->LP, and not use clinical judgement as usual care. Utilizing this as a one-way or a two-way decision rule is a question addressed in this lovely editorial I covered back in April:
https://www.emlitofnote.com/2013/04/how-to-evaluate-decision-instruments.html
I appreciate the post! I most of what you say is also a very reasonable interpretation of the study, but one I'm not entirely ready for. The 100% sensitivity is certainly a laudable goal – but not if more patients are harmed by testing than saved by the rule-out. My fear, as has been demonstrated with other rules, is clinicians will automatically use rule-in as CT->LP, and not use clinical judgement as usual care. Utilizing this as a one-way or a two-way decision rule is a question addressed in this lovely editorial I covered back in April:
https://www.emlitofnote.com/2013/04/how-to-evaluate-decision-instruments.html