Category: Informatics

Attempting Decision-Support For tPA

As I’ve wondered many times before – given the theoretical narrow therapeutic window for tPA in stroke, paired with the heterogenous patient substrate and disease process – why do we consent all patients similarly?  Why do we not provide a more individualized risk/benefit prediction?

Part of the answer is derived from money & politics – there’s no profit in carefully selecting patients for an expensive therapy.  Another part of the answer is the reliability of the evidence base.  And, finally, the last part of the answer is the knowledge translation bit – how can physicians be expected to perform complex multivariate risk-stratification and communicate such information to a layperson in the acute setting?

In this paper, these authors describe the development process of an iPad application specifically designed for pictoral display of individualized risk/benefit for tPA administration in acute ischemic stroke.  Based on time from onset to treatment, age, gender, medical history, NIHSS, weight, and blood pressure, manual entry of these variables into the software provides individualized information regarding outcomes given treatment or non-treatment.

Unfortunately, the prediction instrument – S-TIPI – is based on: NINDS, ECASS II, and ATLANTIS.  Thus, as you might expect, in the most commonly used time frame of 0-3 hours, the outcomes essentially approximate NINDS.  The authors mention they used the UK portion of the Safe Implementation of Thrombolysis in Stroke database and the Virtual International Stroke Trials Archive to refine their calculations, but do not delve into a discussion of predictive accuracy.  Of note, a previous article describing recalibration of S-TIPI indicated an AUC for prediction of only 0.754 to 0.766 – but no such uncertainty, nor their narrowly derived limited data set, are described in this paper.

Regardless, such “precision medicine” decision instruments – for both this and other applications – are of great importance in guiding complex decision-making.  This paper is basically a “check out what we made” piece of literature by a group of authors who will sell you the end result as a product, but it is still an important effort from which to recognize and build.

“Development of a computerised decision aid for thrombolysis in acute stroke care”
http://www.ncbi.nlm.nih.gov/pubmed/25889696

A Window Into Your EHR Sepsis Alert

Hospitals are generally interested in detecting and treating sepsis.  As a result of multiple quality measures, however, now they are deeply in love with detecting and treating sepsis.  And this means: yet another alert in your electronic health record.

One of these alerts, created by the Cerner Corporation, is described in a recent publication in the American Journal of Medical Quality.  Their cloud-based system analyzes patient data in real-time as it enters the EHR and matches the data against the SIRS criteria.  Based on 6200 hospitalizations retrospectively reviewed, the alert fired for 817 (13%) of patients.  Of these, 622 (76%) were either superfluous or erroneous, with the alert occurring either after the clinician had ordered antibiotics or in patients for whom no infection was suspected or treated.  Of the remaining alerts occurring prior to action to treat or diagnose infection, most (89%) occurred in the Emergency Department, and a substantial number (34%) were erroneous.

Therefore, based on the authors’ presented data, 126 of 817 (15%) of SIRS alerts provided accurate, potentially valuable information.  Unfortunately, another 80 patients in the hospitalized cohort received discharge diagnoses of sepsis despite never triggering the tool – meaning false negatives approach nearly 2/3rds the number of potentially useful true positives.  And, finally, these data only describe patients requiring hospitalization – i.e., not including those discharged from the Emergency Department.  We can only speculate regarding the number of alerts triggered on the diverse ED population not requiring hospitalization – every asthmatic, minor trauma, pancreatitis, etc.

The lead author proudly concludes their tool is “an effective approach toward early recognition of sepsis in a hospital setting.”  Of course, the author, employed by Cerner, also declares he has no potential conflicts of interest regarding the publication in question.

So, if the definition of “effective” is lower than probably 10% utility, that is the performance you’re looking it with these SIRS-based tools.  Considering, on one hand, the alert fatigue, and on the other hand, the number of additional interventions and unnecessary tests these sorts of alerts bludgeon physicians into – such unsophisticated SIRS alerts are almost certainly more harm than good.

“Clinical Decision Support for Early Recognition of Sepsis”
http://www.ncbi.nlm.nih.gov/pubmed/25385815

Hi Ur Pt Has AKI For Totes

Do you enjoy receiving pop-up alerts from your electronic health record?  Have you instinctively memorized the fastest series of clicks to “Ignore”?  “Not Clinically Significant”?  “Benefit Outweighs Risk”?  “Not Sepsis”?

How would you like your EHR to call you at home with more of the same?

Acute kidney injury, to be certain, is associated with poorer outcomes in the hospital – mortality, dialysis-dependence, and other morbidities.  Therefore, it makes sense – if an automated monitoring system can easily detect changes and trends, why not alert clinicians to such changes, and nephrotoxic therapies could be avoided.

Interestingly – for both good and bad – the outcomes measured were patient-oriented, randomizing 2393 patients to either “usual care” or text message alerts for changes in serum creatinine.  The goal, overall, was detection of reductions in death, dialysis, or progressive AKI.  While patient-oriented outcomes are, after all, the most important outcomes in medicine – it’s only plausible to improve outcomes if clinicians improve care.  Therefore, measuring the most direct consequence of the intervention might be a better outcome – renal-protective changes in clinician behavior.

Because, unfortunately, despite sending text messages and e-mails directly to responsible clinicians and pharmacists – the only notable change in behavior between the “alert” group and “usual care group” was increased monitoring of serum creatinine.  Chart documentation of AKI, avoidance of intravenous contrast, avoidance of NSAIDs, and other renal-protective behaviors were unchanged, excepting a non-significant trend towards decreased aminoglycoside use.

No change in behavior, no change in outcomes.  Text messages and e-mails alerts!  Can shock collars be far behind?

“Automated, electronic alerts for acute kidney injury: a single-blind, parallel-group, randomised controlled trial”
http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(15)60266-5/fulltext

Social Media in Medicine – Useless!

Or, might it be how you use it that matters?

This is a brief report from the journal Circulation, regarding a self-assessment of their social media strategy.  The editors of the journal performed a prospective, block-randomization of published articles to either social media promotion on Facebook and Twitter, or no promotion, and compared 30-day website page views for each article.  121 articles were randomized to social media and 122 to control, and were generally evenly balanced between article types.

And, the answer – unfortunately, for their 3-person associate editor team – is: no difference.  Articles posted to social media received an average of 409 pageviews within 30-days, compared with 392 to those with no promotion.  Thus, the journal of Circulation declares social media dead – and ultimately generalizes their failures to all cardiovascular journals via their Conclusions section.

So, we should all stop blogging and tweeting?  Or, is journal self-promotion futile?  And, are page views the best measure of the effectiveness of knowledge translation?  Or, is there more nuance and heterogeneity between online strategies, rendering this Circulation data of only passing curiosity?  I tend to believe the latter – but, certainly, it’s an interesting publication I hope inspires other journals to perform their own, similarly rigorous studies.

[Note: if my blog entries receive as many (or more!) pageviews as Circulation articles, does this mean my impact factor is higher than Circulation’s 14.98?]

“A Randomized Trial of Social Media from Circulation”
http://circ.ahajournals.org/content/early/2014/11/17/CIRCULATIONAHA.114.013509.abstract

When Is An Alarm Not An Alarm?

What is the sound of one hand clapping?  If a tree falls in a forest, does it make a sound?  If a healthcare alarm is in no fashion alarming, what judgement ought we make of its existence?

The authors of this study, from UCSF, compose a beautiful, concise introduction to their study, which I will simply reproduce, rather than unimpressively paraphrase:

“Physiologic monitors are plagued with alarms that create a cacophony of sounds and visual alerts causing ‘alarm fatigue’ which creates an unsafe patient environment because a life-threatening event may be missed in this milieu of sensory overload.“

We all, intuitively, know this to be true.  Even the musical mating call of the ventilator, the “life support” of the critically ill, barely raises us from our chairs until such sounds become insistent and sustained.  But, these authors quantified such sounds – and look upon such numbers, ye Mighty, and despair:

2,558,760 alarms on 461 adults over a 31-day study period.

Most alarms – 1,154,201 of them – were due to monitor detection of “arrhythmias”, with the remainder split between vital sign parameters and other technical alarms.  These authors note, in efforts to combat alert fatigue, audible alerts were already restricted to those considered clinically important – which reduced the overall burden to a mere 381,050 audible alarms, or, only 187 audible alarms per bed per day.

Of course, this is the ICU – many of these audible alarms may, in fact, have represented true positives.  And, many did – nearly 60% of the ventricular fibrillation alarms were true positives.  However, next up was asystole at 33% true positives, and it just goes downhill from there – with a mere 3.3% of the 1,299 reviewed ventricular bradycardia alarms classified as true positives.

Dramatic redesign of healthcare alarms is clearly necessary as not to detract from high-quality care.  Physicians are obviously tuning out vast oceans of alerts, alarms, and reminders – and some of them might even be important.

“Insights into the Problem of Alarm Fatigue with Physiologic Monitor Devices: A Comprehensive Observational Study of Consecutive Intensive Care Unit Patients”
http://www.ncbi.nlm.nih.gov/pubmed/25338067

Clinical Informatics Exam Post-Mortem

I rarely break from literature review in my blog posts (although, I used to make the occasional post about Scotch).  However, there are probably enough folks out there in academia planning on taking this examination, or considering an Emergency Medicine Clinical Informatics fellowship – like the ones at Mt. Sinai, BIDMC, and Arizona – to make this diversion of passing interest to a few.

Today is the final day of the 2015 testing window, so everyone taking the test this year has already sat for it or is suffering through it at this moment.  Of course, I’m not going to reveal any specific questions, or talk about a special topic to review (hint, hint), but more my general impressions of the test – as someone who has taken a lot of tests.

The day started out well, as the Pearson Vue registration clerk made a nice comment that I’d gone bald since my picture at my last encounter with them, presumably for USMLE Step 3.  After divesting myself of Twitter-enabled devices, the standard computer-based multiple-choice testing commenced.

First of all, for those who aren’t aware, this is only the second time the American Board of Preventive Medicine has administered the Clinical Informatics board examination.  Furthermore, there are few – probably zero – clinicians currently taking this examination who have completed an ACGME Clinical Informatics fellowship.  They simply don’t exist.  Thus, there is a bit of a perfect storm in which none of us have undergone a specific training curriculum preparing us for this test, plus minimal hearsay/experience from folks who have taken the test, plus a test which is essentially still experimental.

Also, the majority (>90%) of folks taking the test use one of AMIA’s review courses – either the in-person session or the online course and assessment.  These courses step through the core content areas describe for the subspecialty of Clinical Informatics, and, in theory, review the necessary material to obtain a passing score.  After all, presumably, the folks at AMIA designed the subspecialty and wrote most of the questions – they ought to know how to prep for it, right?

Except, as you progress through the computer-based examination, you find the board review course has given you an apparently uselessly superficial overview of many topics.  Most of us taking the examination today, I assume, are current or former clinicians, with some sort of computer science background, and are part-time researchers in a subset of clinical informatics.  This sort of experience gets you about half the questions on the exam in the bag.  Then, about a quarter of the course – if you know every detail of what’s presented in the review course regarding certification organizations, standards terminologies, process change, and leadership – that’s another 50 out of 200 questions you can safely answer.  But, you will need to really have pointlessly memorized a pile of acronyms and their various relationships to get there.  Indeed, the use of acronyms is pervasive enough it’s almost as though their intention is more to weed out those who don’t know some secret handshake of Clinical Informatics, rather than truly assess your domain expertise.

The last quarter of the exam?  The ABPM study guide for the examination states 40% of the exam covers “Health Information Systems” and 20% covers “Leading and Managing Change”.  And, nearly every question I was trying to make useful guesses towards came from those two areas – and covered details either absent from or addressed in some passing vagueness in the AMIA study course.  And, probably as some consequence of this being one of the first administrations of this test, I wasn’t particularly impressed the questions – which were heavy on specific recall, and not hardly on application of knowledge or problem solving.  I’m not sure exactly what resources I’d use to study prior to retaking if I failed, but most of the difference would come down to just rote memorization.

However, because the pass rate was 92% last year, and nearly everyone taking the test used the AMIA course, an average examinee with the average preparation ought yet to be in good shape.  So, presumably, despite my distasteful experience overall – one likely shared by many – we’ll all receive passing scores.

Check back mid-December for the exciting conclusion to this tale.

Update (as noted in comments below):  Passed!

Hopefully future editions of prep courses will gradually attune themselves to the board content, once a few iterations have progressed.  Individuals taking this exam, in the meantime, will need to rely heavily on their medical or prior technical experience, particularly as the curricula for fellowships are fleshed out.  Additionally, the CI exam content is so broad, fellowship trainees will need to specifically target their coursework to areas they lack – for, example, “Leading and Managing Change” as a major content area of the examination will definitely force many informaticians into a knowledge gap.

Interesting times!

Using Patient-Similarity to Predict Pulmonary Embolism

Topological data analysis is one of the many “big data” buzzphrases being thrown about, with roots in non-parametric statistical analysis, and promoted by the Palo Alto startup, Ayasdi.  I’ve done a little experimentation with it, and used it mostly to show the underlying clustering and heterogeneity of the PECARN TBI data set.  My ultimate hypothesis, based on these findings, would be that patient-similarity is a more useful predictor of individual patient risk than the partition analysis used in the original PECARN model.  This technique is similar to the “attribute matching” demonstrated by Jeff Kline in Annals, but of much greater granularity and sophistication.

So, I should be excited to see this paper – using the TDA output to train a neural network classifier for suspected pulmonary embolism.  Using 152 patients, 101 of which were diagnosed with PE, the authors develop a topological network with clustered distributions of diseased and non-diseased individuals, and compare the output from this network to the Wells and Revised Geneva Scores.

The AUC for the neural network was 0.8911, for Wells was 0.74, and Revised Geneva was 0.55. And this sounds fabulous – until it’s noted the neural network is being derived and tested on the same, tiny sample.  There’s no validation set, and, given such a small sample, the likelihood of overfitting is substantial.  I expect performance will degrade substantially when applied to other data sets.

However, even simply as scientific curiosity – I hope to see further testing and refinement of potentially greater value.

“Using Topological Data Analysis for diagnosis pulmonary embolism”
http://arxiv.org/abs/1409.5020
http://www.ayasdi.com/_downloads/A_Data_Driven_Clinical_Predictive_Rule_CPR_for_Pulmonary_Embolism.pdf

How Electronic Health Records Sabotage Care

Our new information overlords bring many benefits to patient care.  No, really, they do.  I’m sure you can come up with one or two aspects of patient  safety improved by modern health information technology.  However, it’s been difficult to demonstrate benefits associated with electronic health records in terms of patient-oriented outcomes because, as we are all well aware, many EHRs inadvertently detract from efficient processes of care.

However, while we intuitively recognize the failings of EHRs, there is still work to be done in cataloguing these errors.  To that end, this study is a review of 100 consecutive closed patient safety investigations in the Veterans Health Administration relating to information technology.  The authors reviewed each case narrative in detail, and divided the errors up into sociotechnical classification of EHR implementation and use.  Unsurprisingly, the most common failures of EHRs are related to failures to provide the correct information in the correct context.  Following that, again, unsurprisingly, were simple software malfunctions and misbehaviors.  Full accounting and examples are provided in Table 2:

Yes, EHRs – the solution to, and cause of, all our problems.

“An analysis of electronic health record-related patient safety concerns”
http://jamia.bmj.com/content/early/2014/05/20/amiajnl-2013-002578.full

Build a New EDIS, Advertise it in Annals for Free

As everyone who has switched from paper to electronic charting and ordering has witnessed, despite some improvements, many processes became greatly more inefficient.  And – it doesn’t matter which Emergency Department information system you use.  Each vendor has its own special liabilities.  Standalone vendors have interoperability issues.  Integrated systems appear to have been designed as an afterthought to the inpatient system.  We have, begrudgingly, learned to tolerate our new electronic masters.

This study, in Annals of Emergency Medicine, describes the efforts of three authors to design an alternative to one of the vendor systems:  Cerner’s FirstNet product.  I have used this product.  I feel their pain.  And, I am in no way surprised these authors are able to design alternative, custom workflows that are faster (as measured in seconds) and more efficient (as measured in clicks) for their prototype system.  It is, essentially, a straw man comparator – as any thoughtful, user-centric, iterative design process could improve upon the current state of most EDIS.

With the outcome never in doubt, the results demonstrated are fundamentally unremarkable and of little scientific value.  And, it finally all makes sense as the recurrent same sad refrain rears its ugly head in the conflict-of-interest declaration:

Dr. Patrick and Mr. Besiso are employees of iCIMS, which is marketing the methodology described in this article.

Cheers to Annals for enabling these authors to use the pages of this journal as a vehicle to sell their consulting service.

“Efficiency Achievements From a User-Developed Real-Time Modifiable Clinical Information System”
http://www.ncbi.nlm.nih.gov/pubmed/24997563

Shared Decision-Making to Reduce Overtesting

Medicine, like life, is full of uncertainty.  Every action or inaction has costs and consequences, both anticipated and unintended.  Permeating through medical culture for many reasons, with the proliferation of tests available, has been a decreased tolerance for this uncertainty and the rise of “zero-miss” medicine.  However, there are some tests that carry with them enough cost and risk, the population harms of the test outweigh the harms of the missed diagnoses.  CTPA for pulmonary embolism is one of those tests.

In this study, these authors attempt to reduce testing for pulmonary embolism by creating a shared decision-making framework to discuss the necessity of testing with patients.  They prospectively enrolled 203 patients presenting to the Emergency Department with dyspnea and, independent of their actual medical evaluation, attempted to ascertain their hypothetical actions were they to be evaluated for PE.  Specifically, they were interested in the “low clinical probability” population whose d-Dimer was elevated above the abnormal threshold – but still below twice-normal the threshold.  For these “borderline” abnormal d-Dimers, the authors created a visual decision tool describing their estimate of the benefit and risk of undergoing CTPA given this specific clinical scenario.

After viewing the benefits and risks of CTPA, 36% of patients in this study stated they would hypothetically decline testing for PE.  Most of the patients (85%) who planned to follow-through with the CTPA did so because they were concerned regarding a possible missed diagnosis of PE, while the remaining hoped the CT would at least also provide additional information regarding their actual diagnosis.  The authors conclude, based on a base case of 2.6 million possible PE evaluations annually, this strategy might save 100,000 CTPAs.

I think the approach these authors promote is generally on the right track.  The challenge, however, is the data used to discuss risks with patients.  From their information graphics, the risks of CTPA – cancer, IV contrast reaction, kidney injury and false positives – are all fair to include, but can be argued greatly regarding their clinical relevance.  Is a transient 25% increase in serum creatinine in a young, healthy person clinically significant?  Is it the same as a cancer diagnosis?  Is it enough to mention there are false-positives from the CTPA without mentioning the risk of having a severe bleeding event from anticoagulation?  Then, in their risk of not having the CTPA information graphic, they devote the bulk of that risk to a 15% chance of the CT identifying a diagnosis that would have otherwise been missed.  I think that significantly overstates the number of additional, clinically important findings requiring urgent treatment that might be identified.  Finally, the risks presented are for the “average” patient – and may be entirely inaccurate across the heterogenous population presenting for dyspnea.

But, any quibbles over the information graphic, limitations, and magnitude of effect are outweighed by the importance of advancing this approach in our practice.  Paternalism is dead, and new tools for communicating with patients will be critical to the future of medicine.

“Patient preferences for testing for pulmonary embolism in the ED using a shared decision-making model”
http://www.ncbi.nlm.nih.gov/pubmed/24370071