Category: Informatics

Better Living Through Better Prediction

It’s Minority Report again – but pre-health, instead of pre-crime.

This is work from Sinai, where they are applying unsupervised machine learning techniques to generate intelligence from the electronic health record.  In less technical terms, they’re building SkyNet – for good.

These authors selected patients with at least five records in their data warehouse in the years leading up to 2013 as their modeling cohort.  Based on these ~700,000 patients, they abstracted all encounters, diagnoses, clinical notes, medications, and structured order data.  These various data types were then pared down to approximately 41,000 features that neither appeared in >80% of the records nor in fewer than five records, and then these features were normalized for analysis.  The novelty in their approach was their specific unsupervised data abstraction, reducing each patient to a dense vector of 500 features.  They then selected patients with at least one new ICD-9 diagnosis recored in their EHR in 2014, and divided them as their validation and test cohorts for disease prediction.

The results varied by diagnosis, but, most importantly, demonstrated their method appears superior to several other methods of abstraction – a non-abstracted “raw features” analysis, principal component analysis, Gaussian mixture model, k-means, and independent component analysis.  Using a random forest model for prediction, their abstraction method – “DeepPatient” – provided the best substrate for future diagnoses.  For example, their method worked best on “diabetes mellitus with complications”, providing an AUC for this diagnosis of 0.907.  Other high-scoring disease predictions including various cancers, cardiovascular disorders, and mental health issues.

Much work remains to be completed before similar technology is applicable in a practical clinical context.  This application does not even specifically account for the actual value of lab tests, only prediction of outcomes based on the co-occurence of other clinical features with a lab test result present.  Prediction strength also varied greatly by disease process; it is likely a more restricted or lightly supervised model will outperform their generic unsupervised general model with regard to specific short-term outcomes relating to emergency care.  And, of course, even when such models are being developed, they will still require testing and practice refinement regarding the traditional challenges balancing accuracy, risk tolerance, and resource utilization.

“Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records”

Changing Clinician Behavior For Low-Value Care

I’ve reported in general terms several times regarding, essentially, the shameful rate of inappropriate antibiotic prescribing for upper respiratory infections.  Choosing Wisely says: stop!  However, aggregated data seems to indicate the effect of Choosing Wisely has been minimal.

This study, from JAMA, is a prospective, cluster-randomized trial of multiple interventions in primary care practices aimed at decreasing inappropriate antibiotic use.  All clinicians received education on inappropriate antibiotic prescribing.  Then, practices and participating clinicians were randomized either to electronic health record interventions of “alternative suggestion” or “accountable justification”, to peer comparisons, or combinations of all three.

The short answer: it all works.  The complicated answer: so did the control intervention.  The baseline rate of inappropriate antibiotic prescribing in the control practices was estimated at 37.1%.  This dropped to 24.0% in the post-intervention period, and reflected a roughly linear constant downward trend throughout the study period.  However, each different intervention, singly and in combination, resulted in a much more pronounced drop in inappropriate prescribing.  While inappropriate prescribing in the control practices had reached mid-teens by the end of the study period, each intervention group was approaching a floor-level in the single digits.  Regarding safety interventions, only one of the seven intervention practice clusters had a significantly higher 30-day revisit rate than control.

While this study describes an intervention for antibiotic prescribing, the basic principles are sound regarding all manner of change management.  Education, as a foundation, paired with decision-support and performance feedback, as shown here, is an effective strategy to influence behavioral change.  These findings are of critical importance as our new healthcare economy continues to mature from a fee-for-service free-for-all to a value-based care collaboration.

“Effect of Behavioral Interventions on Inappropriate Antibiotic Prescribing Among Primary Care Practices”
http://www.ncbi.nlm.nih.gov/pubmed/26864410

Informatics Trek III: The Search For Sepsis

Big data!  It’s all the rage with tweens these days.  Hoverboards, Yik Yak, and predictive analytics are all kids talk about now.

This “big data” application, more specifically, involves the use of an institutional database to derive predictors for mortality in sepsis.  Many decision instruments for various sepsis syndromes already exist – CART, MEDS, mREMS, CURB-65, to name a few – but all suffer from the same flaw: how reliable can a rule with just a handful of predictors be when applied to the complex heterogeneity of humanity?

Machine-learning applications of predictive analytics attempt to create, essentially, Decision Instruments 2.0.  Rather than using linear statistical methods to simply weight a small handful of different predictors, most of these applications utilize the entire data set and some form of clustering.  Most generally, these models replace typical variable weighted scoring with, essentially, a weighted neighborhood scheme, in which similarity to other points helps predict outcomes.

Long story short, this study out of Yale utilized 5,278 visits for acute sepsis and a random forest model to create a training set and a validation set.  The random forest model included all available data points from the electronic health record, while other models used up to 20 predictors based on expert input and prior literature.  For their primary outcome of predicting in-hospital death, the AUC for the random forest model was 0.86 (CI 0.82-0.90), while none of the rest of the models exceeded an AUC of 0.76.

This still simply at the technology demonstration phase, and requires further development to become actionable clinical information.  However, I believe models and techniques like this are our next best paradigm in guiding diagnostic and treatment decisions for our heterogenous patient population.  Many challenges yet remain, particularly in the realm of data quality, but I am excited to see more teams engaged in development of similar tools.

“Prediction of In-hospital Mortality in Emergency Department Patients with Sepsis: A Local Big Data Driven, Machine Learning Approach”
http://www.ncbi.nlm.nih.gov/pubmed/26679719

More Futile “Quality”, vis-à-vis, Alert Fatigue

The electronic health record can be a wonderful tool.  As a single application for orders, results review, and integrated documentation storehouse, it holds massive potential.

Unfortunately, much of the currently realized potential is that of unintended harms and inefficiencies.

Even the most seemingly innocuous of checks – those meant to ensure safe medication ordering – have gone rogue, and no one seems capable of restraining them.  These authors report on the real-world effectiveness of adverse drug alerts related to opiates.  These were not public health-related educational interventions, but, simply, duplicate therapy, drug allergy, drug interaction, and pregnancy/lactation safety alerts.  These commonly used medications frequently generate medication safety alerts, and are reasonable targets for study in the Emergency Department.

In just a 4-month study period, these authors retrospectively identified 826 patients for whom an opiate-related medication safety alert was triggered, and these 4,742 alerts constituted the cohort for analysis.  Of these insightful, timely, and important contextual interruptions, these orders were overridden 96.3% of the time.  And, if only physicians had listened, these overridden alerts would have prevented: zero adverse drug events.

In fact, all 8 opiate-related adverse drug events could not have been prevented by alerts – most of which were itching, anyway.  The authors do attribute 38 potentially prevented adverse drug events to the 3.7% of accepted alerts – although, again, these would probably mostly just have been itching.

Thousands of alerts.  A handful of serious events not preventable.  A few episodes of itching averted.  This is the “quality” universe we live in – one in which these alerts paradoxically make our patients less safe due to sheer volume and the phenomenon of “alert fatigue”.

“Clinically Inconsequential Alerts: The Characteristics of Opioid Drug Alerts and Their Utility in Preventing Adverse Drug Events in the Emergency Department”
http://www.ncbi.nlm.nih.gov/pubmed/26553282

The Internet Knows If You’ll Be Dead

As another Clinical Informatics “window into the future” – a window into the future.

These authors used three years of electronic health record data to derive a predictive Bayesian network for patient status.  Its scope: home, hospitalized, or dead.  There are many simple models for predicting such things, but this one is interesting because it attempts to utilize multiple patient features, vital signs, and laboratory results in a continuously updating algorithm.  Ultimately, their model was capable of predicting outcomes up through one week from the initial hospitalization event.

Some fun tidbits:

  • What mattered most on Day 1?  Neutrophils, Hct, and Lactate.
  • As time goes by, the network thinks knowing whether you’re on the Ward at Day 3 is prognostic.
  • By Day 5, variables like a simple count of the total number of tests received, the presence of cancer, and albumin levels start to gain importance.

Their Bayesian prediction network was best at predicting death, with an average accuracy of 93% and an AUROC of 0.84.  Similarly, the prediction engine was most accurate on Day 1, with an average accuracy for each outcome of 86% and an AUROC of 0.83.  Overall, for the entire week and all three outcomes, the AUROC was 0.82.

What was also quite interesting was the model, while also predicting outcomes during the index hospitalization, also detected readmission events within the time period scope.  The authors provide a few validation examples as demonstrations, and include a patient whose probability of hospitalization was trending upwards at the time of discharge – and subsequently was readmitted.

Minority Report, medicine style.

“Real-time prediction of mortality, readmission, and length of stay using electronic health record data”
http://www.ncbi.nlm.nih.gov/pubmed/26374704

Social Media & Medicine: It’s Great! No, it’s Worthless! Wait, What?

In the pattern of the old “Choose Your Own Adventure” novels, you can be for, or against, the Iran deal – or for, or against, the utility of social media in dissemination of medical knowledge and clinical practice.

In the far corner, the defending champion, the curmudgeons of the old guard, for whom the journals and textbooks hold primacy.  The American Academy of Neurology attempted to determine whether a “social media strategy” for dissemination of new clinical practice guidelines had any effect on patient or physician awareness.  They published new guidelines regarding alternative medicine therapies and multiple sclerosis through their traditional member e-mails and literature.  Then, they posted a podcast; a YouTube video; added Facebook, LinkedIn, and YouTube advertisting; and hosted a Twitter chat with Time magazine and others.  Based on survey responses, they were not able to measure any increased awareness of the guidelines resulting from their social media interventions.

Then, the challenger: Radiopedia.org.  This second study evaluates the online views of three articles concerning incidental thyroid nodules on CT and MRI.  Two of the articles were in the American Journal of Neuroradiology and the American Journal of Roentgenology, and the third was hosted on Radiopedia.org.  The Radiopedia blog – with some cross-polination and promotion by traditional means – received 32,675 page views, compared with 2,421 and 3,064 for the online journal publications, respectively.  This matches the anecdotal experience of many blogging physicians, that their online content exposure far exceeds that of their traditional publications.

What’s my takeaway?  Audience matters, content matters, and execution matters just as much as the medium.  When engaging an audience like those attending or presenting at, say, a conference entitled “Social Media and Critical Care”, digital scholarship may easily exceed the value of traditional vehicles.  Alternatively, for a topic as rockin’ as esoteric neurology guidelines, there might simply be a maximal ceiling of interested parties.

“The Impact of Social Media on Dissemination and Implementation of Clinical Practice Guidelines: A Longitudinal Observational Study.”
http://www.ncbi.nlm.nih.gov/pubmed/26272267

“Using Social Media to Share Your Radiology Research: How Effective Is a Blog Post?”
http://www.ncbi.nlm.nih.gov/pubmed/25959491

Beaten Into Submission By Wrong-Patient Alerts

It’s a classic line: “Doctor, did you mean to order Geodon for room 12?  They’re here for urinary issues.”

And, the rolling of eyes, the harried return to the electronic health record – to cancel an order, re-order on the correct patient, and return to the business at hand.

Unfortunately, the human checks in the system don’t always catch these wrong-patient errors, leading to potential serious harms.  As such, this handful of folks decided to test an intervention intended to reduce wrong-patient orders: a built-in system delay.  For every order, a confirmation screen is generated with contextual patient information.  The innovation in this case, is the alert cannot be dismissed until a 2.5 second timer completes.  The theory being, this extra, mandatory wait time will give the ordering clinician a chance to realize their error and cancel out.

Based on a before-and-after design, and observation of 3,457,342 electronic orders across 5 EDs, implementation of this confirmation screen reduced apparent wrong-patient orders from approximately 2 per 1,000 orders to 1.5 per 1,000.  With an average of 30 order-entry sessions per 12-hour shift in these EDs, this patient verification alert had a measured average impact of a mere 2.1 minutes of time.

Which doesn’t sound like much – until it accumulates across all EDs and patient encounters, and, in just the 4 month study period, this system occupied 562 hours of extra time.  This works out to 70 days of extra physician time in these five EDs.  As Robert Wears then beautifully estimates in his editorial, if this alert were implemented nationwide, it would result in 900,000 additional hours of physician time per year – just staring numbly at an alert to verify the correct patient.

It is fairly clear this demonstration is a suboptimal solution to the problem.  While this alert certainly reduces wrong-patient orders of a measurable magnitude, the number of adverse events avoided is much, much smaller.  However, in the absence of an ideal solution, such alternatives as this tend to take root.  As you imagine and experience the various alerts creeping into the system from every angle, it seems inevitably clear:  we will ultimately spend our entire day just negotiating with the EHR, with zero time remaining for clinical care.

“Intercepting Wrong-Patient Orders in a Computerized Provider Order Entry System”
http://www.ncbi.nlm.nih.gov/pubmed/25534652

“‘Just a Few Seconds of Your Time.’ at Least 130 Million Times a Year”
http://www.ncbi.nlm.nih.gov/pubmed/25724623

Doctor Internet Will Misdiagnose You Now

Technology has insidiously infiltrated all manner of industry.  Many tasks, originally accomplished by humans, have been replaced by computers and robots.  All manner of industrialization is now automated, Deep Blue wins at chess, and Watson wins at Jeopardy!

But, don’t rely on Internet symptom checkers to replace your regular physician.

These authors evaluated 23 different online symptom checkers, ranging from the British National Health Service Symptom Checker to privately owned reference sites such as WebMD, with a variety of underlying methodologies.  The authors fed each symptom checker 45 different standardized patient vignettes, ranging in illness severity from pulmonary embolism to otitis media.  The study evaluated twin goals: are the diagnoses generated accurate?  And, do the tools triage patients to the correct venue for medical care?

Eh.

For symptom checkers providing a diagnosis, the correct diagnosis was provided 34% of the time.  This seems pretty decent – until you go further into the data and note these tools left the correct diagnosis completely off the list another 42% of the time.  Most tools providing triage information performed well at referring emergent cases to high levels of care, with 80% sensitivity.  However, this performance was earned by simply referring the bulk of all cases for emergency evaluation, with 45% of non-emergent and 67% of self-care cases being referred to inappropriate levels of medical care.

Of course, this does not evaluate the performance of these online checkers versus telephone advice lines, or even against primary care physicians given the same limited information.  Before being too quick to tout these results as particularly damning, they should be evaluated in the context of their intended purpose.  Unfortunately, due to their general accessibility and typical over-triage, they are likely driving patients to seek higher levels of care than necessary.

“Evaluation of symptom checkers for self diagnosis and triage: audit study”
http://www.ncbi.nlm.nih.gov/pubmed/26157077

New Text Message: Be a Hero! Go!

This pair of articles from the New England Journal catalogues, happily, the happy endings expected of interventions undertaken to increase early bystander CPR.

The first article simply describes a 21 year review of outcomes in Sweden following out-of-hospital cardiac arrest, measuring by 30-day survival in patients who received bystander CPR prior to EMS arrival, with those who did not.  In this review, 14,869 cases received CPR prior to EMS arrival, with a 30-day survival of 10.5%.  The remaining 15,512 cases did not receive CPR prior to EMS arrival, and survival was 4.0%.  This advantage remained, essentially, after all adjustments.  Thus, as expected, bystander CPR is good.

The second article is the magnificent one, however.  In Stockholm, 5,989 lay volunteers were recruited and trained to perform CPR.  Each of these volunteers also consented to make themselves available by contact on their mobile phone to perform CPR in case of a nearby emergency.  Patients with suspected OHCA were geolocated, along with those enrolled in the study, and randomized into two groups to either contact nearby volunteers, or not.  In the intervention group, 62% received bystander CPR, compared with 48% of the controls.  The magnitude of this difference was statistically significant, but, however, the survival difference of 2.6% (CI -2.1 to 7.8) favoring the intervention was not.

But, I think we can pretty readily agree – if bystander CPR improves survival, and text messages to nearby volunteers improves bystander CPR – it’s a matter of statistical power, not futility of the intervention.  If the cost of recruiting and contacting CPR-capable volunteers is low, it is likely increased neurologically-intact survival is the result.

This a an excellent initiative I hope is copied around the world.

“Early Cardiopulmonary Resuscitation in Out-of-Hospital Cardiac Arrest”
http://www.ncbi.nlm.nih.gov/pubmed/26061835

“Mobile-Phone Dispatch of Laypersons for CPR in Out-of-Hospital Cardiac Arrest”
http://www.ncbi.nlm.nih.gov/pubmed/26061836

EMLitOfNote at SAEM Annual Meeting

The blog will be on hiatus this week – in San Diego!

I’ll be speaking at:
Social Media Boot Camp
May 12, 2015, 1:00 pm – 5:00 pm
with multiple members of the SAEM Social Media Committee

FOAM On The Spot: Integration of Online Resources Into Real-Time Education and Patient Care
May 13, 2015, 1:30 – 2:30 PM
with Anand Swaminathan, Matthew Astin, and Lauren Westafer

From Clicks and Complaints to a Curriculum: Integrating an Essential Informatics Education
May 13, 2015, 2:30 – 3:30 PM
with Nicholas Genes and James McClay

and co-author on an abstract presentation:
Automating an Electronic Pulmonary Embolism Severity Index Tool to Facilitate Computerized Clinical Decision Support
May 14, 2015, 10:30 – 10:45 AM

Hope to see a few of you there between Tuesday and Thursday!