Category: Informatics

Strep Throat? Stay Home!

NBC News covered this useful-seeming innovation last week – a predictive score to help patients decide whether their sore throat might be caused by Group A Strep.  It seems a quite reasonable proposition on the surface – if patients can receive guidance on their pretest likelihood of disease, they might rather not seek unnecessary medical care.  For the 12 million physician visits every year for sore throat, putting a dent in this would account for sizable cost savings.

This study describes retrospective development of a “Home Score” for use by patients, based on a MinuteClinic database of 71,000 sore throat presentations for which a strep swab was performed.  The authors split the data into a derivation and a validation set, and produced a complex mathematic scoring system, from 1 to 100, based off age, fever, cough, and biosurveillance data.  Using a score of 10 as a cut-off, the validation set sensitivity was 99%, specificity was 1%, and the prevalence data used resulted in a validation negative predictive value of 87%.  This NPV, the authors say, is the important number regarding advising patients whether they ought to seek care for GAS.

There are a few issues with this derivation, of course.  First of all, the derivation population is subject to selection bias – as only patients with strep swabs are included.  Then, the MinuteClinic data has to be generalizable to the remaining adult population.  The use of the Home Score also depends on the availability of biosurveillance data for their specialized algorithm.  Finally, their NPV cut-off of 90% would theoretically obviate clinic visits for only 230,000 of the 12 million patients seeking care for sore throat – a large drop, but only a drop in the bucket, nonetheless.

And, the elephant in the room: Group A Strep doesn’t need antibiotics in the United States.  The likelihood of adverse reactions to treatment of GAS exceeds the chance of benefit – whether progression to peritonsilar abscess or rheumatic fever is considered.  A few folks on Twitter chimed in to echo this sentiment when this story was discussed:

@embasic @DrLeanaWen @MDaware @NBCNewsHealth just need to redesign app to say “no you don’t” regardless of sx
— Anand Swaminathan (@EMSwami) November 10, 2013

There are legitimate reasons to visit a physician for sore throat – but, in the U.S., nearly all uncomplicated pharyngitis can safely stay home, GAS or not.

“Participatory Medicine: A Home Score for Streptococcal Pharyngitis Enabled by Real-Time Biosurveillance”
http://www.ncbi.nlm.nih.gov/pubmed/24189592

Another Taste of the Future

Putting my Emergency Informatics hat back on for a day, I’d like to highlight another piece of work that brings us, yet again, another step closer to being replaced by computers.

Or, at the minimum, being highly augmented by computers.

There are multitudinous clinical decision instruments available to supplement physician decision-making.  However, the general unifying element of most instruments is the necessary requirement of physician input.  This interruption of clinical flow reduces acceptability of use, and impedes knowledge translation through the use of these tools.

However, since most clinicians are utilizing Electronic Health Records, we’re already entering the information required for most decision instruments into the patient record.  Usually, this is a combination of structured (click click click) and unstructured (type type type) data.  Structured data is easy for clinical calculators to work with, but has none of the richness communicated by freely typed narrative.  Therefore, clinicians much prefer to utilize typed narrative, at the expense of EHR data quality.

This small experiment out of Cincinnati implemented a natural-language processing and machine-learning automated method to collect information from the EHR.  Structured and unstructured data from 2,100 pediatric patients with abdominal pain were analyzed to extract the elements to calculate the Pediatric Appendicitis Score.  Appropriateness of the Pediatric Appendicitis Score aside, their method performed reasonably well.  It picked up about 87% of the elements of the Score from the record, and was correct when doing so about 86%, as well.  However, this was performed retrospectively – and the authors state this processing would still be substantially delayed by hours following the initial encounter.

So, we’re not quite yet at the point where a parallel process monitors system input and provides real-time diagnostic guidance – but, clearly, this is a window into the future.  The theory:  if an automated process could extract the data required to calculate the score, physicians might be more likely to integrate the score into their practice – and thusly lead to higher quality care through more accurate risk-stratification.

I, for one, welcome our new computer overlords.

“Developing and evaluating an automated appendicitis risk stratification algorithm for pediatric patients in the emergency department”

Replace Us With Computers!

In a preview to the future – who performs better at predicting outcomes, a physician, or a computer?

Unsurprisingly, it’s the computer – and the unfortunate bit is we’re not exactly going up against Watson or the hologram doctor from the U.S.S. Voyager here.

This is Jeff Kline, showing off his rather old, not terribly sophisticated “attribute matching” software.  This software, created back in 2005-ish, is based off a database he created of acute coronary syndrome and pulmonary embolism patients.  He determined a handful of most-predictive variables from this set, and then created a tool that allows physicians to input those specific variables from a newly evaluated patient.  The tool then finds the exact matches in the database and spits back a probability estimate based on the historical reference set.

He sells software based on the algorithm and probably would like to see it perform well.  Sadly, it only performs “okay”.  But, it beats physician gestalt, which is probably better ranked as “poor”.  In their prospective evaluation of 840 cases of acute dyspnea or chest pain of uncertain immediate etiology, physicians (mostly attendings, then residents and midlevels) grossly over-estimated the prevalence of ACS and PE.  Physicians had a mean and median pretest estimate for ACS of 17% and 9%, respectively, and the software guessed 4% and 2%.  Actual retail price:  2.7%.  For PE, physicians were at mean 12% and median 6%, with the software at 6% and 5%.  True prevalence: 1.8%.

I don’t choose this article to highlight Kline’s algorithm, nor the comparison between the two.  Mostly, it’s a fascinating observational study of how poor physician estimates are – far over-stating risk.  Certainly, with this foundation, it’s no wonder we’re over-testing folks in nearly every situation.  The future of medicine involves the next generation of similar decision-support instruments – and we will all benefit.

“Clinician Gestalt Estimate of Pretest Probability for Acute Coronary Syndrome and Pulmonary Embolism in Patients With Chest Pain and Dyspnea.”
http://www.ncbi.nlm.nih.gov/pubmed/24070658

Death From a Thousand Clicks

The modern physician – one of the most highly-skilled, highly-compensated data-entry technicians in history.

This is a prospective, observational evaluation of physician activity in the Emergency Department, focusing mostly the time spent in interaction with the electronic health record.  Specifically, they counted mouse clicks during various documentation, order-entry, and other patient care activities.  The observations were conducted for 60-minute time periods, and then extrapolated out to an entire shift, based on multiple observations.

The observations were taken from a mix of residents, attendings, and physician extenders, and offer a lovely glimpse into the burdensome overhead of modern medicine: 28% of time was spent in patient contact, while 44% was spent performing data-entry tasks.  It requires 6 clicks to order an aspirin, 47 clicks to document a physical examination of back pain, and 187 clicks to complete an entire patient encounter for an admitted patient with chest pain.  This extrapolates out, at a pace of 2.5 patients per hour, to ~4000 clicks for a 10-hour shift.

The authors propose a more efficient documentation system would result in increased time available for patient care, increased patients per hour, and increased RVUs per hour.  While the numbers they generate from this sensitivity analysis for productivity increase are essentially fantastical, the underlying concept is valid: the value proposition for these expensive, inefficient electronic health records is based on maximizing reimbursement and charge capture, not by empowering providers to become more productive.

The EHR in use in this study is McKesson Horizon – but, I’m sure these results are generalizable to most EHRs in use today.

“4000 Clicks: a productivity analysis of electronic medical records in a community hospital ED”
http://www.ncbi.nlm.nih.gov/pubmed/24060331

Bar-Code Scanners in the ED

Welcome to the Emergency Department of the Future.  Soft chimes play in the background.  Screaming children are appropriately muffled.  There is natural light and you can hear the ocean.  Patients and doctors alike are polite and respectful, and a benign happiness seems to radiate from all directions.  A young nurse wafts through the patient care areas with a handheld barcode scanner, verifying and dispensing medications in a timely and accurate fashion.

Everything about that vision is coming to your Emergency Department, everything except the chimes, the quiet, the politeness, and the happiness.  The bar-code scanners, however, perhaps.

This is a pre- and post- study from The Ohio State University regarding their use of handheld scanners for medication verification (BCMA).  Our hospital system uses these throughout the inpatient services to verify and provide decision-support for nurses at the final step of the medication delivery process.  However, given the chaotic nature of the Emergency Department, we have not yet implemented them in that environment.  Ohio State, on the other hand, has forged ahead – requiring all medication administrations be verified by bar-code scanner, excepting a small number of “emergency” medications that may be given via override.  They also excluded patients in their resuscitation areas from this requirement.

Across the 2,000 medication administrations observed in the pre- and post- implementation periods, there were reductions in essentially all types various drug administration errors, leading to 63/996 errors in pre- and 12/982 in the post-.  Therefore, these authors conclude – hurrah!

However, none of these errors were serious – and only one even met criteria for “possible temporary harm”.  The majority of errors were “wrong dose”, and involved sedatives, narcotics, and nausea medications the most.  Certainly, the potential for prevention of a significant drug event may be reduced with this system, but it would require much greater statistical power to detect such an effect.  These authors do not touch much upon any unintended consequences of their implementation – such as delays in treatment, changes in LOS, or qualitative frustration with the system.  A better accounting for these effects would assist in fully assessing the utility of this intervention in the Emergency Department.

“Effect of Barcode-assisted Medication Administration on Emergency Department Medication Errors”
http://www.ncbi.nlm.nih.gov/pubmed/24033623

Still Looking For Positive EHR Effects

Our health system just underwent an upgrade from the 2009 version of an EHR to the 2012 version.  The color scheme is a little different.  The painfully cluttered workflow is not significantly changed.  I’m sure there are many Very Important Features – likely relating to burdensome documentation regulations – but, from a clinical standpoint, it still feels like we’re working with Windows 3.1.

But, we suffer this hacked together kludge because of the promise for tangible improvements in quality of care.  One area that has markedly changed with the advent of EHR is the ability to obtain significant medical histories on our patients – without the need to rely on the imperfect patient interview.  The hope of these authors was that, if they compared patients for whom they had complete records established in the EHR to patient who were EHR naive at their facility, they’d be able to demonstrate improvements in at leasts surrogate markers for patient-oriented outcomes.

Looking retrospectively at three EDs covering 13,227 patient visits, these authors found essentially statistical noise.  Comparing multiple outcomes including hospitalization, ED LOS, quantity lab orders, and hospital mortality, they found inconsistently distributed variation that is more likely attributed to unmeasured confounders than any element of the EHR itself.

Like most folks using EHRs, I suspect there are small, difficult-to-measure improvements in healthcare delivery.  Interoperability and centralized data sources would contribute vastly, I hope, to reduced testing and admission rates without adverse effects on outcomes.  However, we’re still waiting for proof.

“The impact of electronic health records on people with diabetes in three different emergency departments”
http://www.ncbi.nlm.nih.gov/pubmed/23842938

New South Wales Dislikes Cerner

The grass is clearly greener on the other side for these folks at Nepean Hospital in New South Wales, AUS.  This study details the before-and-after Emergency Department core measures as they transitioned from the EDIS system to Cerner’s FirstNet.  As they state in their introduction, “Despite limited literature indicating that FirstNet has decreased performance” and “reports of problems with Cerner programs overseas”, FirstNet was foisted upon them – so it’s clear they have an agenda with this publication.


And, a retrospective, observational study is the perfect vehicle for an agenda.  You pick the criteria you want to measure, the most favorable time period, and voilà!  These authors picked a six month pre-intervention period and a six-month post-intervention period.  Triage categories were similar for that six month period.  And then…they present data on a three-month subset.  Indeed, all their descriptive statistics are of only a three-month subset excepting ambulance offload waiting time – for which they have full six month data.  Why choose a study period fraught with missing data?

Then, yes, by every measure they are less efficient at seeing patients with the Cerner product.  The FirstNet system had been in place for six months by the time they report data – but, it’s still not unreasonable to suggest they’re somewhat suffering the growing pains of inexperience.  Then, they also understaff the ED by 3.2 resident shifts and 3.5 attending shifts per week.  An under-staffed ED for a relatively new implementation of a product with low physician acceptance?  

As little love I have for Cerner FirstNet, I’m not sure this study gives it a fair shot.


“Effect of an electronic medical record information system on emergency department performance”
www.ncbi.nlm.nih.gov/pubmed/23451963

A Muddled Look at ED CPOE

Computerized Provider Order Entry – the defining transition in medicine over the last couple decades.  Love it or hate it, as UCSF’s CEO says, the best way to characterize the industry leader is that it succeeds “not because it’s so good, but because others are so bad.”  A fantastic sentiment for a trillion-dollar industry that has somehow become an unavoidable reality of medical practice.

But, it’s not all doom and gloom.  This systematic review of CPOE in use in the Emergency Department identified 22 articles evaluating different aspects of EDIS – and some were even helpful!  The main area of benefit – which has been demonstrated repeatedly in the informatics literature – was a reduction in medication prescribing errors, overdoses, and potential adverse drug events.  There was no consensus regarding changes in patient flow, length of stay, or time spent in direct patient care.  Then, on the flip side, some CPOE interventions were harmful – the effect of order set use as decision-support was implementation dependent, with some institutions seeing increased testing while others saw decreases.

A muddled look at a muddled landscape with, almost certainly, a muddled immediate future.  There are a lot of decisions being made in boardrooms and committees regarding the use of these systems, and not nearly enough evaluation of the unintended consequences.

“May you live in interesting times,” indeed.

“The Effect of Computerized Provider Order Entry Systems on Clinical Care and Work Processes in Emergency Departments: A Systematic Review of the Quantitative Literature”
www.ncbi.nlm.nih.gov/pubmed/23548404

“ePlacebo”-Controlled Trials?

This is a bit of a fascinating application of clinical informatics – using retrospective patient cohorts and propensity matching techniques to reduce the need for placebo groups in future trials.

This is work done by Pfizer on their own internal database, to address the ethical and financial concerns regarding recruiting large populations for new clinical trials.  For example, if you’re testing a new diabetes medication – do you really need a new control group, or can you sort of re-use the control group you had from the previous trial?  The answer of course, has traditionally been no – but their answer is yes-and-no.  Using their database of over 24,000 trials, they were able to identify 4,075 placebo-controlled groups, with varying degrees of data integrity, crossover, and parallel status.  They then suggest these groups could be used, when appropriate, as comparators in future studies in the same domain.

This is certainly an interesting application of clinical informatics – creating temporal databases of clinical trial patients with the potential to augment the evaluation of new medications.  What’s nice is that these authors appropriately recognize the limitations of such a database, noting it may only supplement, not replace placebo arms in future trials.

“Creation and implementation of a historical controls database from randomized clinical trials”
www.ncbi.nlm.nih.gov/pubmed/23449762

Informatics for Wrong-Patient Ordering

It seems intuitive – if, perhaps, the electronic health record has an updated problem list, and the EHR knows the typical indication of various medications, then the EHR would be able to perform some cursory checks for concordance.  If the orders and the problems are not concordant – then, as these authors propose, perhaps the orders are on the wrong patient?

This study is a retrospective analysis of the authors’ EHR, in which they had previously implemented alerts of this fashion in the interests of identifying problem lists that were not current.  However, after data mining their 127,320 alerts over a 6-year period, they noticed 32 orders in which the order was immediately cancelled on one patient and re-ordered on another.  They then conclude that their problem list alert also has the beneficial side-effect of catching wrong-patient orders.

A bit of a stretch – but, it’s an interesting application of surveillance intelligence.  The good news is, at least, that their problem list intervention is successful (pubmed) – because a 0.25 in 1000 patient alert yield for wrong-patient orders would be abysmal!

“Indication-based prescribing prevents wrong-patient medication errors in computerized provider order entry (CPOE)”
www.ncbi.nlm.nih.gov/pubmed/23396543