Article Type
Article
Changed
Mon, 01/02/2017 - 19:34
Display Headline
Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: A systematic review
Author(s)
Anil N. Makam, MD, MAS
Oanh K. Nguyen, MD, MAS
Andrew D. Auerbach, MD, MPH

Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.[1] Rates of sepsis and sepsis‐related mortality are rising in the United States.[2, 3] Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.[4, 5, 6] Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.[4] Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.[7, 8] As such, delays in administering evidence‐based therapies are common.[8, 9]

Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real time or near real time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (eg, through electronic health record [EHR], e‐mail, or pager alerts).

However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our 2 objectives were: (1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real‐time or near real‐time, and (2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.

MATERIALS AND METHODS

Data Sources and Search Strategies

We searched PubMed MEDLINE, Embase, The Cochrane Library, and the Cumulative Index to Nursing and Allied Health Literature from database inception through June 27, 2014, for all studies that contained the following 3 concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5; Thomson‐Reuters Corp., New York, NY) (see Supporting Information, Appendix, in the online version of this article for our complete search strategy).

Study Selection

Two authors (A.N.M. and O.K.N.) reviewed the citation titles, abstracts, and full‐text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: (1) adult patients (aged 18 years) receiving care either in the emergency department or hospital, (2) outcomes of interest including diagnostic accuracy in identification of sepsis, and/or effectiveness of sepsis alerts on process measures and clinical outcomes evaluated using empiric data, and (3) sepsis alert systems used real time or near real time electronically available data to enable proactive, timely management. We excluded studies that: (1) tested the effect of other electronic interventions that were not sepsis alerts (ie, computerized order sets) for sepsis management; (2) studies solely focused on detecting and treating central line‐associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device‐related infections; and (3) studies evaluating the effectiveness of sepsis alerts without a control group.

Data Extraction and Quality Assessment

Two reviewers (A.N.M. and O.K.N.) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in 1 study were resolved by contacting the corresponding author.[10]

For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[11] For studies evaluating the effectiveness of sepsis alert systems, studies were considered high quality if a contemporaneous control group was present to account for temporal trends (eg, randomized controlled trial or observational analysis with a concurrent control). Fair‐quality studies were before‐and‐after studies that adjusted for potential confounders between time periods. Low‐quality studies included those that did not account for temporal trends, such as before‐and‐after studies using only historical controls without adjustment. Studies that did not use an intention‐to‐treat analysis were also considered low quality. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.[12]

Data Synthesis

To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta‐analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio (LR). Effectiveness was assessed by changes in sepsis care process measures (ie, time to antibiotics) and outcomes (length of stay, mortality).

RESULTS

Description of Studies

Of 1293 titles, 183 qualified for abstract review, 84 for full‐text review, and 8 articles met our inclusion criteria (see Supporting Figure in the online version of this article). Five articles evaluated the diagnostic accuracy of sepsis identification,[10, 13, 14, 15, 16] and 5 articles[10, 14, 17, 18, 19] evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single‐site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).

Characteristics of Studies Evaluating the Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Site No./Type Setting Alert Threshold Gold Standard Definition Gold Standard Measurement No. Study Qualitya

  • NOTE: Abbreviations: ED, emergency department; ICD‐9, International Classification of Diseases, Ninth Revision; ICU, intensive care unit; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[10]

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 1/academic MICU 2 SIRS criteriab Reviewer judgment, not otherwise specified Chart review 560 High
Meurer et al., 200913 1/academic ED 2 SIRS criteria Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria Chart review 248 Low
Nelson J. et al., 201114 1/academic ED 2 SIRS criteria and 2 SBP measurements <90 mm Hg Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved Chart review 1,386 High
Nguyen et al., 201415 1/academic ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Reviewer judgment to confirm SIRS, shock, and presence of a serious infection Chart review 1,095 Low
Thiel et al., 201016 1/academic Wards Recursive partitioning tree analysis including vitals and laboratory resultsc Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy ICD‐9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer 27,674 Low
Characteristics of Studies Evaluating the Effectiveness of Automated Electronic Sepsis Alerts
Source Design Site No./ Type Setting No. Alert System Type Alert Threshold Alert Notificationa Treatment Recommendation Study Qualityb

  • NOTE: Abbreviations: CPOE, computerized provider order entry; ED, emergency department; EHR, electronic health record; ICD‐9, International Classification of Diseases, Ninth Revision; MICU, medical intensive care unit; RCT, randomized control trial; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Passive alerts do not require the provider to acknowledge the alert or take action. Text pages were automatically generated and sent.

  • Assessed by prespecified criteria of study design and intention‐to‐treat protocol.

  • Not an intention‐to‐treat analysis. Only patients meeting SIRS criteria with a provider's affirmative response to a computerized query regarding suspected infection were analyzed.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Not an intention‐to‐treat analysis. Only medical records reviewed of individuals with an ICD‐9 discharge code of sepsis.

  • Not an intention‐to‐treat analysis. Three hundred ninety‐eight patients triggered the alert, but only the 184 (46%) confirmed to have an admission diagnosis of infection by chart review were included in the analysis.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

  • Nurses and physicians on intervention wards received sepsis alert education prior to implementation; no education provided to control wards.

Berger et al., 201017 Before‐after (6 months pre and 6 months post) 1/academic ED 5796c CPOE system 2 SIRS criteria CPOE passive alert Yes: lactate collection Low
Hooper et al., 201210 RCT 1/academic MICU 443 EHR 2 SIRS criteriad Text page and EHR passive alert No High
McRee et al., 201418 Before‐after (6 months pre and 6 months post) 1/academic Wards 171e EHR 2 SIRS criteria Notified nurse, specifics unclear No, but the nurse completed a sepsis risk evaluation flow sheet Low
Nelson et al., 201114 Before‐after (3 months pre and 3 months post) 1/academic ED 184f EHR 2 SIRS criteria and 2 or more SBP readings <90 mm Hg Text page and EHR passive alert Yes: fluid resuscitation, blood culture collection, antibiotic administration, among others Low
Sawyer et al., 201119 Prospective, nonrandomized (2 intervention and 4 control wards) 1/academic Wards 300 EHR Recursive partitioning regression tree algorithm including vitals and lab valuesg Text page to charge nurse who then assessed patient and informed treating physicianh No High

Among the 8 included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.[10, 13, 17, 18]

Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems

The prevalence of sepsis varied substantially between the studies depending on the gold standard definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The 2 studies[14, 16] that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8%4.7%) compared to the 2 studies[10, 13] that defined sepsis as having only 2 or more SIRS criteria with a presumed diagnosis of an infection (27.8%32.5%).

Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Setting Alert Threshold Prevalence, % Sensitivity, % (95% CI) Specificity, % (95% CI) PPV, % (95% CI) NPV, % (95% CI) LR+, (95% CI) LR, (95% CI)

  • NOTE: Abbreviations: ED, emergency department; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • False negative and true negatives unknown due to random sampling of nonalert cases.

  • Estimated value based on random sample of 300 non‐alert cases.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 MICU 2 SIRS criteriaa 36.3 98.9 (95.799.8) 18.1 (14.222.9) 40.7 (36.145.5) 96.7 (87.599.4) 1.21 (1.14‐1.27) 0.06 (0.01‐0.25)
Meurer et al., 200913 ED 2 SIRS criteria 27.8 36.2 (25.348.8) 79.9 (73.185.3) 41.0 (28.854.3) 76.5 (69.682.2) 1.80 (1.17‐2.76) 0.80 (0.67‐0.96)
Nelson et al., 201114 ED 2 SIRS criteria and 2 SBP measurements<90 mm Hg 0.8 63.6 (31.687.8) 99.6 (99.099.8) 53.8 (26.179.6) 99.7 (99.299.9) 145.8 (58.4364.1) 0.37 (0.17‐0.80)
Nguyen et al., 201415 ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Unable to estimateb Unable to estimateb Unable to estimateb 44.7 (41.248.2) 100.0c (98.8100.0) Unable to estimateb Unable to estimateb
Thiel et al., 201016 Wards Recursive partitioning tree analysis including vitals and laboratory resultsd 4.7 17.1 (15.119.3) 96.7 (96.596.9) 20.5 (18.223.0) 95.9 (95.796.2) 5.22 (4.56‐5.98) 0.86 (0.84‐0.88)

All alert systems had suboptimal PPV (20.5%‐53.8%). The 2 studies that designed the sepsis alert to activate by SIRS criteria alone[10, 13] had a positive predictive value of 41% and a positive LR of 1.21 to 1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.[10] that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR=0.06) compared to a similar alert system used by Meurer et al.[13] that studied patients in the ED (NPV=76.5%, negative LR=0.80).

There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.[13, 14] The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV=53.8%, LR+=145.8; NPV=99.7%, LR=0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV=41.0%, LR+=1.80; NPV=76.5%, LR=0.80). Furthermore, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.

Two studies evaluating the diagnostic accuracy of the alert system were deemed to be high quality (Table 4). Three studies were considered low quality1 study did not include all patients in their assessment of diagnostic accuracy13; 1 study consecutively selected alert cases but randomly selected nonalert cases, greatly limiting the assessment of diagnostic accuracy15; and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge International Classification of Diseases, Ninth Revision diagnoses without chart review), with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real‐time data).[16]

Assessment of Bias in Studies Evaluating Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Study Patient Selection Index Test Reference Standard Flow and Timing

  • Determined by 2 independent abstractors using the revised Quality Assessment of Diagnostic Accuracy Studies revised tool.11 Three plus signs indicate the lowest risk for bias and 1 plus sign indicates highest risk for bias.

  • Unclear if the gold standard was interpreted without knowledge of the results of the sepsis alert.

  • Not all patients accounted for in the study. Only patients in the emergency department who were subsequently hospitalized were subjected to the gold standard and were included in the analysis.

  • Consecutive selection for cases, but random selection of noncases greatly limited evaluation of diagnostic accuracy.

  • Gold standard was interpreted with knowledge of the results of the sepsis alert.

  • Discharge International Classification of Diseases, Ninth Revision (ICD‐9) diagnosis codes unlikely to correctly classify patients admitted to the hospital wards and subsequently transferred to the intensive care unit for septic shock and needing vasopressor support in the first 24 hours.

  • There was a delay in time between the sepsis alert triggering and ascertainment of the gold standard (discharge ICD‐9 diagnoses), which may result in misclassification.

Hooper et al., 201210 +++ +++ ++b +++
Meurer et al., 200913 +++ +++ ++b +c
Nelson et al., 201114 +++ +++ ++b +++
Nguyen et al., 201415 +d +++ +e +++
Thiel et al., 201016 +++ +++ +f +g

Effectiveness of Automated Electronic Sepsis Alert Systems

Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, 2 studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),[10, 14] 1 study notified the provider by a passive electronic alert alone,[17] and 1 study only employed an automated text page.[19] Furthermore, if the sepsis alert was activated, 2 studies suggested specific clinical management decisions,[14, 17] 2 studies left clinical management decisions solely to the discretion of the treating provider,[10, 19] and 1 study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.[18]

Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be high‐quality studies based on the use of a contemporaneous control group to account for temporal trends and an intention‐to‐treat analysis.[10, 19] The 2 studies evaluating the effectiveness of a sepsis alert system in the ED were considered low quality due to before‐and‐after designs without an intention‐to‐treat analysis.[14, 17]

Effectiveness of Automated Electronic Sepsis Alerts
Source Outcomes Evaluated Key Findings Quality

  • NOTE: Abbreviations: CI, confidence interval; CXR, chest radiograph; ED, emergency department; HR, hazard ratio; ICU, intensive care unit; IV, intravenous; OR, odds ratio.

Hooper et al., 201210 Primary: time to receipt of antibiotic (new or changed) No difference (6.1 hours for control vs 6.0 hours for intervention, P=0.95) High
Secondary: sepsis‐related process measures and outcomes No difference in amount of 6 hour IV fluid administration (964 mL vs 1,019 mL, P=0.6), collection of blood cultures (adjusted HR 1.01; 95% CI, 0.76 to 1.35), collection of lactate (adjusted HR 0.84; 95% CI, 0.54 to 1.30), ICU length of stay (3.0 vs 3.0 days, P=0.2), hospital length of stay (4.7 vs 5.7 days, P=0.08), and hospital mortality (10% for control vs 14% for intervention, P=0.3)
Sawyer et al., 201119 Primary: sepsis‐related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert Increases in receiving 1 measure (56% for control vs 71% for intervention, P=0.02), antibiotic escalation (24% vs 36%, P=0.04), IV fluid administration (24% vs 38%, P=0.01), and oxygen therapy (8% vs 20%, P=0.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs 52%, P=0.06) and vasopressor initiation (3% vs 6%, P=0.4) High
Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in‐hospital mortality Similar rate of ICU transfer (23% for control vs 26% for intervention, P=0.6), hospital length of stay (7 vs 9 days, median, P=0.8), hospital length of stay after alert (5 vs 6 days, median, P=0.7), and in‐hospital mortality (12% vs 10%, P=0.7)
Berger et al., 201017 Primary: lactate collection in ED Increase in lactate collection in the ED (5.2% before vs 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0% to 9.0%) Low
Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (4 mmol/L), and in‐hospital mortality among hospitalized patients Increase in lactate collection among hospitalized patients (15.3% vs 34.2%, absolute increase of 18.9%, 95% CI, 15.0% to 22.8%); decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% CI, 15.8% to 0.6%), and no significant difference in mortality (5.7% vs 5.2%, absolute decrease of 0.5%, 95% CI, 1.6% to 2.6%, P=0.6)
McRee et al., 201418 Stage of sepsis, length of stay, mortality, discharge location Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs 21.9% after, P>0.05); no difference in length‐of‐stay (8.5 days before vs 8.7 days after, P>0.05). Decrease in mortality (9.3% before vs 1.0% after, P<0.05) and proportion of patients discharged home (25.3% before vs 49.0% after, P<0.05) Low
Nelson et al., 201114 Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation Increases in blood culture collection (OR 2.9; 95% CI, 1.1 to 7.7) and CXR (OR 3.2; 95% CI, 1.1 to 9.5); nonsignificant increases in lactate collection (OR 1.7; 95% CI, 0.9 to 3.2) and antibiotic administration (OR 2.8; 95% CI, 0.9 to 8.3). Only blood cultures were collected in a more timely manner (median of 86 minutes before vs 81 minutes after alert implementation, P=0.03). Low

Neither of the 2 high‐quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.[10, 19] The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In a randomized controlled trial of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.[10] However, in a well‐designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.[19] Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,[14, 17] but without improvement in mortality.[17] The single study that showed improvement in clinical outcomes (in‐hospital mortality and disposition location) was of low quality due to the prestudypoststudy design without adjustment for potential confounders and lack of an intention‐to‐treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).[18] Additionally, the preintervention group had a higher proportion of individuals with septic shock compared to the postintervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the 2 groups rather than due to the intervention.

None of the studies included in this review explicitly reported on the potential harms (eg, excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.[10, 19] Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% confidence interval, 15.8% to 0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.[10] Furthermore, 3 studies did not perform intention‐to‐treat analyses; rather, they included only patients who triggered the alert and also had provider‐suspected or confirmed sepsis,[14, 17] or had a discharge diagnosis for sepsis.[18]

DISCUSSION

The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.

We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (eg, SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with noninfectious diseases,[20] sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of alert fatigueexcessive electronic warnings resulting in physicians disregarding clinically useful alerts.[21] The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average 6 alarms per hour with only 15% of alarms considered to be clinically relevant.[22]

The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.[23] First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a ceiling effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,[10] so there likely is a high noise‐to‐signal ratio with resultant alert fatigue.[21]

In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there are little data to describe the optimal design of sepsis alerts[24, 25] or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (eg, direct prompting with specific clinical management recommendations) or the configuration of the alert (eg, interruptive alert or informational).[24, 25] Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems that also alerted other providers (eg, nurses or rapid response teams). Few studies provided data on harms of these systems (eg, excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (eg, randomized or quasiexperimental designs) or used an intention‐to‐treat approach. Studies that exclude false positive alerts in analyses could bias estimates toward making sepsis alert systems appear more effective than they actually were. Finally, although presumably, deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical benefits and cost‐effectiveness of implementing automated sepsis alerts.

Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non‐English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing 1 of our prespecified research questions compared to the number of peer‐reviewed publications identified by our search strategy.

CONCLUSION

Automated electronic sepsis alert systems have promise in delivering early goal‐directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor PPV and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems that avoid the potential for alert fatigue while improving outcomes.

Acknowledgements

The authors thank Gloria Won, MLIS, for her assistance with developing and performing the literature search strategy and wish her a long and joyous retirement.

Disclosures: Part of Dr. Makam's work on this project was completed while he was a primary care research fellow at the University of California, San Francisco, funded by a National Research Service Award (training grant T32HP19025‐07‐00). Dr. Makam is currently supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR001103). Dr. Nguyen was supported by the Agency for Healthcare Research and Quality (R24HS022428‐01). Dr. Auerbach was supported by an NHLBI K24 grant (K24HL098372). Dr. Makam had full access to the data in the study and takes responsibility for the integrity of the date and accuracy of the data analysis. Study concept and design: all authors. Acquisition of data: Makam and Nguyen. Analysis and interpretation of data: all authors. Drafting of the manuscript: Makam. Critical revision of the manuscript: all authors. Statistical analysis: Makam and Nguyen. The authors have no conflicts of interest to disclose.

Files
Supplementary Information (1)
Supplementary Information (2)
References
  1. Torio CM, Andrews RM. National inpatient hospital costs: the most expensive conditions by payer, 2011: statistical brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
  2. Hall MJ, Williams SN, DeFrances CJ, Golosinskiy A. Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011;(62):1–8.
  3. Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348(16):1546–1554.
  4. Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580–637.
  5. Rivers E, Nguyen B, Havstad S, et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377.
  6. Pro CI, Yealy DM, Kellum JA, et al. A randomized trial of protocol‐based care for early septic shock. N Engl J Med. 2014;370(18):1683–1693.
  7. Turi SK, Ah D. Implementation of early goal‐directed therapy for septic patients in the emergency department: a review of the literature. J Emerg Nurs. 2013;39(1):13–19.
  8. Fong JJ, Cecere K, Unterborn J, Garpestad E, Klee M, Devlin JW. Factors influencing variability in compliance rates and clinical outcomes among three different severe sepsis bundles. Ann Pharmacother. 2007;41(6):929–936.
  9. Ferrer R, Artigas A, Levy MM, et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299(19):2294–2303.
  10. Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012;40(7):2096–2101.
  11. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS‐2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536.
  12. Owens DK, Lohr KN, Atkins D, et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health‐care program. J Clin Epidemiol. 2010;63(5):513–523.
  13. Meurer WJ, Smith BL, Losman ED, et al. Real‐time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):40–45.
  14. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real‐time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500–504.
  15. Nguyen SQ, Mwakalindile E, Booth JS, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.
  16. Thiel SW, Rosini JM, Shannon W, Doherty JA, Micek ST, Kollef MH. Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010;5(1):19–25.
  17. Berger T, Birnbaum A, Bijur P, Kuperman G, Gennis P. A Computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394–407.
  18. McRee L, Thanavaro JL, Moore K, Goldsmith M, Pasvogel A. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546–549.
  19. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real‐time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469–473.
  20. Brun‐Buisson C. The epidemiology of the systemic inflammatory response. Intensive Care Med. 2000;26(suppl 1):S64–S74.
  21. Nanji KC, Slight SP, Seger DL, et al. Overrides of medication‐related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(3):487–491.
  22. Siebig S, Kuhls S, Imhoff M, Gather U, Scholmerich J, Wrede CE. Intensive care unit alarms–how many do we need? Crit Care Med. 2010;38(2):451–456.
  23. Singal G, Currier P. How can we best use electronic data to find and treat the critically ill?*. Crit Care Med. 2012;40(7):2242–2243.
  24. Ash JS, Sittig DF, Dykstra R, et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform. 2010;160(pt 2):806–810.
  25. Wright A, Phansalkar S, Bloomrosen M, et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform. 2010;1(3):331–345.
Article PDF
jhm2347.pdf
Issue
Journal of Hospital Medicine - 10(6)
Publications
Journal of Hospital Medicine
Page Number
396-402
Sections
Reviews
Author(s)
Anil N. Makam, MD, MAS
Oanh K. Nguyen, MD, MAS
Andrew D. Auerbach, MD, MPH
Author(s)
Anil N. Makam, MD, MAS
Oanh K. Nguyen, MD, MAS
Andrew D. Auerbach, MD, MPH
Files
Supplementary Information (1)
Supplementary Information (2)
Files
Supplementary Information (1)
Supplementary Information (2)
Article PDF
jhm2347.pdf
Article PDF
jhm2347.pdf

Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.[1] Rates of sepsis and sepsis‐related mortality are rising in the United States.[2, 3] Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.[4, 5, 6] Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.[4] Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.[7, 8] As such, delays in administering evidence‐based therapies are common.[8, 9]

Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real time or near real time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (eg, through electronic health record [EHR], e‐mail, or pager alerts).

However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our 2 objectives were: (1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real‐time or near real‐time, and (2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.

MATERIALS AND METHODS

Data Sources and Search Strategies

We searched PubMed MEDLINE, Embase, The Cochrane Library, and the Cumulative Index to Nursing and Allied Health Literature from database inception through June 27, 2014, for all studies that contained the following 3 concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5; Thomson‐Reuters Corp., New York, NY) (see Supporting Information, Appendix, in the online version of this article for our complete search strategy).

Study Selection

Two authors (A.N.M. and O.K.N.) reviewed the citation titles, abstracts, and full‐text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: (1) adult patients (aged 18 years) receiving care either in the emergency department or hospital, (2) outcomes of interest including diagnostic accuracy in identification of sepsis, and/or effectiveness of sepsis alerts on process measures and clinical outcomes evaluated using empiric data, and (3) sepsis alert systems used real time or near real time electronically available data to enable proactive, timely management. We excluded studies that: (1) tested the effect of other electronic interventions that were not sepsis alerts (ie, computerized order sets) for sepsis management; (2) studies solely focused on detecting and treating central line‐associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device‐related infections; and (3) studies evaluating the effectiveness of sepsis alerts without a control group.

Data Extraction and Quality Assessment

Two reviewers (A.N.M. and O.K.N.) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in 1 study were resolved by contacting the corresponding author.[10]

For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[11] For studies evaluating the effectiveness of sepsis alert systems, studies were considered high quality if a contemporaneous control group was present to account for temporal trends (eg, randomized controlled trial or observational analysis with a concurrent control). Fair‐quality studies were before‐and‐after studies that adjusted for potential confounders between time periods. Low‐quality studies included those that did not account for temporal trends, such as before‐and‐after studies using only historical controls without adjustment. Studies that did not use an intention‐to‐treat analysis were also considered low quality. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.[12]

Data Synthesis

To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta‐analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio (LR). Effectiveness was assessed by changes in sepsis care process measures (ie, time to antibiotics) and outcomes (length of stay, mortality).

RESULTS

Description of Studies

Of 1293 titles, 183 qualified for abstract review, 84 for full‐text review, and 8 articles met our inclusion criteria (see Supporting Figure in the online version of this article). Five articles evaluated the diagnostic accuracy of sepsis identification,[10, 13, 14, 15, 16] and 5 articles[10, 14, 17, 18, 19] evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single‐site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).

Characteristics of Studies Evaluating the Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Site No./Type Setting Alert Threshold Gold Standard Definition Gold Standard Measurement No. Study Qualitya

  • NOTE: Abbreviations: ED, emergency department; ICD‐9, International Classification of Diseases, Ninth Revision; ICU, intensive care unit; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[10]

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 1/academic MICU 2 SIRS criteriab Reviewer judgment, not otherwise specified Chart review 560 High
Meurer et al., 200913 1/academic ED 2 SIRS criteria Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria Chart review 248 Low
Nelson J. et al., 201114 1/academic ED 2 SIRS criteria and 2 SBP measurements <90 mm Hg Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved Chart review 1,386 High
Nguyen et al., 201415 1/academic ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Reviewer judgment to confirm SIRS, shock, and presence of a serious infection Chart review 1,095 Low
Thiel et al., 201016 1/academic Wards Recursive partitioning tree analysis including vitals and laboratory resultsc Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy ICD‐9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer 27,674 Low
Characteristics of Studies Evaluating the Effectiveness of Automated Electronic Sepsis Alerts
Source Design Site No./ Type Setting No. Alert System Type Alert Threshold Alert Notificationa Treatment Recommendation Study Qualityb

  • NOTE: Abbreviations: CPOE, computerized provider order entry; ED, emergency department; EHR, electronic health record; ICD‐9, International Classification of Diseases, Ninth Revision; MICU, medical intensive care unit; RCT, randomized control trial; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Passive alerts do not require the provider to acknowledge the alert or take action. Text pages were automatically generated and sent.

  • Assessed by prespecified criteria of study design and intention‐to‐treat protocol.

  • Not an intention‐to‐treat analysis. Only patients meeting SIRS criteria with a provider's affirmative response to a computerized query regarding suspected infection were analyzed.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Not an intention‐to‐treat analysis. Only medical records reviewed of individuals with an ICD‐9 discharge code of sepsis.

  • Not an intention‐to‐treat analysis. Three hundred ninety‐eight patients triggered the alert, but only the 184 (46%) confirmed to have an admission diagnosis of infection by chart review were included in the analysis.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

  • Nurses and physicians on intervention wards received sepsis alert education prior to implementation; no education provided to control wards.

Berger et al., 201017 Before‐after (6 months pre and 6 months post) 1/academic ED 5796c CPOE system 2 SIRS criteria CPOE passive alert Yes: lactate collection Low
Hooper et al., 201210 RCT 1/academic MICU 443 EHR 2 SIRS criteriad Text page and EHR passive alert No High
McRee et al., 201418 Before‐after (6 months pre and 6 months post) 1/academic Wards 171e EHR 2 SIRS criteria Notified nurse, specifics unclear No, but the nurse completed a sepsis risk evaluation flow sheet Low
Nelson et al., 201114 Before‐after (3 months pre and 3 months post) 1/academic ED 184f EHR 2 SIRS criteria and 2 or more SBP readings <90 mm Hg Text page and EHR passive alert Yes: fluid resuscitation, blood culture collection, antibiotic administration, among others Low
Sawyer et al., 201119 Prospective, nonrandomized (2 intervention and 4 control wards) 1/academic Wards 300 EHR Recursive partitioning regression tree algorithm including vitals and lab valuesg Text page to charge nurse who then assessed patient and informed treating physicianh No High

Among the 8 included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.[10, 13, 17, 18]

Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems

The prevalence of sepsis varied substantially between the studies depending on the gold standard definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The 2 studies[14, 16] that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8%4.7%) compared to the 2 studies[10, 13] that defined sepsis as having only 2 or more SIRS criteria with a presumed diagnosis of an infection (27.8%32.5%).

Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Setting Alert Threshold Prevalence, % Sensitivity, % (95% CI) Specificity, % (95% CI) PPV, % (95% CI) NPV, % (95% CI) LR+, (95% CI) LR, (95% CI)

  • NOTE: Abbreviations: ED, emergency department; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • False negative and true negatives unknown due to random sampling of nonalert cases.

  • Estimated value based on random sample of 300 non‐alert cases.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 MICU 2 SIRS criteriaa 36.3 98.9 (95.799.8) 18.1 (14.222.9) 40.7 (36.145.5) 96.7 (87.599.4) 1.21 (1.14‐1.27) 0.06 (0.01‐0.25)
Meurer et al., 200913 ED 2 SIRS criteria 27.8 36.2 (25.348.8) 79.9 (73.185.3) 41.0 (28.854.3) 76.5 (69.682.2) 1.80 (1.17‐2.76) 0.80 (0.67‐0.96)
Nelson et al., 201114 ED 2 SIRS criteria and 2 SBP measurements<90 mm Hg 0.8 63.6 (31.687.8) 99.6 (99.099.8) 53.8 (26.179.6) 99.7 (99.299.9) 145.8 (58.4364.1) 0.37 (0.17‐0.80)
Nguyen et al., 201415 ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Unable to estimateb Unable to estimateb Unable to estimateb 44.7 (41.248.2) 100.0c (98.8100.0) Unable to estimateb Unable to estimateb
Thiel et al., 201016 Wards Recursive partitioning tree analysis including vitals and laboratory resultsd 4.7 17.1 (15.119.3) 96.7 (96.596.9) 20.5 (18.223.0) 95.9 (95.796.2) 5.22 (4.56‐5.98) 0.86 (0.84‐0.88)

All alert systems had suboptimal PPV (20.5%‐53.8%). The 2 studies that designed the sepsis alert to activate by SIRS criteria alone[10, 13] had a positive predictive value of 41% and a positive LR of 1.21 to 1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.[10] that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR=0.06) compared to a similar alert system used by Meurer et al.[13] that studied patients in the ED (NPV=76.5%, negative LR=0.80).

There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.[13, 14] The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV=53.8%, LR+=145.8; NPV=99.7%, LR=0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV=41.0%, LR+=1.80; NPV=76.5%, LR=0.80). Furthermore, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.

Two studies evaluating the diagnostic accuracy of the alert system were deemed to be high quality (Table 4). Three studies were considered low quality1 study did not include all patients in their assessment of diagnostic accuracy13; 1 study consecutively selected alert cases but randomly selected nonalert cases, greatly limiting the assessment of diagnostic accuracy15; and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge International Classification of Diseases, Ninth Revision diagnoses without chart review), with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real‐time data).[16]

Assessment of Bias in Studies Evaluating Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Study Patient Selection Index Test Reference Standard Flow and Timing

  • Determined by 2 independent abstractors using the revised Quality Assessment of Diagnostic Accuracy Studies revised tool.11 Three plus signs indicate the lowest risk for bias and 1 plus sign indicates highest risk for bias.

  • Unclear if the gold standard was interpreted without knowledge of the results of the sepsis alert.

  • Not all patients accounted for in the study. Only patients in the emergency department who were subsequently hospitalized were subjected to the gold standard and were included in the analysis.

  • Consecutive selection for cases, but random selection of noncases greatly limited evaluation of diagnostic accuracy.

  • Gold standard was interpreted with knowledge of the results of the sepsis alert.

  • Discharge International Classification of Diseases, Ninth Revision (ICD‐9) diagnosis codes unlikely to correctly classify patients admitted to the hospital wards and subsequently transferred to the intensive care unit for septic shock and needing vasopressor support in the first 24 hours.

  • There was a delay in time between the sepsis alert triggering and ascertainment of the gold standard (discharge ICD‐9 diagnoses), which may result in misclassification.

Hooper et al., 201210 +++ +++ ++b +++
Meurer et al., 200913 +++ +++ ++b +c
Nelson et al., 201114 +++ +++ ++b +++
Nguyen et al., 201415 +d +++ +e +++
Thiel et al., 201016 +++ +++ +f +g

Effectiveness of Automated Electronic Sepsis Alert Systems

Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, 2 studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),[10, 14] 1 study notified the provider by a passive electronic alert alone,[17] and 1 study only employed an automated text page.[19] Furthermore, if the sepsis alert was activated, 2 studies suggested specific clinical management decisions,[14, 17] 2 studies left clinical management decisions solely to the discretion of the treating provider,[10, 19] and 1 study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.[18]

Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be high‐quality studies based on the use of a contemporaneous control group to account for temporal trends and an intention‐to‐treat analysis.[10, 19] The 2 studies evaluating the effectiveness of a sepsis alert system in the ED were considered low quality due to before‐and‐after designs without an intention‐to‐treat analysis.[14, 17]

Effectiveness of Automated Electronic Sepsis Alerts
Source Outcomes Evaluated Key Findings Quality

  • NOTE: Abbreviations: CI, confidence interval; CXR, chest radiograph; ED, emergency department; HR, hazard ratio; ICU, intensive care unit; IV, intravenous; OR, odds ratio.

Hooper et al., 201210 Primary: time to receipt of antibiotic (new or changed) No difference (6.1 hours for control vs 6.0 hours for intervention, P=0.95) High
Secondary: sepsis‐related process measures and outcomes No difference in amount of 6 hour IV fluid administration (964 mL vs 1,019 mL, P=0.6), collection of blood cultures (adjusted HR 1.01; 95% CI, 0.76 to 1.35), collection of lactate (adjusted HR 0.84; 95% CI, 0.54 to 1.30), ICU length of stay (3.0 vs 3.0 days, P=0.2), hospital length of stay (4.7 vs 5.7 days, P=0.08), and hospital mortality (10% for control vs 14% for intervention, P=0.3)
Sawyer et al., 201119 Primary: sepsis‐related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert Increases in receiving 1 measure (56% for control vs 71% for intervention, P=0.02), antibiotic escalation (24% vs 36%, P=0.04), IV fluid administration (24% vs 38%, P=0.01), and oxygen therapy (8% vs 20%, P=0.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs 52%, P=0.06) and vasopressor initiation (3% vs 6%, P=0.4) High
Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in‐hospital mortality Similar rate of ICU transfer (23% for control vs 26% for intervention, P=0.6), hospital length of stay (7 vs 9 days, median, P=0.8), hospital length of stay after alert (5 vs 6 days, median, P=0.7), and in‐hospital mortality (12% vs 10%, P=0.7)
Berger et al., 201017 Primary: lactate collection in ED Increase in lactate collection in the ED (5.2% before vs 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0% to 9.0%) Low
Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (4 mmol/L), and in‐hospital mortality among hospitalized patients Increase in lactate collection among hospitalized patients (15.3% vs 34.2%, absolute increase of 18.9%, 95% CI, 15.0% to 22.8%); decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% CI, 15.8% to 0.6%), and no significant difference in mortality (5.7% vs 5.2%, absolute decrease of 0.5%, 95% CI, 1.6% to 2.6%, P=0.6)
McRee et al., 201418 Stage of sepsis, length of stay, mortality, discharge location Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs 21.9% after, P>0.05); no difference in length‐of‐stay (8.5 days before vs 8.7 days after, P>0.05). Decrease in mortality (9.3% before vs 1.0% after, P<0.05) and proportion of patients discharged home (25.3% before vs 49.0% after, P<0.05) Low
Nelson et al., 201114 Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation Increases in blood culture collection (OR 2.9; 95% CI, 1.1 to 7.7) and CXR (OR 3.2; 95% CI, 1.1 to 9.5); nonsignificant increases in lactate collection (OR 1.7; 95% CI, 0.9 to 3.2) and antibiotic administration (OR 2.8; 95% CI, 0.9 to 8.3). Only blood cultures were collected in a more timely manner (median of 86 minutes before vs 81 minutes after alert implementation, P=0.03). Low

Neither of the 2 high‐quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.[10, 19] The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In a randomized controlled trial of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.[10] However, in a well‐designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.[19] Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,[14, 17] but without improvement in mortality.[17] The single study that showed improvement in clinical outcomes (in‐hospital mortality and disposition location) was of low quality due to the prestudypoststudy design without adjustment for potential confounders and lack of an intention‐to‐treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).[18] Additionally, the preintervention group had a higher proportion of individuals with septic shock compared to the postintervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the 2 groups rather than due to the intervention.

None of the studies included in this review explicitly reported on the potential harms (eg, excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.[10, 19] Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% confidence interval, 15.8% to 0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.[10] Furthermore, 3 studies did not perform intention‐to‐treat analyses; rather, they included only patients who triggered the alert and also had provider‐suspected or confirmed sepsis,[14, 17] or had a discharge diagnosis for sepsis.[18]

DISCUSSION

The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.

We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (eg, SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with noninfectious diseases,[20] sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of alert fatigueexcessive electronic warnings resulting in physicians disregarding clinically useful alerts.[21] The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average 6 alarms per hour with only 15% of alarms considered to be clinically relevant.[22]

The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.[23] First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a ceiling effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,[10] so there likely is a high noise‐to‐signal ratio with resultant alert fatigue.[21]

In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there are little data to describe the optimal design of sepsis alerts[24, 25] or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (eg, direct prompting with specific clinical management recommendations) or the configuration of the alert (eg, interruptive alert or informational).[24, 25] Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems that also alerted other providers (eg, nurses or rapid response teams). Few studies provided data on harms of these systems (eg, excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (eg, randomized or quasiexperimental designs) or used an intention‐to‐treat approach. Studies that exclude false positive alerts in analyses could bias estimates toward making sepsis alert systems appear more effective than they actually were. Finally, although presumably, deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical benefits and cost‐effectiveness of implementing automated sepsis alerts.

Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non‐English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing 1 of our prespecified research questions compared to the number of peer‐reviewed publications identified by our search strategy.

CONCLUSION

Automated electronic sepsis alert systems have promise in delivering early goal‐directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor PPV and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems that avoid the potential for alert fatigue while improving outcomes.

Acknowledgements

The authors thank Gloria Won, MLIS, for her assistance with developing and performing the literature search strategy and wish her a long and joyous retirement.

Disclosures: Part of Dr. Makam's work on this project was completed while he was a primary care research fellow at the University of California, San Francisco, funded by a National Research Service Award (training grant T32HP19025‐07‐00). Dr. Makam is currently supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR001103). Dr. Nguyen was supported by the Agency for Healthcare Research and Quality (R24HS022428‐01). Dr. Auerbach was supported by an NHLBI K24 grant (K24HL098372). Dr. Makam had full access to the data in the study and takes responsibility for the integrity of the date and accuracy of the data analysis. Study concept and design: all authors. Acquisition of data: Makam and Nguyen. Analysis and interpretation of data: all authors. Drafting of the manuscript: Makam. Critical revision of the manuscript: all authors. Statistical analysis: Makam and Nguyen. The authors have no conflicts of interest to disclose.

Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.[1] Rates of sepsis and sepsis‐related mortality are rising in the United States.[2, 3] Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.[4, 5, 6] Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.[4] Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.[7, 8] As such, delays in administering evidence‐based therapies are common.[8, 9]

Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real time or near real time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (eg, through electronic health record [EHR], e‐mail, or pager alerts).

However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our 2 objectives were: (1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real‐time or near real‐time, and (2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.

MATERIALS AND METHODS

Data Sources and Search Strategies

We searched PubMed MEDLINE, Embase, The Cochrane Library, and the Cumulative Index to Nursing and Allied Health Literature from database inception through June 27, 2014, for all studies that contained the following 3 concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5; Thomson‐Reuters Corp., New York, NY) (see Supporting Information, Appendix, in the online version of this article for our complete search strategy).

Study Selection

Two authors (A.N.M. and O.K.N.) reviewed the citation titles, abstracts, and full‐text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: (1) adult patients (aged 18 years) receiving care either in the emergency department or hospital, (2) outcomes of interest including diagnostic accuracy in identification of sepsis, and/or effectiveness of sepsis alerts on process measures and clinical outcomes evaluated using empiric data, and (3) sepsis alert systems used real time or near real time electronically available data to enable proactive, timely management. We excluded studies that: (1) tested the effect of other electronic interventions that were not sepsis alerts (ie, computerized order sets) for sepsis management; (2) studies solely focused on detecting and treating central line‐associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device‐related infections; and (3) studies evaluating the effectiveness of sepsis alerts without a control group.

Data Extraction and Quality Assessment

Two reviewers (A.N.M. and O.K.N.) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in 1 study were resolved by contacting the corresponding author.[10]

For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[11] For studies evaluating the effectiveness of sepsis alert systems, studies were considered high quality if a contemporaneous control group was present to account for temporal trends (eg, randomized controlled trial or observational analysis with a concurrent control). Fair‐quality studies were before‐and‐after studies that adjusted for potential confounders between time periods. Low‐quality studies included those that did not account for temporal trends, such as before‐and‐after studies using only historical controls without adjustment. Studies that did not use an intention‐to‐treat analysis were also considered low quality. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.[12]

Data Synthesis

To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta‐analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio (LR). Effectiveness was assessed by changes in sepsis care process measures (ie, time to antibiotics) and outcomes (length of stay, mortality).

RESULTS

Description of Studies

Of 1293 titles, 183 qualified for abstract review, 84 for full‐text review, and 8 articles met our inclusion criteria (see Supporting Figure in the online version of this article). Five articles evaluated the diagnostic accuracy of sepsis identification,[10, 13, 14, 15, 16] and 5 articles[10, 14, 17, 18, 19] evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single‐site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).

Characteristics of Studies Evaluating the Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Site No./Type Setting Alert Threshold Gold Standard Definition Gold Standard Measurement No. Study Qualitya

  • NOTE: Abbreviations: ED, emergency department; ICD‐9, International Classification of Diseases, Ninth Revision; ICU, intensive care unit; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[10]

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 1/academic MICU 2 SIRS criteriab Reviewer judgment, not otherwise specified Chart review 560 High
Meurer et al., 200913 1/academic ED 2 SIRS criteria Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria Chart review 248 Low
Nelson J. et al., 201114 1/academic ED 2 SIRS criteria and 2 SBP measurements <90 mm Hg Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved Chart review 1,386 High
Nguyen et al., 201415 1/academic ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Reviewer judgment to confirm SIRS, shock, and presence of a serious infection Chart review 1,095 Low
Thiel et al., 201016 1/academic Wards Recursive partitioning tree analysis including vitals and laboratory resultsc Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy ICD‐9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer 27,674 Low
Characteristics of Studies Evaluating the Effectiveness of Automated Electronic Sepsis Alerts
Source Design Site No./ Type Setting No. Alert System Type Alert Threshold Alert Notificationa Treatment Recommendation Study Qualityb

  • NOTE: Abbreviations: CPOE, computerized provider order entry; ED, emergency department; EHR, electronic health record; ICD‐9, International Classification of Diseases, Ninth Revision; MICU, medical intensive care unit; RCT, randomized control trial; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Passive alerts do not require the provider to acknowledge the alert or take action. Text pages were automatically generated and sent.

  • Assessed by prespecified criteria of study design and intention‐to‐treat protocol.

  • Not an intention‐to‐treat analysis. Only patients meeting SIRS criteria with a provider's affirmative response to a computerized query regarding suspected infection were analyzed.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Not an intention‐to‐treat analysis. Only medical records reviewed of individuals with an ICD‐9 discharge code of sepsis.

  • Not an intention‐to‐treat analysis. Three hundred ninety‐eight patients triggered the alert, but only the 184 (46%) confirmed to have an admission diagnosis of infection by chart review were included in the analysis.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

  • Nurses and physicians on intervention wards received sepsis alert education prior to implementation; no education provided to control wards.

Berger et al., 201017 Before‐after (6 months pre and 6 months post) 1/academic ED 5796c CPOE system 2 SIRS criteria CPOE passive alert Yes: lactate collection Low
Hooper et al., 201210 RCT 1/academic MICU 443 EHR 2 SIRS criteriad Text page and EHR passive alert No High
McRee et al., 201418 Before‐after (6 months pre and 6 months post) 1/academic Wards 171e EHR 2 SIRS criteria Notified nurse, specifics unclear No, but the nurse completed a sepsis risk evaluation flow sheet Low
Nelson et al., 201114 Before‐after (3 months pre and 3 months post) 1/academic ED 184f EHR 2 SIRS criteria and 2 or more SBP readings <90 mm Hg Text page and EHR passive alert Yes: fluid resuscitation, blood culture collection, antibiotic administration, among others Low
Sawyer et al., 201119 Prospective, nonrandomized (2 intervention and 4 control wards) 1/academic Wards 300 EHR Recursive partitioning regression tree algorithm including vitals and lab valuesg Text page to charge nurse who then assessed patient and informed treating physicianh No High

Among the 8 included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.[10, 13, 17, 18]

Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems

The prevalence of sepsis varied substantially between the studies depending on the gold standard definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The 2 studies[14, 16] that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8%4.7%) compared to the 2 studies[10, 13] that defined sepsis as having only 2 or more SIRS criteria with a presumed diagnosis of an infection (27.8%32.5%).

Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Setting Alert Threshold Prevalence, % Sensitivity, % (95% CI) Specificity, % (95% CI) PPV, % (95% CI) NPV, % (95% CI) LR+, (95% CI) LR, (95% CI)

  • NOTE: Abbreviations: ED, emergency department; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • False negative and true negatives unknown due to random sampling of nonalert cases.

  • Estimated value based on random sample of 300 non‐alert cases.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 MICU 2 SIRS criteriaa 36.3 98.9 (95.799.8) 18.1 (14.222.9) 40.7 (36.145.5) 96.7 (87.599.4) 1.21 (1.14‐1.27) 0.06 (0.01‐0.25)
Meurer et al., 200913 ED 2 SIRS criteria 27.8 36.2 (25.348.8) 79.9 (73.185.3) 41.0 (28.854.3) 76.5 (69.682.2) 1.80 (1.17‐2.76) 0.80 (0.67‐0.96)
Nelson et al., 201114 ED 2 SIRS criteria and 2 SBP measurements<90 mm Hg 0.8 63.6 (31.687.8) 99.6 (99.099.8) 53.8 (26.179.6) 99.7 (99.299.9) 145.8 (58.4364.1) 0.37 (0.17‐0.80)
Nguyen et al., 201415 ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Unable to estimateb Unable to estimateb Unable to estimateb 44.7 (41.248.2) 100.0c (98.8100.0) Unable to estimateb Unable to estimateb
Thiel et al., 201016 Wards Recursive partitioning tree analysis including vitals and laboratory resultsd 4.7 17.1 (15.119.3) 96.7 (96.596.9) 20.5 (18.223.0) 95.9 (95.796.2) 5.22 (4.56‐5.98) 0.86 (0.84‐0.88)

All alert systems had suboptimal PPV (20.5%‐53.8%). The 2 studies that designed the sepsis alert to activate by SIRS criteria alone[10, 13] had a positive predictive value of 41% and a positive LR of 1.21 to 1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.[10] that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR=0.06) compared to a similar alert system used by Meurer et al.[13] that studied patients in the ED (NPV=76.5%, negative LR=0.80).

There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.[13, 14] The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV=53.8%, LR+=145.8; NPV=99.7%, LR=0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV=41.0%, LR+=1.80; NPV=76.5%, LR=0.80). Furthermore, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.

Two studies evaluating the diagnostic accuracy of the alert system were deemed to be high quality (Table 4). Three studies were considered low quality1 study did not include all patients in their assessment of diagnostic accuracy13; 1 study consecutively selected alert cases but randomly selected nonalert cases, greatly limiting the assessment of diagnostic accuracy15; and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge International Classification of Diseases, Ninth Revision diagnoses without chart review), with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real‐time data).[16]

Assessment of Bias in Studies Evaluating Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Study Patient Selection Index Test Reference Standard Flow and Timing

  • Determined by 2 independent abstractors using the revised Quality Assessment of Diagnostic Accuracy Studies revised tool.11 Three plus signs indicate the lowest risk for bias and 1 plus sign indicates highest risk for bias.

  • Unclear if the gold standard was interpreted without knowledge of the results of the sepsis alert.

  • Not all patients accounted for in the study. Only patients in the emergency department who were subsequently hospitalized were subjected to the gold standard and were included in the analysis.

  • Consecutive selection for cases, but random selection of noncases greatly limited evaluation of diagnostic accuracy.

  • Gold standard was interpreted with knowledge of the results of the sepsis alert.

  • Discharge International Classification of Diseases, Ninth Revision (ICD‐9) diagnosis codes unlikely to correctly classify patients admitted to the hospital wards and subsequently transferred to the intensive care unit for septic shock and needing vasopressor support in the first 24 hours.

  • There was a delay in time between the sepsis alert triggering and ascertainment of the gold standard (discharge ICD‐9 diagnoses), which may result in misclassification.

Hooper et al., 201210 +++ +++ ++b +++
Meurer et al., 200913 +++ +++ ++b +c
Nelson et al., 201114 +++ +++ ++b +++
Nguyen et al., 201415 +d +++ +e +++
Thiel et al., 201016 +++ +++ +f +g

Effectiveness of Automated Electronic Sepsis Alert Systems

Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, 2 studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),[10, 14] 1 study notified the provider by a passive electronic alert alone,[17] and 1 study only employed an automated text page.[19] Furthermore, if the sepsis alert was activated, 2 studies suggested specific clinical management decisions,[14, 17] 2 studies left clinical management decisions solely to the discretion of the treating provider,[10, 19] and 1 study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.[18]

Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be high‐quality studies based on the use of a contemporaneous control group to account for temporal trends and an intention‐to‐treat analysis.[10, 19] The 2 studies evaluating the effectiveness of a sepsis alert system in the ED were considered low quality due to before‐and‐after designs without an intention‐to‐treat analysis.[14, 17]

Effectiveness of Automated Electronic Sepsis Alerts
Source Outcomes Evaluated Key Findings Quality

  • NOTE: Abbreviations: CI, confidence interval; CXR, chest radiograph; ED, emergency department; HR, hazard ratio; ICU, intensive care unit; IV, intravenous; OR, odds ratio.

Hooper et al., 201210 Primary: time to receipt of antibiotic (new or changed) No difference (6.1 hours for control vs 6.0 hours for intervention, P=0.95) High
Secondary: sepsis‐related process measures and outcomes No difference in amount of 6 hour IV fluid administration (964 mL vs 1,019 mL, P=0.6), collection of blood cultures (adjusted HR 1.01; 95% CI, 0.76 to 1.35), collection of lactate (adjusted HR 0.84; 95% CI, 0.54 to 1.30), ICU length of stay (3.0 vs 3.0 days, P=0.2), hospital length of stay (4.7 vs 5.7 days, P=0.08), and hospital mortality (10% for control vs 14% for intervention, P=0.3)
Sawyer et al., 201119 Primary: sepsis‐related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert Increases in receiving 1 measure (56% for control vs 71% for intervention, P=0.02), antibiotic escalation (24% vs 36%, P=0.04), IV fluid administration (24% vs 38%, P=0.01), and oxygen therapy (8% vs 20%, P=0.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs 52%, P=0.06) and vasopressor initiation (3% vs 6%, P=0.4) High
Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in‐hospital mortality Similar rate of ICU transfer (23% for control vs 26% for intervention, P=0.6), hospital length of stay (7 vs 9 days, median, P=0.8), hospital length of stay after alert (5 vs 6 days, median, P=0.7), and in‐hospital mortality (12% vs 10%, P=0.7)
Berger et al., 201017 Primary: lactate collection in ED Increase in lactate collection in the ED (5.2% before vs 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0% to 9.0%) Low
Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (4 mmol/L), and in‐hospital mortality among hospitalized patients Increase in lactate collection among hospitalized patients (15.3% vs 34.2%, absolute increase of 18.9%, 95% CI, 15.0% to 22.8%); decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% CI, 15.8% to 0.6%), and no significant difference in mortality (5.7% vs 5.2%, absolute decrease of 0.5%, 95% CI, 1.6% to 2.6%, P=0.6)
McRee et al., 201418 Stage of sepsis, length of stay, mortality, discharge location Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs 21.9% after, P>0.05); no difference in length‐of‐stay (8.5 days before vs 8.7 days after, P>0.05). Decrease in mortality (9.3% before vs 1.0% after, P<0.05) and proportion of patients discharged home (25.3% before vs 49.0% after, P<0.05) Low
Nelson et al., 201114 Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation Increases in blood culture collection (OR 2.9; 95% CI, 1.1 to 7.7) and CXR (OR 3.2; 95% CI, 1.1 to 9.5); nonsignificant increases in lactate collection (OR 1.7; 95% CI, 0.9 to 3.2) and antibiotic administration (OR 2.8; 95% CI, 0.9 to 8.3). Only blood cultures were collected in a more timely manner (median of 86 minutes before vs 81 minutes after alert implementation, P=0.03). Low

Neither of the 2 high‐quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.[10, 19] The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In a randomized controlled trial of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.[10] However, in a well‐designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.[19] Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,[14, 17] but without improvement in mortality.[17] The single study that showed improvement in clinical outcomes (in‐hospital mortality and disposition location) was of low quality due to the prestudypoststudy design without adjustment for potential confounders and lack of an intention‐to‐treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).[18] Additionally, the preintervention group had a higher proportion of individuals with septic shock compared to the postintervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the 2 groups rather than due to the intervention.

None of the studies included in this review explicitly reported on the potential harms (eg, excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.[10, 19] Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% confidence interval, 15.8% to 0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.[10] Furthermore, 3 studies did not perform intention‐to‐treat analyses; rather, they included only patients who triggered the alert and also had provider‐suspected or confirmed sepsis,[14, 17] or had a discharge diagnosis for sepsis.[18]

DISCUSSION

The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.

We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (eg, SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with noninfectious diseases,[20] sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of alert fatigueexcessive electronic warnings resulting in physicians disregarding clinically useful alerts.[21] The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average 6 alarms per hour with only 15% of alarms considered to be clinically relevant.[22]

The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.[23] First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a ceiling effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,[10] so there likely is a high noise‐to‐signal ratio with resultant alert fatigue.[21]

In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there are little data to describe the optimal design of sepsis alerts[24, 25] or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (eg, direct prompting with specific clinical management recommendations) or the configuration of the alert (eg, interruptive alert or informational).[24, 25] Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems that also alerted other providers (eg, nurses or rapid response teams). Few studies provided data on harms of these systems (eg, excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (eg, randomized or quasiexperimental designs) or used an intention‐to‐treat approach. Studies that exclude false positive alerts in analyses could bias estimates toward making sepsis alert systems appear more effective than they actually were. Finally, although presumably, deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical benefits and cost‐effectiveness of implementing automated sepsis alerts.

Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non‐English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing 1 of our prespecified research questions compared to the number of peer‐reviewed publications identified by our search strategy.

CONCLUSION

Automated electronic sepsis alert systems have promise in delivering early goal‐directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor PPV and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems that avoid the potential for alert fatigue while improving outcomes.

Acknowledgements

The authors thank Gloria Won, MLIS, for her assistance with developing and performing the literature search strategy and wish her a long and joyous retirement.

Disclosures: Part of Dr. Makam's work on this project was completed while he was a primary care research fellow at the University of California, San Francisco, funded by a National Research Service Award (training grant T32HP19025‐07‐00). Dr. Makam is currently supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR001103). Dr. Nguyen was supported by the Agency for Healthcare Research and Quality (R24HS022428‐01). Dr. Auerbach was supported by an NHLBI K24 grant (K24HL098372). Dr. Makam had full access to the data in the study and takes responsibility for the integrity of the date and accuracy of the data analysis. Study concept and design: all authors. Acquisition of data: Makam and Nguyen. Analysis and interpretation of data: all authors. Drafting of the manuscript: Makam. Critical revision of the manuscript: all authors. Statistical analysis: Makam and Nguyen. The authors have no conflicts of interest to disclose.

References
  1. Torio CM, Andrews RM. National inpatient hospital costs: the most expensive conditions by payer, 2011: statistical brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
  2. Hall MJ, Williams SN, DeFrances CJ, Golosinskiy A. Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011;(62):1–8.
  3. Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348(16):1546–1554.
  4. Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580–637.
  5. Rivers E, Nguyen B, Havstad S, et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377.
  6. Pro CI, Yealy DM, Kellum JA, et al. A randomized trial of protocol‐based care for early septic shock. N Engl J Med. 2014;370(18):1683–1693.
  7. Turi SK, Ah D. Implementation of early goal‐directed therapy for septic patients in the emergency department: a review of the literature. J Emerg Nurs. 2013;39(1):13–19.
  8. Fong JJ, Cecere K, Unterborn J, Garpestad E, Klee M, Devlin JW. Factors influencing variability in compliance rates and clinical outcomes among three different severe sepsis bundles. Ann Pharmacother. 2007;41(6):929–936.
  9. Ferrer R, Artigas A, Levy MM, et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299(19):2294–2303.
  10. Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012;40(7):2096–2101.
  11. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS‐2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536.
  12. Owens DK, Lohr KN, Atkins D, et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health‐care program. J Clin Epidemiol. 2010;63(5):513–523.
  13. Meurer WJ, Smith BL, Losman ED, et al. Real‐time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):40–45.
  14. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real‐time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500–504.
  15. Nguyen SQ, Mwakalindile E, Booth JS, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.
  16. Thiel SW, Rosini JM, Shannon W, Doherty JA, Micek ST, Kollef MH. Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010;5(1):19–25.
  17. Berger T, Birnbaum A, Bijur P, Kuperman G, Gennis P. A Computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394–407.
  18. McRee L, Thanavaro JL, Moore K, Goldsmith M, Pasvogel A. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546–549.
  19. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real‐time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469–473.
  20. Brun‐Buisson C. The epidemiology of the systemic inflammatory response. Intensive Care Med. 2000;26(suppl 1):S64–S74.
  21. Nanji KC, Slight SP, Seger DL, et al. Overrides of medication‐related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(3):487–491.
  22. Siebig S, Kuhls S, Imhoff M, Gather U, Scholmerich J, Wrede CE. Intensive care unit alarms–how many do we need? Crit Care Med. 2010;38(2):451–456.
  23. Singal G, Currier P. How can we best use electronic data to find and treat the critically ill?*. Crit Care Med. 2012;40(7):2242–2243.
  24. Ash JS, Sittig DF, Dykstra R, et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform. 2010;160(pt 2):806–810.
  25. Wright A, Phansalkar S, Bloomrosen M, et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform. 2010;1(3):331–345.
References
  1. Torio CM, Andrews RM. National inpatient hospital costs: the most expensive conditions by payer, 2011: statistical brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
  2. Hall MJ, Williams SN, DeFrances CJ, Golosinskiy A. Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011;(62):1–8.
  3. Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348(16):1546–1554.
  4. Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580–637.
  5. Rivers E, Nguyen B, Havstad S, et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377.
  6. Pro CI, Yealy DM, Kellum JA, et al. A randomized trial of protocol‐based care for early septic shock. N Engl J Med. 2014;370(18):1683–1693.
  7. Turi SK, Ah D. Implementation of early goal‐directed therapy for septic patients in the emergency department: a review of the literature. J Emerg Nurs. 2013;39(1):13–19.
  8. Fong JJ, Cecere K, Unterborn J, Garpestad E, Klee M, Devlin JW. Factors influencing variability in compliance rates and clinical outcomes among three different severe sepsis bundles. Ann Pharmacother. 2007;41(6):929–936.
  9. Ferrer R, Artigas A, Levy MM, et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299(19):2294–2303.
  10. Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012;40(7):2096–2101.
  11. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS‐2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536.
  12. Owens DK, Lohr KN, Atkins D, et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health‐care program. J Clin Epidemiol. 2010;63(5):513–523.
  13. Meurer WJ, Smith BL, Losman ED, et al. Real‐time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):40–45.
  14. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real‐time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500–504.
  15. Nguyen SQ, Mwakalindile E, Booth JS, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.
  16. Thiel SW, Rosini JM, Shannon W, Doherty JA, Micek ST, Kollef MH. Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010;5(1):19–25.
  17. Berger T, Birnbaum A, Bijur P, Kuperman G, Gennis P. A Computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394–407.
  18. McRee L, Thanavaro JL, Moore K, Goldsmith M, Pasvogel A. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546–549.
  19. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real‐time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469–473.
  20. Brun‐Buisson C. The epidemiology of the systemic inflammatory response. Intensive Care Med. 2000;26(suppl 1):S64–S74.
  21. Nanji KC, Slight SP, Seger DL, et al. Overrides of medication‐related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(3):487–491.
  22. Siebig S, Kuhls S, Imhoff M, Gather U, Scholmerich J, Wrede CE. Intensive care unit alarms–how many do we need? Crit Care Med. 2010;38(2):451–456.
  23. Singal G, Currier P. How can we best use electronic data to find and treat the critically ill?*. Crit Care Med. 2012;40(7):2242–2243.
  24. Ash JS, Sittig DF, Dykstra R, et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform. 2010;160(pt 2):806–810.
  25. Wright A, Phansalkar S, Bloomrosen M, et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform. 2010;1(3):331–345.
Issue
Journal of Hospital Medicine - 10(6)
Issue
Journal of Hospital Medicine - 10(6)
Page Number
396-402
Page Number
396-402
Publications
Journal of Hospital Medicine
Publications
Journal of Hospital Medicine
Article Type
Article
Display Headline
Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: A systematic review
Display Headline
Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: A systematic review
Sections
Reviews
Article Source
© 2015 Society of Hospital Medicine
Disallow All Ads
Corresponding Author
Anil N. Makam, MD, MAS
Correspondence Location
Address for correspondence and reprint requests: Anil N. Makam, MD, 5323 Harry Hines Blvd., Dallas, TX 75390‐9169; Telephone: 214‐648‐3272; Fax: 314‐648‐3232; E‐mail: [email protected]
Content Gating
Gated (full article locked unless allowed per User)
Gating Strategy
First Peek Free
Article PDF Media
Media Files