Vincent Liu, MD, MS

METHODS

RESULTS

DISCUSSION

CONCLUSION

METHODS

RESULTS

DISCUSSION

CONCLUSION

METHODS

RESULTS

DISCUSSION

CONCLUSION

Patients who deteriorate outside highly monitored settings and who require unplanned transfer to the intensive care unit (ICU) are known to have high mortality and morbidity.[1, 2, 3, 4, 5] The notion that early detection of a deteriorating patient improves outcomes has intuitive appeal and is discussed in a large number of publications.[6, 7, 8, 9, 10] However, much less information is available on what should be done after early detection is made.[11] Existing literature on early warning systems (EWSs) does not provide enough detail to serve as a map for implementation. This lack of transparency is complicated by the fact that, although the comprehensive inpatient electronic medical record (EMR) now constitutes the central locus for clinical practice, much of the existing literature comes from research institutions that may employ home‐grown EMRs, not community hospitals that employ commercially available systems.

In this issue of the Journal of Hospital Medicine, we describe our efforts to bridge that gap by implementing an EWS in a pair of community hospitals. The EWS's development and its basic statistical and electronic infrastructure are described in the articles by Escobar and Dellinger and Escobar et al.[2, 12, 13] In this report, we focus on how we addressed clinicians' primary concern: What do we do when we get an alert? Because it is described in detail by Granich et al.[14] elsewhere in this issue of the Journal of Hospital Medicine, a critical component of our implementation process (ensuring that patient preferences with respect to supportive care are honored) is not discussed.

Our article is divided into the following sections: rationale, preimplementation preparatory work, workflow development, response protocols, challenges and key learnings, and concluding reflections.

RATIONALE

Much of the previous work on the implementation of alarm systems has focused on the statistics behind detection or on the quantification of processes (eg, how many rapid response calls were triggered) or on outcomes such as mortality. The conceptual underpinnings and practical steps necessary for successful integration of an alarm system into the clinicians' workflow have not been articulated. Our theoretical framework was based on (1) improving situational awareness[15] (knowing what is going on around you and what is likely to happen next) and (2) mitigating cognitive errors.

An EWS enhances situational awareness most directly by earlier identification of a problem with a particular patient. As is detailed by Escobar et al.[16] in this issue of the Journal of Hospital Medicine, our EWS extracts EMR data every 6 hours, performs multiple calculations, and then displays 3 scores in real time in the inpatient dashboard (known as the Patient Lists activity in the Epic EMR). The first of these scores is the Laboratory‐Based Acute Physiologic Score, version 2 (LAPS2), an objective severity score whose retrospective version is already in use in Kaiser Permanente Northern California (KPNC) for internal benchmarking.[13] This score captures a patient's overall degree of physiologic instability within the preceding 72 hours. The second is the Comorbidity Point Score, version 2 (COPS2), a longitudinal comorbidity score based on the patient's diagnoses over the preceding 12 months.[13] This score captures a patient's overall comorbidity burden. Thus, it is possible for a patient to be very ill (high COPS2) while also being stable (low LAPS2) or vice versa. Both of these scores have other uses, including prediction of rehospitalization risk in real time,[17] which is also being piloted at KPNC. Finally, the Advanced Alert Monitoring (AAM) score, which integrates the LAPS2 and COPS2 with other variables, provides a 12‐hour deterioration risk, with a threshold value of 8% triggering response protocols. At or above this threshold, which was agreed to prior to implementation, the system achieves 25% sensitivity, 98% specificity, with a number needed to evaluate of 10 to 12, a level of workload that was felt to be acceptable by clinicians. Actions triggered by the EWS may be quite different from those one would take when being notified of a code blue, which is called at the time an event occurs. The EWS focuses attention on patients who might be missed because they do not yet appear critically ill. It also provides a shared, quantifiable measure of a patient's risk that can trigger a standardized plan of action to follow in evaluating and treating a patient.[15]

In addition to enhancing situational awareness, we intended the alarms to produce cognitive change in practitioners. Our goal was to replace medical intuition with analytic, evidence‐based judgment of future illness. We proceeded with the understanding that replacing quick intuition with slower analytic response is an essential skill in developing sound clinical reasoning.[18, 19, 20] The alert encourages physicians to reassess high‐risk patients facilitating a cognitive shift from automatic, error‐prone processing to slower, deliberate processing. Given the busy pace of ward work, slowing down permits clinicians to reassess previously overlooked details. Related to this process of inducing cognitive change is a secondary effect: we uncovered and discussed physician biases. Physicians are subject to potential biases that allow patients to deteriorate.[18, 19, 20] Therefore, we addressed bias through education. By reviewing particular cases of unanticipated deterioration at each hospital facility, we provided evidence for the problem of in‐hospital deterioration. This framed the new tool as an opportunity for improving treatment and encouraged physicians to act on the alert using a structured process.

INTERVENTIONS

Preimplementation Preparatory Work

Initial KPNC data provided strong support for the generally accepted notion that unplanned transfer patients have poor outcomes.[2, 4, 5] However, published reports failed to provide the granular detail clinicians need to implement a response arm at the unit and patient level. In preparation for going live, we conducted a retrospective chart review. This included data from patients hospitalized from January 1, 2011 through December 31, 2012 (additional detail is provided in the Supporting Information, Appendix, in the online version of this article). The key findings from our internal review of subjective documentation preceding deterioration are similar to those described in the literature and summarized in Figure 1, which displays the 5 most common clinical presentations associated with unplanned transfers.

Figure 1

The chart review served several major roles. First, it facilitated cognitive change by eliminating the notion that it can't happen here. Second, it provided considerable guidance on key clinical components that had to be incorporated into the workflow. Third, it engaged the rapid response team (RRT) in reviewing our work retrospectively to identify future opportunities. Finally, the review provided considerable guidance with respect to structuring documentation requirements.

As a result of the above efforts, other processes detailed below, and knowledge described in several of the companion articles in this issue of the Journal of Hospital Medicine, 3 critical elements, which had been explicitly required by our leadership, were in place prior to the go‐live date: a general consensus among hospitalists and nurses that this would be worth testing, a basic clinical response workflow, and an automated checklist for documentation. We refined these in a 2‐week shadowing phase preceding the start date. In this phase, the alerts were not displayed in the EMR. Instead, programmers working on the project notified selected physician leaders by phone. This permitted them to understand exactly what sort of patients were reaching the physiologic threshold so that they could better prepare both RRT registered nurses (RNs) and hospitalists for the go‐live date. This also provided an opportunity to begin refining the documentation process using actual patients.

The original name for our project was Early Detection of Impending Physiologic Deterioration. However, during the preparatory phase, consultation with our public relations staff led to a concern that the name could be frightening to some patients. This highlights the need to consider patient perceptions and how words used in 1 way by physicians can have different connotations to nonclinicians. Consequently, the system was renamed, and it is now referred to as Advance Alert Monitoring (AAM).

Workflow Development

We carefully examined the space where electronic data, graphical user interfaces, and clinical practice blend, a nexus now commonly referred to as workflow or user experience.[21] To promote situational awareness and effect cognitive change, we utilized the Institute for Health Care Improvement's Plan‐Do‐Study‐Act model.[22, 23] We then facilitated the iterative development of a clinician‐endorsed workflow.[22, 23, 24, 25] By adjusting the workflow based on ongoing experience and giving clinicians multiple opportunities to revise (a process that continues to date), we ensured clinicians would approach and endorse the alarm system as a useful tool for decision support.

Table 1 summarizes the work groups assembled for our implementation, and Table 2 provides a system‐oriented checklist indicating key components that need to be in place prior to having an early warning system go live in a hospital. Figure 2 summarizes the alert response protocols we developed through an iterative process at the 2 pilot sites. The care path shown in Figure 2 is the result of considerable revision, mostly due to actual experience acquired following the go live date. The diagram also includes a component that is still work in progress. This is how an emergency department probability estimate (triage support) will be integrated into both the ward as well as the ICU workflows. Although this is beyond the scope of this article, other hospitals may be experimenting with triage support (eg, for sepsis patients), so it is important to consider how one would incorporate such support into workflows.

Table 1.Workgroups Established for Early Warning System Rollout

Workgroup	Goals
NOTE: Abbreviations: POLST, physician orders for life‐sustaining treatment.
Clinical checklist	Perform structured chart review of selected unplanned transfer patients and near misses
	Develop a checklist for mitigation strategies given an alert
Develop documentation standards given an alert
Develop escalation protocol given an alert
Workload and threshold	Determine threshold for sensitivity of alerts and resulting impact on clinician workload
Patient preferences	Prepare background information to be presented to providers regarding end‐of‐life care and POLST orders
	Coordinate with clinical checklist workgroup to generate documentation templates that provide guidance for appropriate management of patients regarding preferences on escalation of care and end‐of‐life care
Electronic medical record coordination	Review proposed electronic medical record changes
	Make recommendation for further changes as needed
Develop plan for rollout of new and/or revised electronic record tools
Designate contact list for questions/emssues that may arise regarding electronic record changes during the pilot
Determine alert display choices and mode of alert notification
Nursing committee	Review staffing needs in anticipation of alert
	Coordinate with workload and threshold group
Develop training calendar to ensure skills necessary for successful implementation of alerts
Make recommendations for potential modification of rapid response team's role in development of a clinical checklist for nurses responding to an alert
Design educational materials for clinicians
Local communication strategy	Develop internal communication plan (for clinical staff not directly involved with pilot)
	Develop external communication plan (for nonclinicians who may hear about the project)

Table 2.Hospital System‐Wide Go Live Checklist

Level	Tasks
Administration	Obtain executive committee approval
	Establish communication protocols with quality assurance and quality improvement committees
Review protocols with medicallegal department
Communication	Write media material for patients and families
	Develop and disseminate scripts for front‐line staff
Develop communication and meet with all relevant front‐line staff on merits of project
Educate all staff on workflow changes and impacts
Clinical preparation	Conduct internal review of unplanned transfers and present results to all clinicians
	Determine service level agreements, ownership of at‐risk patients, who will access alerts
Conduct staff meetings to educate staff
Perform debriefs on relevant cases
Determine desired outcomes, process measures, balancing measures
Determine acceptable clinician burden (alerts/day)
Technology	Establish documentation templates
	Ensure access to new data fields (electronic medical record security process must be followed for access rights)
Workflows	Workflows (clinical response, patient preferences, supportive care, communication, documentation) must be in place prior to actual go live
Shadowing	Testing period (alerts communicated to selected clinicians prior to going live) should occur

Figure 2

RESPONSE PROTOCOLS

At South San Francisco, the RRT consists of an ICU nurse, a respiratory care therapist, and a designated hospitalist; at Sacramento, the team is also augmented by an additional nurse (the house supervisor). In addition to responding to the AAM alerts, RRT nurses respond to other emergency calls such as code blues, stroke alerts, and patient or patient‐familyinitiated rapid response calls. They also expedite time sensitive workups and treatments. They check up on recent transfers from the ICU to ensure continued improvement justifying staying on the ward. Serving as peer educators, they assist with processes such as chest tube or central line insertions, troubleshoot high‐risk medication administration, and ensure that treatment bundles (eg, for sepsis) occur expeditiously.

The RRT reviews EWS scores every 6 hours. The AAM score is seen as soon as providers open the chart, which helps triage patients for evaluation. Because patients can still be at risk even without an elevated AAM score, all normal escalation pathways remain in place. Once an alert is noted in the inpatient dashboard, the RRT nurse obtains a fresh set of vital signs, assesses the patient's clinical status, and informs the physician, social worker, and primary nurse (Figure 2). Team members work with the bedside nurse, providing support with assessment, interventions, plans, and follow‐up. Once advised of the alert, the hospitalist performs a second chart review and evaluates the patient at the bedside to identify factors that could underlie potential deterioration. After this evaluation, the hospitalist documents concerns, orders appropriate interventions (which can include escalation), and determines appropriate follow‐up. We made sure the team knew that respiratory distress, arrhythmias, mental status changes, or worsening infection were responsible for over 80% of in‐hospital deterioration cases. We also involved palliative care earlier in patient care, streamlining the process so the RRT makes just 1 phone call to the social worker, who contacts the palliative care physician and nurse to ensure patients have a designated surrogate in the event of further deterioration.

Our initial documentation template consisted of a comprehensive organ system‐based physician checklist. However, although this was of use to covering physicians unfamiliar with a given patient, it was redundant and annoying to attending providers already familiar with the patient. After more than 30 iterations, we settled on a succinct note that only documented the clinicians' clinical judgment as to what constituted the major risk for deterioration and what the mitigation strategies would be. Both of these judgments are in a checklist format (see Supporting Information, Appendix, in the online version of this article for the components of the physician and nurse notes).

Prior to the implementation of the system, RRT nurses performed proactive rounding by manually checking patient labs and vital signs, an inefficient process due to the poor sensitivity and specificity of individual values. Following implementation of the system, RRT RNs and clinicians switched to sorting patients by the 3 scores (COPS2, LAPS2, AAM). For example, patients may be stable at admission (as evidenced by their AAM score) but be at high risk due to their comorbidities. One approach that has been employed is to proactively check such patients to ensure they have a care directive in place, as is described in the article by Granich et al.[14] The Supportive Care Team (detailed in Granich et al.) assesses needs for palliative care and provides in‐hospital consultation as needed. Social services staff perform chart reviews to ensure a patient surrogate has been defined and also works with patients and their families to clarify goals of care.

CHALLENGES AND KEY LEARNINGS

One challenge that arose was reconciling the periodic nature of the alert (every 6 hours) with physicians' availability, which varied due to different rounding workflows at the 2 sites. Consequently, the alert cycle was changed; at the first site, the cycle was set to 1000‐1600‐2200‐0400, whereas the second site chose 0800‐1400‐2000‐0200.

One essential but problematic component of the clinical response is the issue of documentation. Inadequate documentation could lead to adverse outcomes, clinician malpractice exposure, and placing the entire hospital at risk for enterprise liability when clinical responses are not documented. This issue is complicated by the fact that overzealous efforts could lead to less or no documentation by making it too onerous for busy clinicians. We found that the ease with which data can populate progress notes in the EMR can lead to note bloat. Clearly, no documentation is not enough, and a complete history and physical is too much. Paradoxically, 1 of the issues underlying our problems with documentation was the proactive nature of the alerts themselves; because they are based on an outcome prediction in the next 12 hours, documenting the response to them may lack (perceived) urgency.

Shortly after the system went live, a patient who had been recently transferred out to the ward from the ICU triggered an alert. As a response was mounted, the team realized that existing ward protocols did not specify which physician service (intensivist or hospitalist) was responsible for patients who were transitioning from 1 unit to another. We also had to perform multiple revisions of the protocols specifying how alerts were handled when they occurred at times of change of shift. Eventually, we settled on having the combination of a hospitalist and an RRT nurse as the cornerstone of the response, with the hospitalist service as the primary owner of the entire process, but this arrangement might need to be varied in different settings. As a result of the experience with the pilot, the business case for deployment in the remaining 19 hospitals includes a formal budget request so that all have properly staffed RRTs, although the issue of primary ownership of the alert process for different patient types (eg, surgical patients) will be decided on a hospital‐by‐hospital basis. These experiences raise the intriguing possibility that implementation of alert systems can lead to the identification of systemic gaps in existing protocols. These gaps can include specific components of the hospital service agreements between multiple departments (emergency, hospital medicine, ICU, palliative care, surgery) as well as problems with existing workflows.

In addition to ongoing tweaking of care protocols, 3 issues remain unresolved. First is the issue of documentation. The current documentation notes are not completely satisfactory, and we are working with the KPNC EMR administrators to refine the tool. Desirable refinements include (1) having the system scores populate in more accessible sectors of the EMR where their retrieval will facilitate increased automation of the note writing process, (2) changing the note type to a note that will facilitate process audits, and (3) linking the note to other EMR tools so that the response arm can be tracked more formally. The second issue is the need to develop strategies to address staff turnover; for example, newer staff may not have received the same degree of exposure to the system as those who were there when it was started. Finally, due to limited resources, we have done very limited work on more mechanistic analyses of the clinical response itself. For example, it would be desirable to perform a formal quantitative, risk‐adjusted process‐outcome analysis of why some patients' outcomes are better than others following an alert.

Finally, it is also the case that we have had some unexpected occurrences that hint at new uses and benefits of alert systems. One of these is the phenomenon of chasing the alert. Some clinicians, on their own, have taken a more proactive stance in the care of patients in whom the AAM score is rising or near the alert threshold. This has 2 potential consequences. Some patients are stabilized and thus do not reach threshold instability levels. In other cases, patients reach threshold but the response team is informed that things are already under control. A second unexpected result is increased requests for COPS2 scores by clinicians who have heard about the system, particularly surgeons who would like to use the comorbidity scores as a screening tool in the outpatient setting. Because KPNC is an integrated system, it is not likely that such alternatives will be implemented immediately without considerable analysis, but it is clear that the system's deployment has captured the clinicians' imagination.

CONCLUSIONS AND FUTURE DIRECTIONS

Our preparatory efforts have been successful. We have found that embedding an EWS in a commercially available EMR is acceptable to hospital physicians and nurses. We have developed a coordinated workflow for mitigation and escalation that is tightly linked to the availability of probabilistic alerts in real time. Although resource limitations have precluded us from conducting formal clinician surveys, the EWS has been discussed at multiple hospital‐wide as well as department‐specific meetings. Although there have been requests for clarification, refinements, and modifications in workflows, no one has suggested that the system be discontinued. Further, many of the other KPNC hospitals have requested that the EWS be deployed at their site. We have examined KPNC databases that track patient complaints and have not found any complaints that could be linked to the EWS. Most importantly, the existence of the workflows we have developed has played a major role in KPNC's decision to deploy the system in its remaining hospitals.

Although alert fatigue is the number 1 reason that clinicians do not utilize embedded clinical decision support,[26] simply calibrating statistical models is insufficient. Careful consideration of clinicians' needs and responsibilities, particularly around ownership of patients and documentation, is essential. Such consideration needs to include planning time and socializing the system (providing multiple venues for clinicians to learn about the system as well as participate in the process for using it).

We anticipate that, as the system leaves the pilot stage and becomes a routine component of hospital care, additional enhancements (eg, sending notifications to smart phones, providing an alert response tracking system) will be added. Our organization is also implementing real‐time concurrent review of inpatient EMRs (eg, for proactive detection of an expanded range of potential process failures), and work is underway on how to link the workflows we describe here with this effort. As has been the case with other systems,[27] it is likely that we will eventually move to continuous scanning of patient data rather than only every 6 hours. Given that the basic workflow is quite robust and amenable to local modifications, we are confident that our clinicians and hospitals will adapt to future system enhancements.

Lastly, we intend to conduct additional research on the clinical response itself. In particular, we consider it extremely important to conduct formal quantitative analyses on why some patients' outcomes are better than others following an alert. A key component of this effort will be to develop tools that can permit an automatedor nearly automatedassessment of the clinical response. For example, we are considering automated approaches that would scan the EMR for the presence of specific orders, notes, vital signs patterns, and laboratory tests following an alert. Whereas it may not be possible to dispense with manual chart review, even partial automation of a feedback process could lead to significant enhancement of our quality improvement efforts.

Patients who deteriorate outside highly monitored settings and who require unplanned transfer to the intensive care unit (ICU) are known to have high mortality and morbidity.[1, 2, 3, 4, 5] The notion that early detection of a deteriorating patient improves outcomes has intuitive appeal and is discussed in a large number of publications.[6, 7, 8, 9, 10] However, much less information is available on what should be done after early detection is made.[11] Existing literature on early warning systems (EWSs) does not provide enough detail to serve as a map for implementation. This lack of transparency is complicated by the fact that, although the comprehensive inpatient electronic medical record (EMR) now constitutes the central locus for clinical practice, much of the existing literature comes from research institutions that may employ home‐grown EMRs, not community hospitals that employ commercially available systems.

In this issue of the Journal of Hospital Medicine, we describe our efforts to bridge that gap by implementing an EWS in a pair of community hospitals. The EWS's development and its basic statistical and electronic infrastructure are described in the articles by Escobar and Dellinger and Escobar et al.[2, 12, 13] In this report, we focus on how we addressed clinicians' primary concern: What do we do when we get an alert? Because it is described in detail by Granich et al.[14] elsewhere in this issue of the Journal of Hospital Medicine, a critical component of our implementation process (ensuring that patient preferences with respect to supportive care are honored) is not discussed.

Our article is divided into the following sections: rationale, preimplementation preparatory work, workflow development, response protocols, challenges and key learnings, and concluding reflections.

RATIONALE

Much of the previous work on the implementation of alarm systems has focused on the statistics behind detection or on the quantification of processes (eg, how many rapid response calls were triggered) or on outcomes such as mortality. The conceptual underpinnings and practical steps necessary for successful integration of an alarm system into the clinicians' workflow have not been articulated. Our theoretical framework was based on (1) improving situational awareness[15] (knowing what is going on around you and what is likely to happen next) and (2) mitigating cognitive errors.

An EWS enhances situational awareness most directly by earlier identification of a problem with a particular patient. As is detailed by Escobar et al.[16] in this issue of the Journal of Hospital Medicine, our EWS extracts EMR data every 6 hours, performs multiple calculations, and then displays 3 scores in real time in the inpatient dashboard (known as the Patient Lists activity in the Epic EMR). The first of these scores is the Laboratory‐Based Acute Physiologic Score, version 2 (LAPS2), an objective severity score whose retrospective version is already in use in Kaiser Permanente Northern California (KPNC) for internal benchmarking.[13] This score captures a patient's overall degree of physiologic instability within the preceding 72 hours. The second is the Comorbidity Point Score, version 2 (COPS2), a longitudinal comorbidity score based on the patient's diagnoses over the preceding 12 months.[13] This score captures a patient's overall comorbidity burden. Thus, it is possible for a patient to be very ill (high COPS2) while also being stable (low LAPS2) or vice versa. Both of these scores have other uses, including prediction of rehospitalization risk in real time,[17] which is also being piloted at KPNC. Finally, the Advanced Alert Monitoring (AAM) score, which integrates the LAPS2 and COPS2 with other variables, provides a 12‐hour deterioration risk, with a threshold value of 8% triggering response protocols. At or above this threshold, which was agreed to prior to implementation, the system achieves 25% sensitivity, 98% specificity, with a number needed to evaluate of 10 to 12, a level of workload that was felt to be acceptable by clinicians. Actions triggered by the EWS may be quite different from those one would take when being notified of a code blue, which is called at the time an event occurs. The EWS focuses attention on patients who might be missed because they do not yet appear critically ill. It also provides a shared, quantifiable measure of a patient's risk that can trigger a standardized plan of action to follow in evaluating and treating a patient.[15]

In addition to enhancing situational awareness, we intended the alarms to produce cognitive change in practitioners. Our goal was to replace medical intuition with analytic, evidence‐based judgment of future illness. We proceeded with the understanding that replacing quick intuition with slower analytic response is an essential skill in developing sound clinical reasoning.[18, 19, 20] The alert encourages physicians to reassess high‐risk patients facilitating a cognitive shift from automatic, error‐prone processing to slower, deliberate processing. Given the busy pace of ward work, slowing down permits clinicians to reassess previously overlooked details. Related to this process of inducing cognitive change is a secondary effect: we uncovered and discussed physician biases. Physicians are subject to potential biases that allow patients to deteriorate.[18, 19, 20] Therefore, we addressed bias through education. By reviewing particular cases of unanticipated deterioration at each hospital facility, we provided evidence for the problem of in‐hospital deterioration. This framed the new tool as an opportunity for improving treatment and encouraged physicians to act on the alert using a structured process.

INTERVENTIONS

Preimplementation Preparatory Work

Initial KPNC data provided strong support for the generally accepted notion that unplanned transfer patients have poor outcomes.[2, 4, 5] However, published reports failed to provide the granular detail clinicians need to implement a response arm at the unit and patient level. In preparation for going live, we conducted a retrospective chart review. This included data from patients hospitalized from January 1, 2011 through December 31, 2012 (additional detail is provided in the Supporting Information, Appendix, in the online version of this article). The key findings from our internal review of subjective documentation preceding deterioration are similar to those described in the literature and summarized in Figure 1, which displays the 5 most common clinical presentations associated with unplanned transfers.

Figure 1

The chart review served several major roles. First, it facilitated cognitive change by eliminating the notion that it can't happen here. Second, it provided considerable guidance on key clinical components that had to be incorporated into the workflow. Third, it engaged the rapid response team (RRT) in reviewing our work retrospectively to identify future opportunities. Finally, the review provided considerable guidance with respect to structuring documentation requirements.

As a result of the above efforts, other processes detailed below, and knowledge described in several of the companion articles in this issue of the Journal of Hospital Medicine, 3 critical elements, which had been explicitly required by our leadership, were in place prior to the go‐live date: a general consensus among hospitalists and nurses that this would be worth testing, a basic clinical response workflow, and an automated checklist for documentation. We refined these in a 2‐week shadowing phase preceding the start date. In this phase, the alerts were not displayed in the EMR. Instead, programmers working on the project notified selected physician leaders by phone. This permitted them to understand exactly what sort of patients were reaching the physiologic threshold so that they could better prepare both RRT registered nurses (RNs) and hospitalists for the go‐live date. This also provided an opportunity to begin refining the documentation process using actual patients.

The original name for our project was Early Detection of Impending Physiologic Deterioration. However, during the preparatory phase, consultation with our public relations staff led to a concern that the name could be frightening to some patients. This highlights the need to consider patient perceptions and how words used in 1 way by physicians can have different connotations to nonclinicians. Consequently, the system was renamed, and it is now referred to as Advance Alert Monitoring (AAM).

Workflow Development

We carefully examined the space where electronic data, graphical user interfaces, and clinical practice blend, a nexus now commonly referred to as workflow or user experience.[21] To promote situational awareness and effect cognitive change, we utilized the Institute for Health Care Improvement's Plan‐Do‐Study‐Act model.[22, 23] We then facilitated the iterative development of a clinician‐endorsed workflow.[22, 23, 24, 25] By adjusting the workflow based on ongoing experience and giving clinicians multiple opportunities to revise (a process that continues to date), we ensured clinicians would approach and endorse the alarm system as a useful tool for decision support.

Table 1 summarizes the work groups assembled for our implementation, and Table 2 provides a system‐oriented checklist indicating key components that need to be in place prior to having an early warning system go live in a hospital. Figure 2 summarizes the alert response protocols we developed through an iterative process at the 2 pilot sites. The care path shown in Figure 2 is the result of considerable revision, mostly due to actual experience acquired following the go live date. The diagram also includes a component that is still work in progress. This is how an emergency department probability estimate (triage support) will be integrated into both the ward as well as the ICU workflows. Although this is beyond the scope of this article, other hospitals may be experimenting with triage support (eg, for sepsis patients), so it is important to consider how one would incorporate such support into workflows.

Table 1.Workgroups Established for Early Warning System Rollout

Workgroup	Goals
NOTE: Abbreviations: POLST, physician orders for life‐sustaining treatment.
Clinical checklist	Perform structured chart review of selected unplanned transfer patients and near misses
	Develop a checklist for mitigation strategies given an alert
Develop documentation standards given an alert
Develop escalation protocol given an alert
Workload and threshold	Determine threshold for sensitivity of alerts and resulting impact on clinician workload
Patient preferences	Prepare background information to be presented to providers regarding end‐of‐life care and POLST orders
	Coordinate with clinical checklist workgroup to generate documentation templates that provide guidance for appropriate management of patients regarding preferences on escalation of care and end‐of‐life care
Electronic medical record coordination	Review proposed electronic medical record changes
	Make recommendation for further changes as needed
Develop plan for rollout of new and/or revised electronic record tools
Designate contact list for questions/emssues that may arise regarding electronic record changes during the pilot
Determine alert display choices and mode of alert notification
Nursing committee	Review staffing needs in anticipation of alert
	Coordinate with workload and threshold group
Develop training calendar to ensure skills necessary for successful implementation of alerts
Make recommendations for potential modification of rapid response team's role in development of a clinical checklist for nurses responding to an alert
Design educational materials for clinicians
Local communication strategy	Develop internal communication plan (for clinical staff not directly involved with pilot)
	Develop external communication plan (for nonclinicians who may hear about the project)

Table 2.Hospital System‐Wide Go Live Checklist

Level	Tasks
Administration	Obtain executive committee approval
	Establish communication protocols with quality assurance and quality improvement committees
Review protocols with medicallegal department
Communication	Write media material for patients and families
	Develop and disseminate scripts for front‐line staff
Develop communication and meet with all relevant front‐line staff on merits of project
Educate all staff on workflow changes and impacts
Clinical preparation	Conduct internal review of unplanned transfers and present results to all clinicians
	Determine service level agreements, ownership of at‐risk patients, who will access alerts
Conduct staff meetings to educate staff
Perform debriefs on relevant cases
Determine desired outcomes, process measures, balancing measures
Determine acceptable clinician burden (alerts/day)
Technology	Establish documentation templates
	Ensure access to new data fields (electronic medical record security process must be followed for access rights)
Workflows	Workflows (clinical response, patient preferences, supportive care, communication, documentation) must be in place prior to actual go live
Shadowing	Testing period (alerts communicated to selected clinicians prior to going live) should occur

Figure 2

RESPONSE PROTOCOLS

At South San Francisco, the RRT consists of an ICU nurse, a respiratory care therapist, and a designated hospitalist; at Sacramento, the team is also augmented by an additional nurse (the house supervisor). In addition to responding to the AAM alerts, RRT nurses respond to other emergency calls such as code blues, stroke alerts, and patient or patient‐familyinitiated rapid response calls. They also expedite time sensitive workups and treatments. They check up on recent transfers from the ICU to ensure continued improvement justifying staying on the ward. Serving as peer educators, they assist with processes such as chest tube or central line insertions, troubleshoot high‐risk medication administration, and ensure that treatment bundles (eg, for sepsis) occur expeditiously.

The RRT reviews EWS scores every 6 hours. The AAM score is seen as soon as providers open the chart, which helps triage patients for evaluation. Because patients can still be at risk even without an elevated AAM score, all normal escalation pathways remain in place. Once an alert is noted in the inpatient dashboard, the RRT nurse obtains a fresh set of vital signs, assesses the patient's clinical status, and informs the physician, social worker, and primary nurse (Figure 2). Team members work with the bedside nurse, providing support with assessment, interventions, plans, and follow‐up. Once advised of the alert, the hospitalist performs a second chart review and evaluates the patient at the bedside to identify factors that could underlie potential deterioration. After this evaluation, the hospitalist documents concerns, orders appropriate interventions (which can include escalation), and determines appropriate follow‐up. We made sure the team knew that respiratory distress, arrhythmias, mental status changes, or worsening infection were responsible for over 80% of in‐hospital deterioration cases. We also involved palliative care earlier in patient care, streamlining the process so the RRT makes just 1 phone call to the social worker, who contacts the palliative care physician and nurse to ensure patients have a designated surrogate in the event of further deterioration.

Our initial documentation template consisted of a comprehensive organ system‐based physician checklist. However, although this was of use to covering physicians unfamiliar with a given patient, it was redundant and annoying to attending providers already familiar with the patient. After more than 30 iterations, we settled on a succinct note that only documented the clinicians' clinical judgment as to what constituted the major risk for deterioration and what the mitigation strategies would be. Both of these judgments are in a checklist format (see Supporting Information, Appendix, in the online version of this article for the components of the physician and nurse notes).

Prior to the implementation of the system, RRT nurses performed proactive rounding by manually checking patient labs and vital signs, an inefficient process due to the poor sensitivity and specificity of individual values. Following implementation of the system, RRT RNs and clinicians switched to sorting patients by the 3 scores (COPS2, LAPS2, AAM). For example, patients may be stable at admission (as evidenced by their AAM score) but be at high risk due to their comorbidities. One approach that has been employed is to proactively check such patients to ensure they have a care directive in place, as is described in the article by Granich et al.[14] The Supportive Care Team (detailed in Granich et al.) assesses needs for palliative care and provides in‐hospital consultation as needed. Social services staff perform chart reviews to ensure a patient surrogate has been defined and also works with patients and their families to clarify goals of care.

CHALLENGES AND KEY LEARNINGS

One challenge that arose was reconciling the periodic nature of the alert (every 6 hours) with physicians' availability, which varied due to different rounding workflows at the 2 sites. Consequently, the alert cycle was changed; at the first site, the cycle was set to 1000‐1600‐2200‐0400, whereas the second site chose 0800‐1400‐2000‐0200.

One essential but problematic component of the clinical response is the issue of documentation. Inadequate documentation could lead to adverse outcomes, clinician malpractice exposure, and placing the entire hospital at risk for enterprise liability when clinical responses are not documented. This issue is complicated by the fact that overzealous efforts could lead to less or no documentation by making it too onerous for busy clinicians. We found that the ease with which data can populate progress notes in the EMR can lead to note bloat. Clearly, no documentation is not enough, and a complete history and physical is too much. Paradoxically, 1 of the issues underlying our problems with documentation was the proactive nature of the alerts themselves; because they are based on an outcome prediction in the next 12 hours, documenting the response to them may lack (perceived) urgency.

Shortly after the system went live, a patient who had been recently transferred out to the ward from the ICU triggered an alert. As a response was mounted, the team realized that existing ward protocols did not specify which physician service (intensivist or hospitalist) was responsible for patients who were transitioning from 1 unit to another. We also had to perform multiple revisions of the protocols specifying how alerts were handled when they occurred at times of change of shift. Eventually, we settled on having the combination of a hospitalist and an RRT nurse as the cornerstone of the response, with the hospitalist service as the primary owner of the entire process, but this arrangement might need to be varied in different settings. As a result of the experience with the pilot, the business case for deployment in the remaining 19 hospitals includes a formal budget request so that all have properly staffed RRTs, although the issue of primary ownership of the alert process for different patient types (eg, surgical patients) will be decided on a hospital‐by‐hospital basis. These experiences raise the intriguing possibility that implementation of alert systems can lead to the identification of systemic gaps in existing protocols. These gaps can include specific components of the hospital service agreements between multiple departments (emergency, hospital medicine, ICU, palliative care, surgery) as well as problems with existing workflows.

In addition to ongoing tweaking of care protocols, 3 issues remain unresolved. First is the issue of documentation. The current documentation notes are not completely satisfactory, and we are working with the KPNC EMR administrators to refine the tool. Desirable refinements include (1) having the system scores populate in more accessible sectors of the EMR where their retrieval will facilitate increased automation of the note writing process, (2) changing the note type to a note that will facilitate process audits, and (3) linking the note to other EMR tools so that the response arm can be tracked more formally. The second issue is the need to develop strategies to address staff turnover; for example, newer staff may not have received the same degree of exposure to the system as those who were there when it was started. Finally, due to limited resources, we have done very limited work on more mechanistic analyses of the clinical response itself. For example, it would be desirable to perform a formal quantitative, risk‐adjusted process‐outcome analysis of why some patients' outcomes are better than others following an alert.

Finally, it is also the case that we have had some unexpected occurrences that hint at new uses and benefits of alert systems. One of these is the phenomenon of chasing the alert. Some clinicians, on their own, have taken a more proactive stance in the care of patients in whom the AAM score is rising or near the alert threshold. This has 2 potential consequences. Some patients are stabilized and thus do not reach threshold instability levels. In other cases, patients reach threshold but the response team is informed that things are already under control. A second unexpected result is increased requests for COPS2 scores by clinicians who have heard about the system, particularly surgeons who would like to use the comorbidity scores as a screening tool in the outpatient setting. Because KPNC is an integrated system, it is not likely that such alternatives will be implemented immediately without considerable analysis, but it is clear that the system's deployment has captured the clinicians' imagination.

CONCLUSIONS AND FUTURE DIRECTIONS

Our preparatory efforts have been successful. We have found that embedding an EWS in a commercially available EMR is acceptable to hospital physicians and nurses. We have developed a coordinated workflow for mitigation and escalation that is tightly linked to the availability of probabilistic alerts in real time. Although resource limitations have precluded us from conducting formal clinician surveys, the EWS has been discussed at multiple hospital‐wide as well as department‐specific meetings. Although there have been requests for clarification, refinements, and modifications in workflows, no one has suggested that the system be discontinued. Further, many of the other KPNC hospitals have requested that the EWS be deployed at their site. We have examined KPNC databases that track patient complaints and have not found any complaints that could be linked to the EWS. Most importantly, the existence of the workflows we have developed has played a major role in KPNC's decision to deploy the system in its remaining hospitals.

Although alert fatigue is the number 1 reason that clinicians do not utilize embedded clinical decision support,[26] simply calibrating statistical models is insufficient. Careful consideration of clinicians' needs and responsibilities, particularly around ownership of patients and documentation, is essential. Such consideration needs to include planning time and socializing the system (providing multiple venues for clinicians to learn about the system as well as participate in the process for using it).

We anticipate that, as the system leaves the pilot stage and becomes a routine component of hospital care, additional enhancements (eg, sending notifications to smart phones, providing an alert response tracking system) will be added. Our organization is also implementing real‐time concurrent review of inpatient EMRs (eg, for proactive detection of an expanded range of potential process failures), and work is underway on how to link the workflows we describe here with this effort. As has been the case with other systems,[27] it is likely that we will eventually move to continuous scanning of patient data rather than only every 6 hours. Given that the basic workflow is quite robust and amenable to local modifications, we are confident that our clinicians and hospitals will adapt to future system enhancements.

Lastly, we intend to conduct additional research on the clinical response itself. In particular, we consider it extremely important to conduct formal quantitative analyses on why some patients' outcomes are better than others following an alert. A key component of this effort will be to develop tools that can permit an automatedor nearly automatedassessment of the clinical response. For example, we are considering automated approaches that would scan the EMR for the presence of specific orders, notes, vital signs patterns, and laboratory tests following an alert. Whereas it may not be possible to dispense with manual chart review, even partial automation of a feedback process could lead to significant enhancement of our quality improvement efforts.

Patients who deteriorate outside highly monitored settings and who require unplanned transfer to the intensive care unit (ICU) are known to have high mortality and morbidity.[1, 2, 3, 4, 5] The notion that early detection of a deteriorating patient improves outcomes has intuitive appeal and is discussed in a large number of publications.[6, 7, 8, 9, 10] However, much less information is available on what should be done after early detection is made.[11] Existing literature on early warning systems (EWSs) does not provide enough detail to serve as a map for implementation. This lack of transparency is complicated by the fact that, although the comprehensive inpatient electronic medical record (EMR) now constitutes the central locus for clinical practice, much of the existing literature comes from research institutions that may employ home‐grown EMRs, not community hospitals that employ commercially available systems.

In this issue of the Journal of Hospital Medicine, we describe our efforts to bridge that gap by implementing an EWS in a pair of community hospitals. The EWS's development and its basic statistical and electronic infrastructure are described in the articles by Escobar and Dellinger and Escobar et al.[2, 12, 13] In this report, we focus on how we addressed clinicians' primary concern: What do we do when we get an alert? Because it is described in detail by Granich et al.[14] elsewhere in this issue of the Journal of Hospital Medicine, a critical component of our implementation process (ensuring that patient preferences with respect to supportive care are honored) is not discussed.

Our article is divided into the following sections: rationale, preimplementation preparatory work, workflow development, response protocols, challenges and key learnings, and concluding reflections.

RATIONALE

Much of the previous work on the implementation of alarm systems has focused on the statistics behind detection or on the quantification of processes (eg, how many rapid response calls were triggered) or on outcomes such as mortality. The conceptual underpinnings and practical steps necessary for successful integration of an alarm system into the clinicians' workflow have not been articulated. Our theoretical framework was based on (1) improving situational awareness[15] (knowing what is going on around you and what is likely to happen next) and (2) mitigating cognitive errors.

An EWS enhances situational awareness most directly by earlier identification of a problem with a particular patient. As is detailed by Escobar et al.[16] in this issue of the Journal of Hospital Medicine, our EWS extracts EMR data every 6 hours, performs multiple calculations, and then displays 3 scores in real time in the inpatient dashboard (known as the Patient Lists activity in the Epic EMR). The first of these scores is the Laboratory‐Based Acute Physiologic Score, version 2 (LAPS2), an objective severity score whose retrospective version is already in use in Kaiser Permanente Northern California (KPNC) for internal benchmarking.[13] This score captures a patient's overall degree of physiologic instability within the preceding 72 hours. The second is the Comorbidity Point Score, version 2 (COPS2), a longitudinal comorbidity score based on the patient's diagnoses over the preceding 12 months.[13] This score captures a patient's overall comorbidity burden. Thus, it is possible for a patient to be very ill (high COPS2) while also being stable (low LAPS2) or vice versa. Both of these scores have other uses, including prediction of rehospitalization risk in real time,[17] which is also being piloted at KPNC. Finally, the Advanced Alert Monitoring (AAM) score, which integrates the LAPS2 and COPS2 with other variables, provides a 12‐hour deterioration risk, with a threshold value of 8% triggering response protocols. At or above this threshold, which was agreed to prior to implementation, the system achieves 25% sensitivity, 98% specificity, with a number needed to evaluate of 10 to 12, a level of workload that was felt to be acceptable by clinicians. Actions triggered by the EWS may be quite different from those one would take when being notified of a code blue, which is called at the time an event occurs. The EWS focuses attention on patients who might be missed because they do not yet appear critically ill. It also provides a shared, quantifiable measure of a patient's risk that can trigger a standardized plan of action to follow in evaluating and treating a patient.[15]

In addition to enhancing situational awareness, we intended the alarms to produce cognitive change in practitioners. Our goal was to replace medical intuition with analytic, evidence‐based judgment of future illness. We proceeded with the understanding that replacing quick intuition with slower analytic response is an essential skill in developing sound clinical reasoning.[18, 19, 20] The alert encourages physicians to reassess high‐risk patients facilitating a cognitive shift from automatic, error‐prone processing to slower, deliberate processing. Given the busy pace of ward work, slowing down permits clinicians to reassess previously overlooked details. Related to this process of inducing cognitive change is a secondary effect: we uncovered and discussed physician biases. Physicians are subject to potential biases that allow patients to deteriorate.[18, 19, 20] Therefore, we addressed bias through education. By reviewing particular cases of unanticipated deterioration at each hospital facility, we provided evidence for the problem of in‐hospital deterioration. This framed the new tool as an opportunity for improving treatment and encouraged physicians to act on the alert using a structured process.

INTERVENTIONS

Preimplementation Preparatory Work

Initial KPNC data provided strong support for the generally accepted notion that unplanned transfer patients have poor outcomes.[2, 4, 5] However, published reports failed to provide the granular detail clinicians need to implement a response arm at the unit and patient level. In preparation for going live, we conducted a retrospective chart review. This included data from patients hospitalized from January 1, 2011 through December 31, 2012 (additional detail is provided in the Supporting Information, Appendix, in the online version of this article). The key findings from our internal review of subjective documentation preceding deterioration are similar to those described in the literature and summarized in Figure 1, which displays the 5 most common clinical presentations associated with unplanned transfers.

Figure 1

The chart review served several major roles. First, it facilitated cognitive change by eliminating the notion that it can't happen here. Second, it provided considerable guidance on key clinical components that had to be incorporated into the workflow. Third, it engaged the rapid response team (RRT) in reviewing our work retrospectively to identify future opportunities. Finally, the review provided considerable guidance with respect to structuring documentation requirements.

As a result of the above efforts, other processes detailed below, and knowledge described in several of the companion articles in this issue of the Journal of Hospital Medicine, 3 critical elements, which had been explicitly required by our leadership, were in place prior to the go‐live date: a general consensus among hospitalists and nurses that this would be worth testing, a basic clinical response workflow, and an automated checklist for documentation. We refined these in a 2‐week shadowing phase preceding the start date. In this phase, the alerts were not displayed in the EMR. Instead, programmers working on the project notified selected physician leaders by phone. This permitted them to understand exactly what sort of patients were reaching the physiologic threshold so that they could better prepare both RRT registered nurses (RNs) and hospitalists for the go‐live date. This also provided an opportunity to begin refining the documentation process using actual patients.

The original name for our project was Early Detection of Impending Physiologic Deterioration. However, during the preparatory phase, consultation with our public relations staff led to a concern that the name could be frightening to some patients. This highlights the need to consider patient perceptions and how words used in 1 way by physicians can have different connotations to nonclinicians. Consequently, the system was renamed, and it is now referred to as Advance Alert Monitoring (AAM).

Workflow Development

We carefully examined the space where electronic data, graphical user interfaces, and clinical practice blend, a nexus now commonly referred to as workflow or user experience.[21] To promote situational awareness and effect cognitive change, we utilized the Institute for Health Care Improvement's Plan‐Do‐Study‐Act model.[22, 23] We then facilitated the iterative development of a clinician‐endorsed workflow.[22, 23, 24, 25] By adjusting the workflow based on ongoing experience and giving clinicians multiple opportunities to revise (a process that continues to date), we ensured clinicians would approach and endorse the alarm system as a useful tool for decision support.

Table 1 summarizes the work groups assembled for our implementation, and Table 2 provides a system‐oriented checklist indicating key components that need to be in place prior to having an early warning system go live in a hospital. Figure 2 summarizes the alert response protocols we developed through an iterative process at the 2 pilot sites. The care path shown in Figure 2 is the result of considerable revision, mostly due to actual experience acquired following the go live date. The diagram also includes a component that is still work in progress. This is how an emergency department probability estimate (triage support) will be integrated into both the ward as well as the ICU workflows. Although this is beyond the scope of this article, other hospitals may be experimenting with triage support (eg, for sepsis patients), so it is important to consider how one would incorporate such support into workflows.

Table 1.Workgroups Established for Early Warning System Rollout

Workgroup	Goals
NOTE: Abbreviations: POLST, physician orders for life‐sustaining treatment.
Clinical checklist	Perform structured chart review of selected unplanned transfer patients and near misses
	Develop a checklist for mitigation strategies given an alert
Develop documentation standards given an alert
Develop escalation protocol given an alert
Workload and threshold	Determine threshold for sensitivity of alerts and resulting impact on clinician workload
Patient preferences	Prepare background information to be presented to providers regarding end‐of‐life care and POLST orders
	Coordinate with clinical checklist workgroup to generate documentation templates that provide guidance for appropriate management of patients regarding preferences on escalation of care and end‐of‐life care
Electronic medical record coordination	Review proposed electronic medical record changes
	Make recommendation for further changes as needed
Develop plan for rollout of new and/or revised electronic record tools
Designate contact list for questions/emssues that may arise regarding electronic record changes during the pilot
Determine alert display choices and mode of alert notification
Nursing committee	Review staffing needs in anticipation of alert
	Coordinate with workload and threshold group
Develop training calendar to ensure skills necessary for successful implementation of alerts
Make recommendations for potential modification of rapid response team's role in development of a clinical checklist for nurses responding to an alert
Design educational materials for clinicians
Local communication strategy	Develop internal communication plan (for clinical staff not directly involved with pilot)
	Develop external communication plan (for nonclinicians who may hear about the project)

Table 2.Hospital System‐Wide Go Live Checklist

Level	Tasks
Administration	Obtain executive committee approval
	Establish communication protocols with quality assurance and quality improvement committees
Review protocols with medicallegal department
Communication	Write media material for patients and families
	Develop and disseminate scripts for front‐line staff
Develop communication and meet with all relevant front‐line staff on merits of project
Educate all staff on workflow changes and impacts
Clinical preparation	Conduct internal review of unplanned transfers and present results to all clinicians
	Determine service level agreements, ownership of at‐risk patients, who will access alerts
Conduct staff meetings to educate staff
Perform debriefs on relevant cases
Determine desired outcomes, process measures, balancing measures
Determine acceptable clinician burden (alerts/day)
Technology	Establish documentation templates
	Ensure access to new data fields (electronic medical record security process must be followed for access rights)
Workflows	Workflows (clinical response, patient preferences, supportive care, communication, documentation) must be in place prior to actual go live
Shadowing	Testing period (alerts communicated to selected clinicians prior to going live) should occur

Figure 2

RESPONSE PROTOCOLS

At South San Francisco, the RRT consists of an ICU nurse, a respiratory care therapist, and a designated hospitalist; at Sacramento, the team is also augmented by an additional nurse (the house supervisor). In addition to responding to the AAM alerts, RRT nurses respond to other emergency calls such as code blues, stroke alerts, and patient or patient‐familyinitiated rapid response calls. They also expedite time sensitive workups and treatments. They check up on recent transfers from the ICU to ensure continued improvement justifying staying on the ward. Serving as peer educators, they assist with processes such as chest tube or central line insertions, troubleshoot high‐risk medication administration, and ensure that treatment bundles (eg, for sepsis) occur expeditiously.

The RRT reviews EWS scores every 6 hours. The AAM score is seen as soon as providers open the chart, which helps triage patients for evaluation. Because patients can still be at risk even without an elevated AAM score, all normal escalation pathways remain in place. Once an alert is noted in the inpatient dashboard, the RRT nurse obtains a fresh set of vital signs, assesses the patient's clinical status, and informs the physician, social worker, and primary nurse (Figure 2). Team members work with the bedside nurse, providing support with assessment, interventions, plans, and follow‐up. Once advised of the alert, the hospitalist performs a second chart review and evaluates the patient at the bedside to identify factors that could underlie potential deterioration. After this evaluation, the hospitalist documents concerns, orders appropriate interventions (which can include escalation), and determines appropriate follow‐up. We made sure the team knew that respiratory distress, arrhythmias, mental status changes, or worsening infection were responsible for over 80% of in‐hospital deterioration cases. We also involved palliative care earlier in patient care, streamlining the process so the RRT makes just 1 phone call to the social worker, who contacts the palliative care physician and nurse to ensure patients have a designated surrogate in the event of further deterioration.

Our initial documentation template consisted of a comprehensive organ system‐based physician checklist. However, although this was of use to covering physicians unfamiliar with a given patient, it was redundant and annoying to attending providers already familiar with the patient. After more than 30 iterations, we settled on a succinct note that only documented the clinicians' clinical judgment as to what constituted the major risk for deterioration and what the mitigation strategies would be. Both of these judgments are in a checklist format (see Supporting Information, Appendix, in the online version of this article for the components of the physician and nurse notes).

Prior to the implementation of the system, RRT nurses performed proactive rounding by manually checking patient labs and vital signs, an inefficient process due to the poor sensitivity and specificity of individual values. Following implementation of the system, RRT RNs and clinicians switched to sorting patients by the 3 scores (COPS2, LAPS2, AAM). For example, patients may be stable at admission (as evidenced by their AAM score) but be at high risk due to their comorbidities. One approach that has been employed is to proactively check such patients to ensure they have a care directive in place, as is described in the article by Granich et al.[14] The Supportive Care Team (detailed in Granich et al.) assesses needs for palliative care and provides in‐hospital consultation as needed. Social services staff perform chart reviews to ensure a patient surrogate has been defined and also works with patients and their families to clarify goals of care.

CHALLENGES AND KEY LEARNINGS

One challenge that arose was reconciling the periodic nature of the alert (every 6 hours) with physicians' availability, which varied due to different rounding workflows at the 2 sites. Consequently, the alert cycle was changed; at the first site, the cycle was set to 1000‐1600‐2200‐0400, whereas the second site chose 0800‐1400‐2000‐0200.

One essential but problematic component of the clinical response is the issue of documentation. Inadequate documentation could lead to adverse outcomes, clinician malpractice exposure, and placing the entire hospital at risk for enterprise liability when clinical responses are not documented. This issue is complicated by the fact that overzealous efforts could lead to less or no documentation by making it too onerous for busy clinicians. We found that the ease with which data can populate progress notes in the EMR can lead to note bloat. Clearly, no documentation is not enough, and a complete history and physical is too much. Paradoxically, 1 of the issues underlying our problems with documentation was the proactive nature of the alerts themselves; because they are based on an outcome prediction in the next 12 hours, documenting the response to them may lack (perceived) urgency.

Shortly after the system went live, a patient who had been recently transferred out to the ward from the ICU triggered an alert. As a response was mounted, the team realized that existing ward protocols did not specify which physician service (intensivist or hospitalist) was responsible for patients who were transitioning from 1 unit to another. We also had to perform multiple revisions of the protocols specifying how alerts were handled when they occurred at times of change of shift. Eventually, we settled on having the combination of a hospitalist and an RRT nurse as the cornerstone of the response, with the hospitalist service as the primary owner of the entire process, but this arrangement might need to be varied in different settings. As a result of the experience with the pilot, the business case for deployment in the remaining 19 hospitals includes a formal budget request so that all have properly staffed RRTs, although the issue of primary ownership of the alert process for different patient types (eg, surgical patients) will be decided on a hospital‐by‐hospital basis. These experiences raise the intriguing possibility that implementation of alert systems can lead to the identification of systemic gaps in existing protocols. These gaps can include specific components of the hospital service agreements between multiple departments (emergency, hospital medicine, ICU, palliative care, surgery) as well as problems with existing workflows.

In addition to ongoing tweaking of care protocols, 3 issues remain unresolved. First is the issue of documentation. The current documentation notes are not completely satisfactory, and we are working with the KPNC EMR administrators to refine the tool. Desirable refinements include (1) having the system scores populate in more accessible sectors of the EMR where their retrieval will facilitate increased automation of the note writing process, (2) changing the note type to a note that will facilitate process audits, and (3) linking the note to other EMR tools so that the response arm can be tracked more formally. The second issue is the need to develop strategies to address staff turnover; for example, newer staff may not have received the same degree of exposure to the system as those who were there when it was started. Finally, due to limited resources, we have done very limited work on more mechanistic analyses of the clinical response itself. For example, it would be desirable to perform a formal quantitative, risk‐adjusted process‐outcome analysis of why some patients' outcomes are better than others following an alert.

Finally, it is also the case that we have had some unexpected occurrences that hint at new uses and benefits of alert systems. One of these is the phenomenon of chasing the alert. Some clinicians, on their own, have taken a more proactive stance in the care of patients in whom the AAM score is rising or near the alert threshold. This has 2 potential consequences. Some patients are stabilized and thus do not reach threshold instability levels. In other cases, patients reach threshold but the response team is informed that things are already under control. A second unexpected result is increased requests for COPS2 scores by clinicians who have heard about the system, particularly surgeons who would like to use the comorbidity scores as a screening tool in the outpatient setting. Because KPNC is an integrated system, it is not likely that such alternatives will be implemented immediately without considerable analysis, but it is clear that the system's deployment has captured the clinicians' imagination.

CONCLUSIONS AND FUTURE DIRECTIONS

Our preparatory efforts have been successful. We have found that embedding an EWS in a commercially available EMR is acceptable to hospital physicians and nurses. We have developed a coordinated workflow for mitigation and escalation that is tightly linked to the availability of probabilistic alerts in real time. Although resource limitations have precluded us from conducting formal clinician surveys, the EWS has been discussed at multiple hospital‐wide as well as department‐specific meetings. Although there have been requests for clarification, refinements, and modifications in workflows, no one has suggested that the system be discontinued. Further, many of the other KPNC hospitals have requested that the EWS be deployed at their site. We have examined KPNC databases that track patient complaints and have not found any complaints that could be linked to the EWS. Most importantly, the existence of the workflows we have developed has played a major role in KPNC's decision to deploy the system in its remaining hospitals.

Although alert fatigue is the number 1 reason that clinicians do not utilize embedded clinical decision support,[26] simply calibrating statistical models is insufficient. Careful consideration of clinicians' needs and responsibilities, particularly around ownership of patients and documentation, is essential. Such consideration needs to include planning time and socializing the system (providing multiple venues for clinicians to learn about the system as well as participate in the process for using it).

We anticipate that, as the system leaves the pilot stage and becomes a routine component of hospital care, additional enhancements (eg, sending notifications to smart phones, providing an alert response tracking system) will be added. Our organization is also implementing real‐time concurrent review of inpatient EMRs (eg, for proactive detection of an expanded range of potential process failures), and work is underway on how to link the workflows we describe here with this effort. As has been the case with other systems,[27] it is likely that we will eventually move to continuous scanning of patient data rather than only every 6 hours. Given that the basic workflow is quite robust and amenable to local modifications, we are confident that our clinicians and hospitals will adapt to future system enhancements.

Lastly, we intend to conduct additional research on the clinical response itself. In particular, we consider it extremely important to conduct formal quantitative analyses on why some patients' outcomes are better than others following an alert. A key component of this effort will be to develop tools that can permit an automatedor nearly automatedassessment of the clinical response. For example, we are considering automated approaches that would scan the EMR for the presence of specific orders, notes, vital signs patterns, and laboratory tests following an alert. Whereas it may not be possible to dispense with manual chart review, even partial automation of a feedback process could lead to significant enhancement of our quality improvement efforts.

Sepsis, the systemic inflammatory response to infection, is a major public health concern.[1] Worldwide, sepsis affects millions of hospitalized patients each year.[2] In the United States, it is the single most expensive cause of hospitalization.[3, 4, 5, 6] Multiple studies suggest that sepsis hospitalizations are also increasing in frequency.[3, 6, 7, 8, 9, 10]

Improved sepsis care has dramatically reduced in‐hospital mortality.[11, 12, 13] However, the result is a growing number of sepsis survivors discharged with new disability.[1, 9, 14, 15, 16] Despite being a common cause of hospitalization, little is known about how to improve postsepsis care.[15, 17, 18, 19] This contrasts with other, often less common, hospital conditions for which many studies evaluating readmission and postdischarge care are available.[20, 21, 22, 23] Identifying the factors contributing to high utilization could lend critical insight to designing interventions that improve long‐term sepsis outcomes.[24]

We conducted a retrospective study of sepsis patients discharged in 2010 at Kaiser Permanente Northern California (KPNC) to describe their posthospital trajectories. In this diverse community‐hospitalbased population, we sought to identify the patient‐level factors that impact the posthospital healthcare utilization of sepsis survivors.

METHODS

This study was approved by the KPNC institutional review board.

RESULTS

One‐Year Survival

A total of 5479 (86.4%) patients survived their index sepsis hospitalization. Overall survival after living discharge was 90.5% (range, 89.6%91.2%) at 30 days and 71.3% (range, 70.1%72.5%) at 1 year. However, postsepsis survival was strongly modified by age (Figure 1). For example, 1‐year survival was 94.1% (range, 91.2%96.0%) for <45 year olds and 54.4% (range, 51.5%57.2%) for 85 year olds (P<0.01). Survival was also modified by predicted mortality, however, not by ICU admission during index hospitalization (P=0.18) (see Supporting Appendix, Figure 1, in the online version of this article).

Figure 1

Healthcare Utilization

The unadjusted difference between pre‐ and postsepsis healthcare utilization among survivors was statistically significant for most categories but of modest clinical significance (see Supporting Appendix, Table 4, in the online version of this article). For example, the mean number of presepsis hospitalizations was 0.9 (1.4) compared to 1.0 (1.5) postsepsis (P<0.01). After adjusting for postsepsis living days, the difference in utilization was more pronounced (Figure 2). Overall, there was roughly a 3‐fold increase in the mean percentage of living days spent in facility‐based care between patients' pre‐ and postsepsis phases (5.3% vs 15.0%, P<0.01). Again, the difference was strongly modified by age. For patients aged <45 years, the difference was not statistically significant (2.4% vs 2.9%, P=0.32), whereas for those aged 65 years, it was highly significant (6.2% vs 18.5%, P<0.01).

Figure 2

Factors associated with early readmission included severity of illness, comorbid disease burden, index hospital length of stay, and intensive care (Table 3). However, the dominant factor explaining variation in the risk of early readmission was patients' prior comorbid disease burden (73.9%), followed by acute severity of illness (12.4%), total hospital length of stay (6.6%), and the need for intensive care (5.2%). Severity of illness and age were also significantly associated with higher odds of high postsepsis utilization; however, the dominant factor contributing to this risk was a history of high presepsis utilization (64.2%).

DISCUSSION

In this population‐based study in a community healthcare system, the impact of sepsis extended well beyond the initial hospitalization. One in 6 sepsis survivors was readmitted within 30 days, and roughly half were readmitted within 1 year. Fewer than half of rehospitalizations were for sepsis. Patients had a 3‐fold increase in the percentage of living days spent in hospitals or care facilities after sepsis hospitalization. Although age and acute severity of illness strongly modified healthcare utilization and mortality after sepsis, the dominant factors contributing to early readmission and high utilization ratescomorbid disease burden and presepsis healthcare utilizationwere present prior to hospitalization.

Sepsis is the single most expensive cause of US hospitalizations.[3, 4, 5] Despite its prevalence, there are little contemporary data identifying factors that impact healthcare utilization among sepsis survivors.[9, 16, 17, 19, 24, 36, 37] Recently, Prescott and others found that in Medicare beneficiaries, following severe sepsis, healthcare utilization was markedly increased.[17] More than one‐quarter of survivors were readmitted within 30 days, and 63.8% were readmitted within a year. Severe sepsis survivors also spent an average of 26% of their living days in a healthcare facility, a nearly 4‐fold increase compared to their presepsis phase. The current study included a population with a broader age and severity range; however, in a similar subgroup of patients, for those aged 65 years within the highest predicted mortality quartile, the frequency of readmission was similar. These findings are concordant with those from prior studies.[17, 19, 36, 37]

Among sepsis survivors, most readmissions were not for sepsis or infectious diagnoses, which is a novel finding with implications for designing approaches to reduce rehospitalization. The pattern in sepsis is similar to that seen in other common and costly hospital conditions.[17, 20, 23, 38, 39, 40] For example, between 18% and 25% of Medicare beneficiaries hospitalized for heart failure, acute myocardial infarction, or pneumonia were readmitted within 30 days; fewer than one‐third had the same diagnosis.[20] The timing of readmission in our sepsis cohort was also similar to that seen in other conditions.[20] For example, the median time of early readmission in this study was 11 days; it was between 10 and 12 days for patients with heart failure, pneumonia, and myocardial infarction.[20]

Krumholz and others suggest that the pattern of early rehospitalization after common acute conditions reflects a posthospital syndromean acquired, transient period of vulnerabilitythat could be the byproduct of common hospital factors.[20, 41] Such universal impairments might result from new physical and neurocognitive disability, nutritional deficiency, and sleep deprivation or delirium, among others.[41] If this construct were also true in sepsis, it could have important implications on the design of postsepsis care. However, prior studies suggest that sepsis patients may be particularly vulnerable to the sequelae of hospitalization.[2, 42, 43, 44, 45]

Among Medicare beneficiaries, Iwashyna and others reported that hospitalizations for severe sepsis resulted in significant increases in physical limitations and moderate to severe cognitive impairment.[1, 14, 46] Encephalopathy, sleep deprivation, and delirium are also frequently seen in sepsis patients.[47, 48] Furthermore, sepsis patients frequently need intensive care, which is also associated with increased patient disability and injury.[16, 46, 49, 50] We found that severity of illness and the need for intensive care were both predictive of the need for early readmission following sepsis. We also confirmed the results of prior studies suggesting that sepsis outcomes are strongly modified by age.[16, 19, 43, 51]

However, we found that the dominant factors contributing to patients' health trajectories were conditions present prior to admission. This finding is in accord with prior suggestions that acute severity of illness only partially predicts patients facing adverse posthospital sequelae.[23, 41, 52] Among sepsis patients, prior work demonstrates that inadequate consideration for presepsis level of function and utilization can result in an overestimation of the impact of sepsis on postdischarge health.[52, 53] Further, we found that the need for intensive care was not independently associated with an increased risk of high postsepsis utilization after adjusting for illness severity, a finding also seen in prior studies.[17, 23, 38, 51]

Taken together, our findings might suggest that an optimal approach to posthospital care in sepsis should focus on treatment approaches that address disease‐specific problems within the much larger context of common hospital risks. However, further study is necessary to clearly define the mechanisms by which age, severity of illness, and intensive care affect subsequent healthcare utilization. Furthermore, sepsis patients are a heterogeneous population in terms of severity of illness, site and pathogen of infection, and underlying comorbidity whose posthospital course remains incompletely characterized, limiting our ability to draw strong inferences.

These results should be interpreted in light of the study's limitations. First, our cohort included patients with healthcare insurance within a community‐based healthcare system. Care within the KPNC system, which bears similarities with accountable care organizations, is enhanced through service integration and a comprehensive health information system. Although prior studies suggest that these characteristics result in improved population‐based care, it is unclear whether there is a similar impact in hospital‐based conditions such as sepsis.[54, 55] Furthermore, care within an integrated system may impact posthospital utilization patterns and could limit generalizability. However, prior studies demonstrate the similarity of KPNC members to other patients in the same region in terms of age, socioeconomics, overall health behaviors, and racial/ethnic diversity.[56] Second, our study did not characterize organ dysfunction based on diagnosis coding, a common feature of sepsis studies that lack detailed physiologic severity data.[4, 5, 6, 8, 26] Instead, we focused on using granular laboratory and vital signs data to ensure accurate risk adjustment using a validated system developed in >400,000 hospitalizations.[30] Although this method may hamper comparisons with existing studies, traditional methods of grading severity by diagnosis codes can be vulnerable to biases resulting in wide variability.[10, 23, 26, 57, 58] Nonetheless, it is likely that characterizing preexisting and acute organ dysfunction will improve risk stratification in the heterogeneous sepsis population. Third, this study did not include data regarding patients' functional status, which has been shown to strongly predict patient outcomes following hospitalization. Fourth, this study did not address the cost of care following sepsis hospitalizations.[19, 59] Finally, our study excluded patients with incomplete utilization records, a choice designed to avoid the spurious inferences that can result from such comparisons.[53]

In summary, we found that sepsis exacted a considerable toll on patients in the hospital and in the year following discharge. Sepsis patients were frequently rehospitalized within a month of discharge, and on average had a 3‐fold increase in their subsequent time spent in healthcare facilities. Although age, severity of illness, and the need for ICU care impacted postsepsis utilization, the dominant contributing factorscomorbid disease burden or presepsis utilizationwere present prior to sepsis hospitalization. Early readmission patterns in sepsis appeared similar to those seen in other important hospital conditions, suggesting a role for shared posthospital, rather than just postsepsis, care approaches.

Sepsis, the systemic inflammatory response to infection, is a major public health concern.[1] Worldwide, sepsis affects millions of hospitalized patients each year.[2] In the United States, it is the single most expensive cause of hospitalization.[3, 4, 5, 6] Multiple studies suggest that sepsis hospitalizations are also increasing in frequency.[3, 6, 7, 8, 9, 10]

Improved sepsis care has dramatically reduced in‐hospital mortality.[11, 12, 13] However, the result is a growing number of sepsis survivors discharged with new disability.[1, 9, 14, 15, 16] Despite being a common cause of hospitalization, little is known about how to improve postsepsis care.[15, 17, 18, 19] This contrasts with other, often less common, hospital conditions for which many studies evaluating readmission and postdischarge care are available.[20, 21, 22, 23] Identifying the factors contributing to high utilization could lend critical insight to designing interventions that improve long‐term sepsis outcomes.[24]

We conducted a retrospective study of sepsis patients discharged in 2010 at Kaiser Permanente Northern California (KPNC) to describe their posthospital trajectories. In this diverse community‐hospitalbased population, we sought to identify the patient‐level factors that impact the posthospital healthcare utilization of sepsis survivors.

METHODS

This study was approved by the KPNC institutional review board.

RESULTS

One‐Year Survival

A total of 5479 (86.4%) patients survived their index sepsis hospitalization. Overall survival after living discharge was 90.5% (range, 89.6%91.2%) at 30 days and 71.3% (range, 70.1%72.5%) at 1 year. However, postsepsis survival was strongly modified by age (Figure 1). For example, 1‐year survival was 94.1% (range, 91.2%96.0%) for <45 year olds and 54.4% (range, 51.5%57.2%) for 85 year olds (P<0.01). Survival was also modified by predicted mortality, however, not by ICU admission during index hospitalization (P=0.18) (see Supporting Appendix, Figure 1, in the online version of this article).

Figure 1

Healthcare Utilization

The unadjusted difference between pre‐ and postsepsis healthcare utilization among survivors was statistically significant for most categories but of modest clinical significance (see Supporting Appendix, Table 4, in the online version of this article). For example, the mean number of presepsis hospitalizations was 0.9 (1.4) compared to 1.0 (1.5) postsepsis (P<0.01). After adjusting for postsepsis living days, the difference in utilization was more pronounced (Figure 2). Overall, there was roughly a 3‐fold increase in the mean percentage of living days spent in facility‐based care between patients' pre‐ and postsepsis phases (5.3% vs 15.0%, P<0.01). Again, the difference was strongly modified by age. For patients aged <45 years, the difference was not statistically significant (2.4% vs 2.9%, P=0.32), whereas for those aged 65 years, it was highly significant (6.2% vs 18.5%, P<0.01).

Figure 2

Factors associated with early readmission included severity of illness, comorbid disease burden, index hospital length of stay, and intensive care (Table 3). However, the dominant factor explaining variation in the risk of early readmission was patients' prior comorbid disease burden (73.9%), followed by acute severity of illness (12.4%), total hospital length of stay (6.6%), and the need for intensive care (5.2%). Severity of illness and age were also significantly associated with higher odds of high postsepsis utilization; however, the dominant factor contributing to this risk was a history of high presepsis utilization (64.2%).

DISCUSSION

In this population‐based study in a community healthcare system, the impact of sepsis extended well beyond the initial hospitalization. One in 6 sepsis survivors was readmitted within 30 days, and roughly half were readmitted within 1 year. Fewer than half of rehospitalizations were for sepsis. Patients had a 3‐fold increase in the percentage of living days spent in hospitals or care facilities after sepsis hospitalization. Although age and acute severity of illness strongly modified healthcare utilization and mortality after sepsis, the dominant factors contributing to early readmission and high utilization ratescomorbid disease burden and presepsis healthcare utilizationwere present prior to hospitalization.

Sepsis is the single most expensive cause of US hospitalizations.[3, 4, 5] Despite its prevalence, there are little contemporary data identifying factors that impact healthcare utilization among sepsis survivors.[9, 16, 17, 19, 24, 36, 37] Recently, Prescott and others found that in Medicare beneficiaries, following severe sepsis, healthcare utilization was markedly increased.[17] More than one‐quarter of survivors were readmitted within 30 days, and 63.8% were readmitted within a year. Severe sepsis survivors also spent an average of 26% of their living days in a healthcare facility, a nearly 4‐fold increase compared to their presepsis phase. The current study included a population with a broader age and severity range; however, in a similar subgroup of patients, for those aged 65 years within the highest predicted mortality quartile, the frequency of readmission was similar. These findings are concordant with those from prior studies.[17, 19, 36, 37]

Among sepsis survivors, most readmissions were not for sepsis or infectious diagnoses, which is a novel finding with implications for designing approaches to reduce rehospitalization. The pattern in sepsis is similar to that seen in other common and costly hospital conditions.[17, 20, 23, 38, 39, 40] For example, between 18% and 25% of Medicare beneficiaries hospitalized for heart failure, acute myocardial infarction, or pneumonia were readmitted within 30 days; fewer than one‐third had the same diagnosis.[20] The timing of readmission in our sepsis cohort was also similar to that seen in other conditions.[20] For example, the median time of early readmission in this study was 11 days; it was between 10 and 12 days for patients with heart failure, pneumonia, and myocardial infarction.[20]

Krumholz and others suggest that the pattern of early rehospitalization after common acute conditions reflects a posthospital syndromean acquired, transient period of vulnerabilitythat could be the byproduct of common hospital factors.[20, 41] Such universal impairments might result from new physical and neurocognitive disability, nutritional deficiency, and sleep deprivation or delirium, among others.[41] If this construct were also true in sepsis, it could have important implications on the design of postsepsis care. However, prior studies suggest that sepsis patients may be particularly vulnerable to the sequelae of hospitalization.[2, 42, 43, 44, 45]

Among Medicare beneficiaries, Iwashyna and others reported that hospitalizations for severe sepsis resulted in significant increases in physical limitations and moderate to severe cognitive impairment.[1, 14, 46] Encephalopathy, sleep deprivation, and delirium are also frequently seen in sepsis patients.[47, 48] Furthermore, sepsis patients frequently need intensive care, which is also associated with increased patient disability and injury.[16, 46, 49, 50] We found that severity of illness and the need for intensive care were both predictive of the need for early readmission following sepsis. We also confirmed the results of prior studies suggesting that sepsis outcomes are strongly modified by age.[16, 19, 43, 51]

However, we found that the dominant factors contributing to patients' health trajectories were conditions present prior to admission. This finding is in accord with prior suggestions that acute severity of illness only partially predicts patients facing adverse posthospital sequelae.[23, 41, 52] Among sepsis patients, prior work demonstrates that inadequate consideration for presepsis level of function and utilization can result in an overestimation of the impact of sepsis on postdischarge health.[52, 53] Further, we found that the need for intensive care was not independently associated with an increased risk of high postsepsis utilization after adjusting for illness severity, a finding also seen in prior studies.[17, 23, 38, 51]

Taken together, our findings might suggest that an optimal approach to posthospital care in sepsis should focus on treatment approaches that address disease‐specific problems within the much larger context of common hospital risks. However, further study is necessary to clearly define the mechanisms by which age, severity of illness, and intensive care affect subsequent healthcare utilization. Furthermore, sepsis patients are a heterogeneous population in terms of severity of illness, site and pathogen of infection, and underlying comorbidity whose posthospital course remains incompletely characterized, limiting our ability to draw strong inferences.

These results should be interpreted in light of the study's limitations. First, our cohort included patients with healthcare insurance within a community‐based healthcare system. Care within the KPNC system, which bears similarities with accountable care organizations, is enhanced through service integration and a comprehensive health information system. Although prior studies suggest that these characteristics result in improved population‐based care, it is unclear whether there is a similar impact in hospital‐based conditions such as sepsis.[54, 55] Furthermore, care within an integrated system may impact posthospital utilization patterns and could limit generalizability. However, prior studies demonstrate the similarity of KPNC members to other patients in the same region in terms of age, socioeconomics, overall health behaviors, and racial/ethnic diversity.[56] Second, our study did not characterize organ dysfunction based on diagnosis coding, a common feature of sepsis studies that lack detailed physiologic severity data.[4, 5, 6, 8, 26] Instead, we focused on using granular laboratory and vital signs data to ensure accurate risk adjustment using a validated system developed in >400,000 hospitalizations.[30] Although this method may hamper comparisons with existing studies, traditional methods of grading severity by diagnosis codes can be vulnerable to biases resulting in wide variability.[10, 23, 26, 57, 58] Nonetheless, it is likely that characterizing preexisting and acute organ dysfunction will improve risk stratification in the heterogeneous sepsis population. Third, this study did not include data regarding patients' functional status, which has been shown to strongly predict patient outcomes following hospitalization. Fourth, this study did not address the cost of care following sepsis hospitalizations.[19, 59] Finally, our study excluded patients with incomplete utilization records, a choice designed to avoid the spurious inferences that can result from such comparisons.[53]