Yixin Chen, PhD

Rapid response systems (RRSs) have been developed to identify and treat deteriorating patients on general hospital units.[1] The most commonly proposed approach to the problem of identifying and stabilizing deteriorating hospitalized patients includes some combination of an early warning system to detect the deterioration and an RRS to deal with it. We previously demonstrated that a relatively simple hospital‐specific prediction model employing routine laboratory values and vital sign data is capable of predicting clinical deterioration, the need for intensive care unit (ICU) transfer, and hospital mortality in patients admitted to general medicine units.[2, 3, 4, 5, 6]

Hospital readmissions within 30 days of hospital discharge occur often and are difficult to predict. Starting in 2013, readmission penalties have been applied to specific conditions in the United States (acute myocardial infarction, heart failure, and pneumonia), with the expectation that additional conditions will be added to this group in years to come.[7, 8] Unfortunately, interventions developed to date have not been universally successful in preventing hospital readmissions for various medical conditions and patient types.[9] One potential explanation for this is the inability to reliably predict which patients are at risk for readmission to better target preventative interventions. Predictors of hospital readmission can be disease specific, such as the presence of multivessel disease in patients hospitalized with myocardial infarction,[10] or more general, such as lack of available medical follow‐up postdischarge.[11] Therefore, we performed a study to determine whether the occurrence of automated clinical deterioration alerts (CDAs) predicted 30‐day hospital readmission.

METHODS

RESULTS

The final cohort had 3015 patients with a mean age of 57.5 17.5 years and 47.8% males. The most common reasons for hospital admission were infection or sepsis syndrome including pneumonia and urinary tract infections (23.6%), congestive heart failure or other cardiac conditions (18.4%), respiratory distress including chronic obstructive pulmonary disease (16.2%), acute or chronic renal failure (9.7%), gastrointestinal disorders (8.4%), and diabetes mellitus management (7.4%). Overall, there were 567 (18.8%) patients who were readmitted within 30 days of their hospital discharge date.

Table 2 shows the characteristics of patients readmitted within 30 days and of patients not requiring hospital readmission within 30 days. Patients requiring hospital readmission within 30 days were younger and had significantly more comorbidities as manifested by significantly greater Charlson scores and individual comorbidities including coronary artery disease, congestive heart disease, peripheral vascular disease, connective tissue disease, cirrhosis, diabetes mellitus with end‐organ complications, renal failure, and metastatic cancer. Patients with a 30‐day readmission had significantly longer duration of hospitalization, more emergency department visits in the 6 months prior to the index hospitalization, lower minimum hemoglobin measurements, higher minimum serum creatinine values, and were more likely to have Medicare or Medicaid insurance compared to patients without a 30‐day readmission.

There were 1141 (34.4%) patients that triggered a CDA. Patients triggering a CDA were significantly more likely to have a 30‐day readmission compared to those who did not trigger a CDA (23.6% vs 15.9%; P < 0.001). Patients triggering a CDA were also significantly more likely to be readmitted within 60 days (31.7% vs 22.1%; P < 0.001) and 90 days (35.8% vs 26.2%; P < 0.001) compared to patients who did not trigger a CDA. Multiple logistic regression identified the triggering of a CDA to be independently associated with 30‐day readmission (OR: 1.40; 95% CI: 1.26‐1.55; P = 0.001) (Table 3). Other independent predictors of 30‐day readmission were: an emergency department visit in the previous 6 months, increasing age in 1‐year increments, presence of connective tissue disease, diabetes mellitus with end‐organ complications, chronic renal disease, cirrhosis, and metastatic cancer (Hosmer‐Lemeshow goodness of fit test, 0.363). Figure 1 reveals the ROC curves for the logistic regression model (Table 3) with and without the CDA variable. As the ROC curves document, the 2 models had similar sensitivity for the entire range of specificities. Reflecting this, the area under the ROC curve for the model inclusive of the CDA variable equaled 0.675 (95% CI: 0.649‐0.700), whereas the area under the ROC curve for the model excluding the CDA variable equaled 0.658 (95% CI: 0.632‐0.684).

Figure 1

DISCUSSION

We demonstrated that the occurrence of an automated CDA is associated with increased risk for 30‐day hospital readmission. However, the addition of the CDA variable to the other variables identified to be independently associated with 30‐day readmission (Table 3) did not significantly add to the overall predictive accuracy of the derived logistic regression model. Other investigators have previously attempted to develop automated predictors of hospital readmission. Amarasingham et al. developed a real‐time electronic predictive model that identifies hospitalized heart failure patients at high risk for readmission or death from clinical and nonclinical risk factors present on admission.[13] Their electronic model demonstrated good discrimination for 30‐day mortality and readmission and performed as well, or better than, models developed by the Center for Medicaid and Medicare Services and the Acute Decompensated Heart Failure Registry. Similarly, Baillie et al. developed an automated prediction model that was effectively integrated into an existing EMR and identified patients on admission who were at risk for readmission within 30 days of discharge.[14] Our automated CDA differs from these previous risk predictors by surveying patients throughout their hospital stay as opposed to identifying risk for readmission at a single time point.

Several limitations of our study should be recognized. First, this was a noninterventional study aimed at examining the ability of CDAs to predict hospital readmission. Future studies are needed to assess whether the use of enhanced readmission prediction algorithms can be utilized to avert hospital readmissions. Second, the data derive from a single center, and this necessarily limits the generalizability of our findings. As such, our results may not reflect what one might see at other institutions. For example, Barnes‐Jewish Hospital has a regional referral pattern that includes community hospitals, regional long‐term acute care hospitals, nursing homes, and chronic wound, dialysis, and infusion clinics. This may explain, in part, the relatively high rate of hospital readmission observed in our cohort. Third, there is the possibility that CDAs were associated with readmission by chance given the number of potential predictor variables examined. The importance of CDAs as a determinant of rehospitalization requires confirmation in other independent populations. Fourth, it is likely that we did not capture all hospital readmissions, primarily those occurring outside of our hospital system. Therefore, we may have underestimated the actual rates of readmission for this cohort. Finally, we cannot be certain that all important predictors of hospital readmission were captured in this study.

The development of an accurate real‐time early warning system has the potential to identify patients at risk for various adverse outcomes including clinical deterioration, hospital death, and postdischarge readmission. By identifying patients at greatest risk for readmission, valuable healthcare resources can be better targeted to such populations. Our findings suggest that existing readmission predictors may suboptimally risk‐stratify patients, and it may be important to include additional clinical variables if pay for performance and other across‐institution comparisons are to be fair to institutions that care for more seriously ill patients. The variables identified as predictors of 30‐day hospital readmission in our study, with the exception of a CDA, are all readily identifiable clinical characteristics. The modest incremental value of a CDA to these clinical characteristics suggests that they would suffice for the identification of patients at high risk for hospital readmission. This is especially important for safety‐net institutions not routinely employing automated CDAs. These safety‐net hospitals provide a disproportionate level of care for patients who otherwise would have difficulty obtaining inpatient medical care and disproportionately carry the greatest burden of hospital readmissions.[15]

Rapid response systems (RRSs) have been developed to identify and treat deteriorating patients on general hospital units.[1] The most commonly proposed approach to the problem of identifying and stabilizing deteriorating hospitalized patients includes some combination of an early warning system to detect the deterioration and an RRS to deal with it. We previously demonstrated that a relatively simple hospital‐specific prediction model employing routine laboratory values and vital sign data is capable of predicting clinical deterioration, the need for intensive care unit (ICU) transfer, and hospital mortality in patients admitted to general medicine units.[2, 3, 4, 5, 6]

Hospital readmissions within 30 days of hospital discharge occur often and are difficult to predict. Starting in 2013, readmission penalties have been applied to specific conditions in the United States (acute myocardial infarction, heart failure, and pneumonia), with the expectation that additional conditions will be added to this group in years to come.[7, 8] Unfortunately, interventions developed to date have not been universally successful in preventing hospital readmissions for various medical conditions and patient types.[9] One potential explanation for this is the inability to reliably predict which patients are at risk for readmission to better target preventative interventions. Predictors of hospital readmission can be disease specific, such as the presence of multivessel disease in patients hospitalized with myocardial infarction,[10] or more general, such as lack of available medical follow‐up postdischarge.[11] Therefore, we performed a study to determine whether the occurrence of automated clinical deterioration alerts (CDAs) predicted 30‐day hospital readmission.

METHODS

RESULTS

The final cohort had 3015 patients with a mean age of 57.5 17.5 years and 47.8% males. The most common reasons for hospital admission were infection or sepsis syndrome including pneumonia and urinary tract infections (23.6%), congestive heart failure or other cardiac conditions (18.4%), respiratory distress including chronic obstructive pulmonary disease (16.2%), acute or chronic renal failure (9.7%), gastrointestinal disorders (8.4%), and diabetes mellitus management (7.4%). Overall, there were 567 (18.8%) patients who were readmitted within 30 days of their hospital discharge date.

Table 2 shows the characteristics of patients readmitted within 30 days and of patients not requiring hospital readmission within 30 days. Patients requiring hospital readmission within 30 days were younger and had significantly more comorbidities as manifested by significantly greater Charlson scores and individual comorbidities including coronary artery disease, congestive heart disease, peripheral vascular disease, connective tissue disease, cirrhosis, diabetes mellitus with end‐organ complications, renal failure, and metastatic cancer. Patients with a 30‐day readmission had significantly longer duration of hospitalization, more emergency department visits in the 6 months prior to the index hospitalization, lower minimum hemoglobin measurements, higher minimum serum creatinine values, and were more likely to have Medicare or Medicaid insurance compared to patients without a 30‐day readmission.

There were 1141 (34.4%) patients that triggered a CDA. Patients triggering a CDA were significantly more likely to have a 30‐day readmission compared to those who did not trigger a CDA (23.6% vs 15.9%; P < 0.001). Patients triggering a CDA were also significantly more likely to be readmitted within 60 days (31.7% vs 22.1%; P < 0.001) and 90 days (35.8% vs 26.2%; P < 0.001) compared to patients who did not trigger a CDA. Multiple logistic regression identified the triggering of a CDA to be independently associated with 30‐day readmission (OR: 1.40; 95% CI: 1.26‐1.55; P = 0.001) (Table 3). Other independent predictors of 30‐day readmission were: an emergency department visit in the previous 6 months, increasing age in 1‐year increments, presence of connective tissue disease, diabetes mellitus with end‐organ complications, chronic renal disease, cirrhosis, and metastatic cancer (Hosmer‐Lemeshow goodness of fit test, 0.363). Figure 1 reveals the ROC curves for the logistic regression model (Table 3) with and without the CDA variable. As the ROC curves document, the 2 models had similar sensitivity for the entire range of specificities. Reflecting this, the area under the ROC curve for the model inclusive of the CDA variable equaled 0.675 (95% CI: 0.649‐0.700), whereas the area under the ROC curve for the model excluding the CDA variable equaled 0.658 (95% CI: 0.632‐0.684).

Figure 1

DISCUSSION

We demonstrated that the occurrence of an automated CDA is associated with increased risk for 30‐day hospital readmission. However, the addition of the CDA variable to the other variables identified to be independently associated with 30‐day readmission (Table 3) did not significantly add to the overall predictive accuracy of the derived logistic regression model. Other investigators have previously attempted to develop automated predictors of hospital readmission. Amarasingham et al. developed a real‐time electronic predictive model that identifies hospitalized heart failure patients at high risk for readmission or death from clinical and nonclinical risk factors present on admission.[13] Their electronic model demonstrated good discrimination for 30‐day mortality and readmission and performed as well, or better than, models developed by the Center for Medicaid and Medicare Services and the Acute Decompensated Heart Failure Registry. Similarly, Baillie et al. developed an automated prediction model that was effectively integrated into an existing EMR and identified patients on admission who were at risk for readmission within 30 days of discharge.[14] Our automated CDA differs from these previous risk predictors by surveying patients throughout their hospital stay as opposed to identifying risk for readmission at a single time point.

Several limitations of our study should be recognized. First, this was a noninterventional study aimed at examining the ability of CDAs to predict hospital readmission. Future studies are needed to assess whether the use of enhanced readmission prediction algorithms can be utilized to avert hospital readmissions. Second, the data derive from a single center, and this necessarily limits the generalizability of our findings. As such, our results may not reflect what one might see at other institutions. For example, Barnes‐Jewish Hospital has a regional referral pattern that includes community hospitals, regional long‐term acute care hospitals, nursing homes, and chronic wound, dialysis, and infusion clinics. This may explain, in part, the relatively high rate of hospital readmission observed in our cohort. Third, there is the possibility that CDAs were associated with readmission by chance given the number of potential predictor variables examined. The importance of CDAs as a determinant of rehospitalization requires confirmation in other independent populations. Fourth, it is likely that we did not capture all hospital readmissions, primarily those occurring outside of our hospital system. Therefore, we may have underestimated the actual rates of readmission for this cohort. Finally, we cannot be certain that all important predictors of hospital readmission were captured in this study.

The development of an accurate real‐time early warning system has the potential to identify patients at risk for various adverse outcomes including clinical deterioration, hospital death, and postdischarge readmission. By identifying patients at greatest risk for readmission, valuable healthcare resources can be better targeted to such populations. Our findings suggest that existing readmission predictors may suboptimally risk‐stratify patients, and it may be important to include additional clinical variables if pay for performance and other across‐institution comparisons are to be fair to institutions that care for more seriously ill patients. The variables identified as predictors of 30‐day hospital readmission in our study, with the exception of a CDA, are all readily identifiable clinical characteristics. The modest incremental value of a CDA to these clinical characteristics suggests that they would suffice for the identification of patients at high risk for hospital readmission. This is especially important for safety‐net institutions not routinely employing automated CDAs. These safety‐net hospitals provide a disproportionate level of care for patients who otherwise would have difficulty obtaining inpatient medical care and disproportionately carry the greatest burden of hospital readmissions.[15]

Rapid response systems (RRSs) have been developed to identify and treat deteriorating patients on general hospital units.[1] The most commonly proposed approach to the problem of identifying and stabilizing deteriorating hospitalized patients includes some combination of an early warning system to detect the deterioration and an RRS to deal with it. We previously demonstrated that a relatively simple hospital‐specific prediction model employing routine laboratory values and vital sign data is capable of predicting clinical deterioration, the need for intensive care unit (ICU) transfer, and hospital mortality in patients admitted to general medicine units.[2, 3, 4, 5, 6]

Hospital readmissions within 30 days of hospital discharge occur often and are difficult to predict. Starting in 2013, readmission penalties have been applied to specific conditions in the United States (acute myocardial infarction, heart failure, and pneumonia), with the expectation that additional conditions will be added to this group in years to come.[7, 8] Unfortunately, interventions developed to date have not been universally successful in preventing hospital readmissions for various medical conditions and patient types.[9] One potential explanation for this is the inability to reliably predict which patients are at risk for readmission to better target preventative interventions. Predictors of hospital readmission can be disease specific, such as the presence of multivessel disease in patients hospitalized with myocardial infarction,[10] or more general, such as lack of available medical follow‐up postdischarge.[11] Therefore, we performed a study to determine whether the occurrence of automated clinical deterioration alerts (CDAs) predicted 30‐day hospital readmission.

METHODS

RESULTS

The final cohort had 3015 patients with a mean age of 57.5 17.5 years and 47.8% males. The most common reasons for hospital admission were infection or sepsis syndrome including pneumonia and urinary tract infections (23.6%), congestive heart failure or other cardiac conditions (18.4%), respiratory distress including chronic obstructive pulmonary disease (16.2%), acute or chronic renal failure (9.7%), gastrointestinal disorders (8.4%), and diabetes mellitus management (7.4%). Overall, there were 567 (18.8%) patients who were readmitted within 30 days of their hospital discharge date.

Table 2 shows the characteristics of patients readmitted within 30 days and of patients not requiring hospital readmission within 30 days. Patients requiring hospital readmission within 30 days were younger and had significantly more comorbidities as manifested by significantly greater Charlson scores and individual comorbidities including coronary artery disease, congestive heart disease, peripheral vascular disease, connective tissue disease, cirrhosis, diabetes mellitus with end‐organ complications, renal failure, and metastatic cancer. Patients with a 30‐day readmission had significantly longer duration of hospitalization, more emergency department visits in the 6 months prior to the index hospitalization, lower minimum hemoglobin measurements, higher minimum serum creatinine values, and were more likely to have Medicare or Medicaid insurance compared to patients without a 30‐day readmission.

There were 1141 (34.4%) patients that triggered a CDA. Patients triggering a CDA were significantly more likely to have a 30‐day readmission compared to those who did not trigger a CDA (23.6% vs 15.9%; P < 0.001). Patients triggering a CDA were also significantly more likely to be readmitted within 60 days (31.7% vs 22.1%; P < 0.001) and 90 days (35.8% vs 26.2%; P < 0.001) compared to patients who did not trigger a CDA. Multiple logistic regression identified the triggering of a CDA to be independently associated with 30‐day readmission (OR: 1.40; 95% CI: 1.26‐1.55; P = 0.001) (Table 3). Other independent predictors of 30‐day readmission were: an emergency department visit in the previous 6 months, increasing age in 1‐year increments, presence of connective tissue disease, diabetes mellitus with end‐organ complications, chronic renal disease, cirrhosis, and metastatic cancer (Hosmer‐Lemeshow goodness of fit test, 0.363). Figure 1 reveals the ROC curves for the logistic regression model (Table 3) with and without the CDA variable. As the ROC curves document, the 2 models had similar sensitivity for the entire range of specificities. Reflecting this, the area under the ROC curve for the model inclusive of the CDA variable equaled 0.675 (95% CI: 0.649‐0.700), whereas the area under the ROC curve for the model excluding the CDA variable equaled 0.658 (95% CI: 0.632‐0.684).

Figure 1

DISCUSSION

We demonstrated that the occurrence of an automated CDA is associated with increased risk for 30‐day hospital readmission. However, the addition of the CDA variable to the other variables identified to be independently associated with 30‐day readmission (Table 3) did not significantly add to the overall predictive accuracy of the derived logistic regression model. Other investigators have previously attempted to develop automated predictors of hospital readmission. Amarasingham et al. developed a real‐time electronic predictive model that identifies hospitalized heart failure patients at high risk for readmission or death from clinical and nonclinical risk factors present on admission.[13] Their electronic model demonstrated good discrimination for 30‐day mortality and readmission and performed as well, or better than, models developed by the Center for Medicaid and Medicare Services and the Acute Decompensated Heart Failure Registry. Similarly, Baillie et al. developed an automated prediction model that was effectively integrated into an existing EMR and identified patients on admission who were at risk for readmission within 30 days of discharge.[14] Our automated CDA differs from these previous risk predictors by surveying patients throughout their hospital stay as opposed to identifying risk for readmission at a single time point.

Several limitations of our study should be recognized. First, this was a noninterventional study aimed at examining the ability of CDAs to predict hospital readmission. Future studies are needed to assess whether the use of enhanced readmission prediction algorithms can be utilized to avert hospital readmissions. Second, the data derive from a single center, and this necessarily limits the generalizability of our findings. As such, our results may not reflect what one might see at other institutions. For example, Barnes‐Jewish Hospital has a regional referral pattern that includes community hospitals, regional long‐term acute care hospitals, nursing homes, and chronic wound, dialysis, and infusion clinics. This may explain, in part, the relatively high rate of hospital readmission observed in our cohort. Third, there is the possibility that CDAs were associated with readmission by chance given the number of potential predictor variables examined. The importance of CDAs as a determinant of rehospitalization requires confirmation in other independent populations. Fourth, it is likely that we did not capture all hospital readmissions, primarily those occurring outside of our hospital system. Therefore, we may have underestimated the actual rates of readmission for this cohort. Finally, we cannot be certain that all important predictors of hospital readmission were captured in this study.

The development of an accurate real‐time early warning system has the potential to identify patients at risk for various adverse outcomes including clinical deterioration, hospital death, and postdischarge readmission. By identifying patients at greatest risk for readmission, valuable healthcare resources can be better targeted to such populations. Our findings suggest that existing readmission predictors may suboptimally risk‐stratify patients, and it may be important to include additional clinical variables if pay for performance and other across‐institution comparisons are to be fair to institutions that care for more seriously ill patients. The variables identified as predictors of 30‐day hospital readmission in our study, with the exception of a CDA, are all readily identifiable clinical characteristics. The modest incremental value of a CDA to these clinical characteristics suggests that they would suffice for the identification of patients at high risk for hospital readmission. This is especially important for safety‐net institutions not routinely employing automated CDAs. These safety‐net hospitals provide a disproportionate level of care for patients who otherwise would have difficulty obtaining inpatient medical care and disproportionately carry the greatest burden of hospital readmissions.[15]

Patients deemed suitable for care on a general hospital unit are not expected to deteriorate; however, triage systems are not perfect, and some patients on general nursing units do develop critical illness during their hospitalization. Fortunately, there is mounting evidence that deteriorating patients exhibit measurable pathologic changes that could possibly be used to identify them prior to significant adverse outcomes, such as cardiac arrest.[1, 2, 3] Given the evidence that unplanned intensive care unit (ICU) transfers of patients on general units result in worse outcomes than more controlled ICU admissions,[1, 4, 5, 6] it is logical to assume that earlier identification of a deteriorating patient could provide a window of opportunity to prevent adverse outcomes.

The most commonly proposed systematic solution to the problem of identifying and stabilizing deteriorating patients on general hospital units includes some combination of an early warning system (EWS) to detect the deterioration and a rapid response team (RRT) to deal with it.[7, 8, 9, 10] We previously demonstrated that a relatively simple hospital‐specific method for generating EWS alerts derived from the electronic medical record (EMR) database is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general inpatient medicine units.[11, 12, 13, 14] However, our data also showed that simply providing the EWS alerts to these nursing units did not result in any demonstrable improvement in patient outcomes.[14] Therefore, we set out to determine whether linking real‐time EWS alerts to an intervention and notification of the RRT for patient evaluation could improve the outcomes of patients cared for on general inpatient units.

METHODS

RESULTS

Between January 15, 2013 and May 9, 2013, there were 4731 consecutive patients admitted to the 8 inpatient units and electronically screened as the base population for this investigation. Five hundred seventy‐one (12.1%) patients triggered an alert and were enrolled into the study (Figure 1). There were 286 patients assigned to the intervention group and 285 assigned to the control group. No patients were lost to follow‐up. Demographics, reason for hospital admission, and comorbidities of the 2 groups were similar (Table 1). The number of patients having a separate RRT call by the primary nursing team on the hospital units within 24 hours of generating an alert was greater for the intervention group but did not reach statistical significance (19.9% vs 16.5%; odds ratio: 1.260; 95% confidence interval [CI]: 0.8231.931). Table 2 provides the new diagnostic and therapeutic interventions initiated within 24 hours after a EWS alert was generated. Patients in the intervention group were significantly more likely to have their primary care team physician notified by an RRT nurse regarding medical condition issues and to have oximetry and telemetry started, whereas control patients were significantly more likely to have new antibiotic orders written within 24 hours of generating an alert.

Figure 1

Fifty‐one patients (17.8%) randomly assigned to the intervention group required ICU transfer compared with 52 of 285 patients (18.2%) in the control group (odds ratio: 0.972; 95% CI: 0.6351.490; P=0.898) (Table 3). Twenty‐one patients (7.3%) randomly assigned to the intervention group expired during their hospitalization compared with 22 of 285 patients (7.7%) in the control group (odds ratio: 0.947; 95%CI: 0.5091.764; P=0.865). Hospital length of stay was 8.49.5 days (median, 4.5 days; interquartile range, 2.311.4 days) for patients randomized to the intervention group and 9.411.1 days (median, 5.3 days; interquartile range, 3.211.2 days) for patients randomized to the control group (P=0.038). The ICU length of stay was 4.86.6 days (median, 2.9 days; interquartile range, 1.76.5 days) for patients randomized to the intervention group and 5.86.4 days (median, 2.9 days; interquartile range, 1.57.4) for patients randomized to the control group (P=0.812).The number of patients requiring transfer to a nursing home or long‐term acute care hospital was similar for patients in the intervention and control groups (26.9% vs 26.3%; odds ratio: 1.032; 95% CI: 0.7121.495; P=0.870). Similarly, the number of patients requiring hospital readmission before 30 days and 180 days, respectively, was similar for the 2 treatment groups (Table 3). For the combined study population, the EWS alerts were triggered 94138 hours (median, 27 hours; interquartile range, 7132 hours) prior to ICU transfer and 250204 hours (median200 hours; interquartile range, 54347 hours) prior to hospital mortality. The number of RRT calls for the 8 medicine units studied progressively increased from the start of the RRT program in 2009 through 2013 (121 in 2009, 194 in 2010, 298 in 2011, 415 in 2012, 415 in 2013; P<0.001 for the trend).

DISCUSSION

We demonstrated that a real‐time EWS alert sent to a RRT nurse was associated with a modest reduction in hospital length of stay, but similar rates of hospital mortality, ICU transfer, and subsequent need for placement in a long‐term care setting compared with usual care. We also found the number of RRT calls to have increased progressively from 2009 to the present on the study units examined.

Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[6] Bapoje et al. evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[19] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in an EWS.[20, 21] Organizations like the Institute for Healthcare Improvement have called for the development and routine implementation of EWSs to direct the activities of RRTs and improve outcomes.[22] However, a recent systematic review found that much of the evidence in support of EWSs and emergency response teams is of poor quality and lacking prospective randomized trials.[23]

Our earlier experience demonstrated that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our EWS.[14] Previous investigations have also had difficulty in demonstrating consistent outcome improvements with the use of EWSs and RRTs.[24, 25, 26, 27, 28, 29, 30, 31, 32] As a result of mandates from quality improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[33, 34] Linking RRT actions with a validated real‐time alert may represent a way of improving the overall effectiveness of such teams for monitoring general hospital units, short of having all hospitalized patients in units staffed and monitored to provide higher levels of supervision (eg, ICUs, step‐down units).[9, 35]

An alternative approach to preventing patient deterioration is to provide closer overall monitoring. This has been accomplished by employing nursing personnel to increase monitoring, or with the use of automated monitoring equipment. Bellomo et al. showed that the deployment of electronic automated vital sign monitors on general hospital units was associated with improved utilization of RRTs, increased patient survival, and decreased time for vital sign measurement and recording.[36] Laurens and Dwyer found that implementation of medical emergency teams (METs) to respond to predefined MET activation criteria as observed by hospital staff resulted in reduced hospital mortality and reduced need for ICU transfer.[37] However, other investigators have observed that imperfect implementation of nursing‐performed observational monitoring resulted in no demonstrable benefit, illustrating the limitations of this approach.[38] Our findings suggest that nursing care of patients on general hospital units may be enhanced with the use of an EWS alert sent to the RRT. This is supported by the observation that communications between the RRT and the primary care teams was greater as was the use of telemetry and oximetry in the intervention arm. Moreover, there appears to have been a learning effect for the nursing staff that occurred on our study units, as evidenced by the increased number of RRT calls that occurred between 2009 and 2013. This change in nursing practices on these units certainly made it more difficult for us to observe outcome differences in our current study with the prescribed intervention, reinforcing the notion that evaluating an already established practice is a difficult proposition.[39]

Our study has several important limitations. First, the EWS alert was developed and validated at Barnes‐Jewish Hospital.[11, 12, 13, 14] We cannot say whether this alert will perform similarly in another hospital. Second, the EWS alert only contains data from medical patients. Development and validation of EWS alerts for other hospitalized populations, including surgical and pediatric patients, are needed to make such systems more generalizable. Third, the primary clinical outcome employed for this trial was problematic. Transfer to an ICU may not be an optimal outcome variable, as it may be desirable to transfer alerted patients to an ICU, which can be perceived to represent a soft landing for such patients once an alert has been generated. A better measure could be 30‐day all‐cause mortality, which would not be subject to clinician biases. Finally, we could not specifically identify explanations for the greater use of antibiotics in the control group despite similar rates of infection for both study arms. Future studies should closely evaluate the ability of EWS alerts to alter specific therapies (eg, reduce antibiotic utilization).

In summary, we have demonstrated that an EWS alert linked to a RRT likely contributed to a modest reduction in hospital length of stay, but no reductions in hospital mortality and ICU transfer. These findings suggest that inpatient deterioration on general hospital units can be identified and linked to a specific intervention. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general hospital units but also intervene to improve their outcomes. We are moving forward with the development of a 2‐tiered EWS utilizing both EMR data and real‐time streamed vital sign data, to determine if we can further improve the prediction of clinical deterioration and potentially intervene in a more clinically meaningful manner.

Patients deemed suitable for care on a general hospital unit are not expected to deteriorate; however, triage systems are not perfect, and some patients on general nursing units do develop critical illness during their hospitalization. Fortunately, there is mounting evidence that deteriorating patients exhibit measurable pathologic changes that could possibly be used to identify them prior to significant adverse outcomes, such as cardiac arrest.[1, 2, 3] Given the evidence that unplanned intensive care unit (ICU) transfers of patients on general units result in worse outcomes than more controlled ICU admissions,[1, 4, 5, 6] it is logical to assume that earlier identification of a deteriorating patient could provide a window of opportunity to prevent adverse outcomes.

The most commonly proposed systematic solution to the problem of identifying and stabilizing deteriorating patients on general hospital units includes some combination of an early warning system (EWS) to detect the deterioration and a rapid response team (RRT) to deal with it.[7, 8, 9, 10] We previously demonstrated that a relatively simple hospital‐specific method for generating EWS alerts derived from the electronic medical record (EMR) database is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general inpatient medicine units.[11, 12, 13, 14] However, our data also showed that simply providing the EWS alerts to these nursing units did not result in any demonstrable improvement in patient outcomes.[14] Therefore, we set out to determine whether linking real‐time EWS alerts to an intervention and notification of the RRT for patient evaluation could improve the outcomes of patients cared for on general inpatient units.

METHODS

RESULTS

Between January 15, 2013 and May 9, 2013, there were 4731 consecutive patients admitted to the 8 inpatient units and electronically screened as the base population for this investigation. Five hundred seventy‐one (12.1%) patients triggered an alert and were enrolled into the study (Figure 1). There were 286 patients assigned to the intervention group and 285 assigned to the control group. No patients were lost to follow‐up. Demographics, reason for hospital admission, and comorbidities of the 2 groups were similar (Table 1). The number of patients having a separate RRT call by the primary nursing team on the hospital units within 24 hours of generating an alert was greater for the intervention group but did not reach statistical significance (19.9% vs 16.5%; odds ratio: 1.260; 95% confidence interval [CI]: 0.8231.931). Table 2 provides the new diagnostic and therapeutic interventions initiated within 24 hours after a EWS alert was generated. Patients in the intervention group were significantly more likely to have their primary care team physician notified by an RRT nurse regarding medical condition issues and to have oximetry and telemetry started, whereas control patients were significantly more likely to have new antibiotic orders written within 24 hours of generating an alert.

Figure 1

Fifty‐one patients (17.8%) randomly assigned to the intervention group required ICU transfer compared with 52 of 285 patients (18.2%) in the control group (odds ratio: 0.972; 95% CI: 0.6351.490; P=0.898) (Table 3). Twenty‐one patients (7.3%) randomly assigned to the intervention group expired during their hospitalization compared with 22 of 285 patients (7.7%) in the control group (odds ratio: 0.947; 95%CI: 0.5091.764; P=0.865). Hospital length of stay was 8.49.5 days (median, 4.5 days; interquartile range, 2.311.4 days) for patients randomized to the intervention group and 9.411.1 days (median, 5.3 days; interquartile range, 3.211.2 days) for patients randomized to the control group (P=0.038). The ICU length of stay was 4.86.6 days (median, 2.9 days; interquartile range, 1.76.5 days) for patients randomized to the intervention group and 5.86.4 days (median, 2.9 days; interquartile range, 1.57.4) for patients randomized to the control group (P=0.812).The number of patients requiring transfer to a nursing home or long‐term acute care hospital was similar for patients in the intervention and control groups (26.9% vs 26.3%; odds ratio: 1.032; 95% CI: 0.7121.495; P=0.870). Similarly, the number of patients requiring hospital readmission before 30 days and 180 days, respectively, was similar for the 2 treatment groups (Table 3). For the combined study population, the EWS alerts were triggered 94138 hours (median, 27 hours; interquartile range, 7132 hours) prior to ICU transfer and 250204 hours (median200 hours; interquartile range, 54347 hours) prior to hospital mortality. The number of RRT calls for the 8 medicine units studied progressively increased from the start of the RRT program in 2009 through 2013 (121 in 2009, 194 in 2010, 298 in 2011, 415 in 2012, 415 in 2013; P<0.001 for the trend).

DISCUSSION

We demonstrated that a real‐time EWS alert sent to a RRT nurse was associated with a modest reduction in hospital length of stay, but similar rates of hospital mortality, ICU transfer, and subsequent need for placement in a long‐term care setting compared with usual care. We also found the number of RRT calls to have increased progressively from 2009 to the present on the study units examined.

Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[6] Bapoje et al. evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[19] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in an EWS.[20, 21] Organizations like the Institute for Healthcare Improvement have called for the development and routine implementation of EWSs to direct the activities of RRTs and improve outcomes.[22] However, a recent systematic review found that much of the evidence in support of EWSs and emergency response teams is of poor quality and lacking prospective randomized trials.[23]

Our earlier experience demonstrated that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our EWS.[14] Previous investigations have also had difficulty in demonstrating consistent outcome improvements with the use of EWSs and RRTs.[24, 25, 26, 27, 28, 29, 30, 31, 32] As a result of mandates from quality improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[33, 34] Linking RRT actions with a validated real‐time alert may represent a way of improving the overall effectiveness of such teams for monitoring general hospital units, short of having all hospitalized patients in units staffed and monitored to provide higher levels of supervision (eg, ICUs, step‐down units).[9, 35]

An alternative approach to preventing patient deterioration is to provide closer overall monitoring. This has been accomplished by employing nursing personnel to increase monitoring, or with the use of automated monitoring equipment. Bellomo et al. showed that the deployment of electronic automated vital sign monitors on general hospital units was associated with improved utilization of RRTs, increased patient survival, and decreased time for vital sign measurement and recording.[36] Laurens and Dwyer found that implementation of medical emergency teams (METs) to respond to predefined MET activation criteria as observed by hospital staff resulted in reduced hospital mortality and reduced need for ICU transfer.[37] However, other investigators have observed that imperfect implementation of nursing‐performed observational monitoring resulted in no demonstrable benefit, illustrating the limitations of this approach.[38] Our findings suggest that nursing care of patients on general hospital units may be enhanced with the use of an EWS alert sent to the RRT. This is supported by the observation that communications between the RRT and the primary care teams was greater as was the use of telemetry and oximetry in the intervention arm. Moreover, there appears to have been a learning effect for the nursing staff that occurred on our study units, as evidenced by the increased number of RRT calls that occurred between 2009 and 2013. This change in nursing practices on these units certainly made it more difficult for us to observe outcome differences in our current study with the prescribed intervention, reinforcing the notion that evaluating an already established practice is a difficult proposition.[39]

Our study has several important limitations. First, the EWS alert was developed and validated at Barnes‐Jewish Hospital.[11, 12, 13, 14] We cannot say whether this alert will perform similarly in another hospital. Second, the EWS alert only contains data from medical patients. Development and validation of EWS alerts for other hospitalized populations, including surgical and pediatric patients, are needed to make such systems more generalizable. Third, the primary clinical outcome employed for this trial was problematic. Transfer to an ICU may not be an optimal outcome variable, as it may be desirable to transfer alerted patients to an ICU, which can be perceived to represent a soft landing for such patients once an alert has been generated. A better measure could be 30‐day all‐cause mortality, which would not be subject to clinician biases. Finally, we could not specifically identify explanations for the greater use of antibiotics in the control group despite similar rates of infection for both study arms. Future studies should closely evaluate the ability of EWS alerts to alter specific therapies (eg, reduce antibiotic utilization).

In summary, we have demonstrated that an EWS alert linked to a RRT likely contributed to a modest reduction in hospital length of stay, but no reductions in hospital mortality and ICU transfer. These findings suggest that inpatient deterioration on general hospital units can be identified and linked to a specific intervention. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general hospital units but also intervene to improve their outcomes. We are moving forward with the development of a 2‐tiered EWS utilizing both EMR data and real‐time streamed vital sign data, to determine if we can further improve the prediction of clinical deterioration and potentially intervene in a more clinically meaningful manner.

Patients deemed suitable for care on a general hospital unit are not expected to deteriorate; however, triage systems are not perfect, and some patients on general nursing units do develop critical illness during their hospitalization. Fortunately, there is mounting evidence that deteriorating patients exhibit measurable pathologic changes that could possibly be used to identify them prior to significant adverse outcomes, such as cardiac arrest.[1, 2, 3] Given the evidence that unplanned intensive care unit (ICU) transfers of patients on general units result in worse outcomes than more controlled ICU admissions,[1, 4, 5, 6] it is logical to assume that earlier identification of a deteriorating patient could provide a window of opportunity to prevent adverse outcomes.

The most commonly proposed systematic solution to the problem of identifying and stabilizing deteriorating patients on general hospital units includes some combination of an early warning system (EWS) to detect the deterioration and a rapid response team (RRT) to deal with it.[7, 8, 9, 10] We previously demonstrated that a relatively simple hospital‐specific method for generating EWS alerts derived from the electronic medical record (EMR) database is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general inpatient medicine units.[11, 12, 13, 14] However, our data also showed that simply providing the EWS alerts to these nursing units did not result in any demonstrable improvement in patient outcomes.[14] Therefore, we set out to determine whether linking real‐time EWS alerts to an intervention and notification of the RRT for patient evaluation could improve the outcomes of patients cared for on general inpatient units.

METHODS

RESULTS

Between January 15, 2013 and May 9, 2013, there were 4731 consecutive patients admitted to the 8 inpatient units and electronically screened as the base population for this investigation. Five hundred seventy‐one (12.1%) patients triggered an alert and were enrolled into the study (Figure 1). There were 286 patients assigned to the intervention group and 285 assigned to the control group. No patients were lost to follow‐up. Demographics, reason for hospital admission, and comorbidities of the 2 groups were similar (Table 1). The number of patients having a separate RRT call by the primary nursing team on the hospital units within 24 hours of generating an alert was greater for the intervention group but did not reach statistical significance (19.9% vs 16.5%; odds ratio: 1.260; 95% confidence interval [CI]: 0.8231.931). Table 2 provides the new diagnostic and therapeutic interventions initiated within 24 hours after a EWS alert was generated. Patients in the intervention group were significantly more likely to have their primary care team physician notified by an RRT nurse regarding medical condition issues and to have oximetry and telemetry started, whereas control patients were significantly more likely to have new antibiotic orders written within 24 hours of generating an alert.

Figure 1

Fifty‐one patients (17.8%) randomly assigned to the intervention group required ICU transfer compared with 52 of 285 patients (18.2%) in the control group (odds ratio: 0.972; 95% CI: 0.6351.490; P=0.898) (Table 3). Twenty‐one patients (7.3%) randomly assigned to the intervention group expired during their hospitalization compared with 22 of 285 patients (7.7%) in the control group (odds ratio: 0.947; 95%CI: 0.5091.764; P=0.865). Hospital length of stay was 8.49.5 days (median, 4.5 days; interquartile range, 2.311.4 days) for patients randomized to the intervention group and 9.411.1 days (median, 5.3 days; interquartile range, 3.211.2 days) for patients randomized to the control group (P=0.038). The ICU length of stay was 4.86.6 days (median, 2.9 days; interquartile range, 1.76.5 days) for patients randomized to the intervention group and 5.86.4 days (median, 2.9 days; interquartile range, 1.57.4) for patients randomized to the control group (P=0.812).The number of patients requiring transfer to a nursing home or long‐term acute care hospital was similar for patients in the intervention and control groups (26.9% vs 26.3%; odds ratio: 1.032; 95% CI: 0.7121.495; P=0.870). Similarly, the number of patients requiring hospital readmission before 30 days and 180 days, respectively, was similar for the 2 treatment groups (Table 3). For the combined study population, the EWS alerts were triggered 94138 hours (median, 27 hours; interquartile range, 7132 hours) prior to ICU transfer and 250204 hours (median200 hours; interquartile range, 54347 hours) prior to hospital mortality. The number of RRT calls for the 8 medicine units studied progressively increased from the start of the RRT program in 2009 through 2013 (121 in 2009, 194 in 2010, 298 in 2011, 415 in 2012, 415 in 2013; P<0.001 for the trend).

DISCUSSION

We demonstrated that a real‐time EWS alert sent to a RRT nurse was associated with a modest reduction in hospital length of stay, but similar rates of hospital mortality, ICU transfer, and subsequent need for placement in a long‐term care setting compared with usual care. We also found the number of RRT calls to have increased progressively from 2009 to the present on the study units examined.

Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[6] Bapoje et al. evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[19] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in an EWS.[20, 21] Organizations like the Institute for Healthcare Improvement have called for the development and routine implementation of EWSs to direct the activities of RRTs and improve outcomes.[22] However, a recent systematic review found that much of the evidence in support of EWSs and emergency response teams is of poor quality and lacking prospective randomized trials.[23]

Our earlier experience demonstrated that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our EWS.[14] Previous investigations have also had difficulty in demonstrating consistent outcome improvements with the use of EWSs and RRTs.[24, 25, 26, 27, 28, 29, 30, 31, 32] As a result of mandates from quality improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[33, 34] Linking RRT actions with a validated real‐time alert may represent a way of improving the overall effectiveness of such teams for monitoring general hospital units, short of having all hospitalized patients in units staffed and monitored to provide higher levels of supervision (eg, ICUs, step‐down units).[9, 35]

An alternative approach to preventing patient deterioration is to provide closer overall monitoring. This has been accomplished by employing nursing personnel to increase monitoring, or with the use of automated monitoring equipment. Bellomo et al. showed that the deployment of electronic automated vital sign monitors on general hospital units was associated with improved utilization of RRTs, increased patient survival, and decreased time for vital sign measurement and recording.[36] Laurens and Dwyer found that implementation of medical emergency teams (METs) to respond to predefined MET activation criteria as observed by hospital staff resulted in reduced hospital mortality and reduced need for ICU transfer.[37] However, other investigators have observed that imperfect implementation of nursing‐performed observational monitoring resulted in no demonstrable benefit, illustrating the limitations of this approach.[38] Our findings suggest that nursing care of patients on general hospital units may be enhanced with the use of an EWS alert sent to the RRT. This is supported by the observation that communications between the RRT and the primary care teams was greater as was the use of telemetry and oximetry in the intervention arm. Moreover, there appears to have been a learning effect for the nursing staff that occurred on our study units, as evidenced by the increased number of RRT calls that occurred between 2009 and 2013. This change in nursing practices on these units certainly made it more difficult for us to observe outcome differences in our current study with the prescribed intervention, reinforcing the notion that evaluating an already established practice is a difficult proposition.[39]

Our study has several important limitations. First, the EWS alert was developed and validated at Barnes‐Jewish Hospital.[11, 12, 13, 14] We cannot say whether this alert will perform similarly in another hospital. Second, the EWS alert only contains data from medical patients. Development and validation of EWS alerts for other hospitalized populations, including surgical and pediatric patients, are needed to make such systems more generalizable. Third, the primary clinical outcome employed for this trial was problematic. Transfer to an ICU may not be an optimal outcome variable, as it may be desirable to transfer alerted patients to an ICU, which can be perceived to represent a soft landing for such patients once an alert has been generated. A better measure could be 30‐day all‐cause mortality, which would not be subject to clinician biases. Finally, we could not specifically identify explanations for the greater use of antibiotics in the control group despite similar rates of infection for both study arms. Future studies should closely evaluate the ability of EWS alerts to alter specific therapies (eg, reduce antibiotic utilization).

In summary, we have demonstrated that an EWS alert linked to a RRT likely contributed to a modest reduction in hospital length of stay, but no reductions in hospital mortality and ICU transfer. These findings suggest that inpatient deterioration on general hospital units can be identified and linked to a specific intervention. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general hospital units but also intervene to improve their outcomes. We are moving forward with the development of a 2‐tiered EWS utilizing both EMR data and real‐time streamed vital sign data, to determine if we can further improve the prediction of clinical deterioration and potentially intervene in a more clinically meaningful manner.

Timely interventions are essential in the management of complex medical conditions such as new‐onset sepsis in order to prevent rapid progression to severe sepsis and septic shock.[1, 2, 3, 4, 5] Similarly, rapid identification and appropriate treatment of other medical and surgical conditions have been associated with improved outcomes.[6, 7, 8] We previously developed a real‐time, computerized prediction tool (PT) using recursive partitioning regression tree analysis for the identification of impending sepsis for use on general hospital wards.[9] We also showed that implementation of a real‐time computerized sepsis alert on hospital wards based on the PT resulted in increased use of early interventions, including antibiotic escalation, intravenous fluids, oxygen therapy, and diagnostics in patients identified as at risk.[10]

The first goal of this study was to develop an updated PT for use on hospital wards that could be used to predict subsequent global clinical deterioration and the need for a higher level of care. The second goal was to determine whether simply providing a real‐time alert to nursing staff based on the updated PT resulted in any demonstrable changes in patient outcomes.

METHODS

RESULTS

DISCUSSION

We have demonstrated that a relatively simple hospital‐specific method for generating a PT derived from routine laboratory and hemodynamic values is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general hospital wards. We also found that the PT identified a sicker patient population as manifest by longer hospital LOS. The methods used in generating this real‐time PT are relatively simple and easily executed with the use of an electronic medical record (EMR) system. However, our data also showed that simply providing an alert to nursing units based on the PT did not result in any demonstrable improvement in patient outcomes. Moreover, our PT and intervention in their current form have substantial limitations, including low sensitivity and positive predictive value, high possibility of alert fatigue, and no clear clinical impact. These limitations suggest that this approach has limited applicability in its current form.

Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[13] Bapoje et al evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[14] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in a PT or early warning system (EWS). Keller et al evaluated 50 consecutive general medical patients with unplanned ICU transfers between 2003 and 2004.[15] Using a case‐control methodology, these investigators found shock index values>0.85 to be the best predictor for subsequent unplanned ICU transfer (P<0.02; odds ratio: 3.0).

Organizations such as the Institute for Healthcare Improvement have called for the development and implementation of EWSs in order to direct the activities of RRTs and improve outcomes.[16] Escobar et al carried out a retrospective case‐control study using as the unit of analysis 12‐hour patient shifts on hospital wards.[17] Using logistic regression and split validation, they developed a PT for ICU transfer from clinical variables available in their EMR. The EMR derived PT had a C‐statistic of 0.845 in the derivation dataset and 0.775 in the validation dataset, concluding that EMR‐based detection of impending deterioration outside the ICU is feasible in integrated healthcare delivery systems.

We found that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our PT. This may have been due to simply relying on the alerted nursing staff to make phone calls to physicians and not linking a specific and effective patient‐directed intervention to the PT. Other investigators have similarly observed that the use of an EWS or PT may not result in outcome improvements.[18] Gao et al performed an analysis of 31 studies describing hospital track and trigger EWSs.[19] They found little evidence of reliability, validity, and utility of these systems. Peebles et al showed that even when high‐risk non‐ICU patients are identified, delays in providing appropriate therapies occur, which may explain the lack of efficacy of EWSs and RRTs.[20] These observations suggest that there is currently a paucity of validated interventions available to improve outcome in deteriorating patients, despite our ability to identify patients who are at risk for such deterioration.

As a result of mandates from quality‐improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[21] However, as noted above, there is limited evidence to suggest that RRTs contribute to improved patient outcomes.[22, 23, 24, 25, 26, 27] The potential importance of this is reflected in a recent report suggesting that 2900 US hospitals now have rapid‐response systems in place without clear demonstration of their overall efficacy.[28] Linking rapid‐response interventions with a validated real‐time alert may represent a way of improving the effectiveness of such interventions.[29, 30, 31, 32, 33, 34] Our data showed that hospital LOS was statistically longer among alerted patients compared with nonalerted patients. This supports the conclusion that the alerts helped identify a sicker group of patients, but the nursing alerts did not appear to change outcomes. This finding also seems to refute the hypothesis that simply linking an intervention to a PT will improve outcomes, albeit the intervention we employed may not have been robust enough to influence patient outcomes.

The development of accurate real‐time EWSs holds the potential to identify patients at risk for clinical deterioration at an earlier point in time when rescue interventions can be implemented in a potentially more effective manner in both adults and children.[35] Unfortunately, the ideal intervention to be applied in this situation is unknown. Our experience suggests that successful interventions will require a more integrated approach than simply providing an alert with general management principles. As a result of our experience, we are undertaking a randomized clinical trial in 2013 to determine whether linking a patient‐specific intervention to a PT will result in improved outcomes. The intervention we will be testing is to have the RRT immediately notified about alerted patients so as to formally evaluate them and to determine the need for therapeutic interventions, and to administer such interventions as needed and/or transfer the alerted patients to a higher level of care as deemed necessary. Additionally, we are updating our PT with more temporal data to determine if this will improve its accuracy. One of these updates will include linking the PT to wirelessly obtained continuous oximetry and heart‐rate data, using minimally intrusive sensors, to establish a 2‐tiered EWS.[11]

Our study has several important limitations. First, the PT was developed using local data, and thus the results may not be applicable to other settings. However, our model shares many of the characteristics identified in other clinical‐deterioration PTs.[15, 17] Second, the positive prediction value of 15.2% for ICU transfer may not be clinically useful due to the large number of false‐positive results. Moreover, the large number of false positives could result in alert fatigue, causing alerts to be ignored. Third, although the charge nurses were supposed to call the responsible physicians for the alerted patients, we did not determine whether all these calls occurred or whether they resulted in any meaningful changes in monitoring or patient treatment. This is important because lack of an effective intervention or treatment would make the intervention group much more like our control group. Future studies are needed to assess the impact of an integrated intervention (eg, notification of experienced RRT members with adequate resource access) to determine if patient outcomes can be impacted by the use of an EWS. Finally, we did not compare the performance of our PT to other models such as the modified early warning score (MEWS).

An additional limitation to consider is that our PT offered no new information to the nurse manager, or the PT did not change the opinions of the charge nurses. This is supported by a recent study of 63 serious adverse outcomes in a Belgian teaching hospital where death was the final outcome.[36] Survey results revealed that nurses were often unaware that their patients were deteriorating before the crisis. Nurses also reported threshold levels for concern for abnormal vital signs that suggested they would call for assistance relatively late in clinical crises. The limited ability of nursing staff to identify deteriorating patients is also supported by a recent simulation study demonstrating that nurses did identify that patients were deteriorating, but as each patient deteriorated staff performance declined, with a reduction in all observational records and actions.[37]

In summary, we have demonstrated that a relatively simple hospital‐specific PT could accurately identify patients on general medicine wards who subsequently developed clinical deterioration and the need for ICU transfer, as well as hospital mortality. However, no improvements in patient outcomes were found from reporting this information to nursing wards on a real‐time basis. The low positive predictive value of the alerts, local development of the EWS, and absence of improved outcomes substantially limits the broader application of this system in its current form. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general wards but also intervene to improve their outcomes.

Timely interventions are essential in the management of complex medical conditions such as new‐onset sepsis in order to prevent rapid progression to severe sepsis and septic shock.[1, 2, 3, 4, 5] Similarly, rapid identification and appropriate treatment of other medical and surgical conditions have been associated with improved outcomes.[6, 7, 8] We previously developed a real‐time, computerized prediction tool (PT) using recursive partitioning regression tree analysis for the identification of impending sepsis for use on general hospital wards.[9] We also showed that implementation of a real‐time computerized sepsis alert on hospital wards based on the PT resulted in increased use of early interventions, including antibiotic escalation, intravenous fluids, oxygen therapy, and diagnostics in patients identified as at risk.[10]

The first goal of this study was to develop an updated PT for use on hospital wards that could be used to predict subsequent global clinical deterioration and the need for a higher level of care. The second goal was to determine whether simply providing a real‐time alert to nursing staff based on the updated PT resulted in any demonstrable changes in patient outcomes.

METHODS

RESULTS

DISCUSSION

We have demonstrated that a relatively simple hospital‐specific method for generating a PT derived from routine laboratory and hemodynamic values is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general hospital wards. We also found that the PT identified a sicker patient population as manifest by longer hospital LOS. The methods used in generating this real‐time PT are relatively simple and easily executed with the use of an electronic medical record (EMR) system. However, our data also showed that simply providing an alert to nursing units based on the PT did not result in any demonstrable improvement in patient outcomes. Moreover, our PT and intervention in their current form have substantial limitations, including low sensitivity and positive predictive value, high possibility of alert fatigue, and no clear clinical impact. These limitations suggest that this approach has limited applicability in its current form.

Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[13] Bapoje et al evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[14] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in a PT or early warning system (EWS). Keller et al evaluated 50 consecutive general medical patients with unplanned ICU transfers between 2003 and 2004.[15] Using a case‐control methodology, these investigators found shock index values>0.85 to be the best predictor for subsequent unplanned ICU transfer (P<0.02; odds ratio: 3.0).

Organizations such as the Institute for Healthcare Improvement have called for the development and implementation of EWSs in order to direct the activities of RRTs and improve outcomes.[16] Escobar et al carried out a retrospective case‐control study using as the unit of analysis 12‐hour patient shifts on hospital wards.[17] Using logistic regression and split validation, they developed a PT for ICU transfer from clinical variables available in their EMR. The EMR derived PT had a C‐statistic of 0.845 in the derivation dataset and 0.775 in the validation dataset, concluding that EMR‐based detection of impending deterioration outside the ICU is feasible in integrated healthcare delivery systems.

We found that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our PT. This may have been due to simply relying on the alerted nursing staff to make phone calls to physicians and not linking a specific and effective patient‐directed intervention to the PT. Other investigators have similarly observed that the use of an EWS or PT may not result in outcome improvements.[18] Gao et al performed an analysis of 31 studies describing hospital track and trigger EWSs.[19] They found little evidence of reliability, validity, and utility of these systems. Peebles et al showed that even when high‐risk non‐ICU patients are identified, delays in providing appropriate therapies occur, which may explain the lack of efficacy of EWSs and RRTs.[20] These observations suggest that there is currently a paucity of validated interventions available to improve outcome in deteriorating patients, despite our ability to identify patients who are at risk for such deterioration.

As a result of mandates from quality‐improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[21] However, as noted above, there is limited evidence to suggest that RRTs contribute to improved patient outcomes.[22, 23, 24, 25, 26, 27] The potential importance of this is reflected in a recent report suggesting that 2900 US hospitals now have rapid‐response systems in place without clear demonstration of their overall efficacy.[28] Linking rapid‐response interventions with a validated real‐time alert may represent a way of improving the effectiveness of such interventions.[29, 30, 31, 32, 33, 34] Our data showed that hospital LOS was statistically longer among alerted patients compared with nonalerted patients. This supports the conclusion that the alerts helped identify a sicker group of patients, but the nursing alerts did not appear to change outcomes. This finding also seems to refute the hypothesis that simply linking an intervention to a PT will improve outcomes, albeit the intervention we employed may not have been robust enough to influence patient outcomes.

The development of accurate real‐time EWSs holds the potential to identify patients at risk for clinical deterioration at an earlier point in time when rescue interventions can be implemented in a potentially more effective manner in both adults and children.[35] Unfortunately, the ideal intervention to be applied in this situation is unknown. Our experience suggests that successful interventions will require a more integrated approach than simply providing an alert with general management principles. As a result of our experience, we are undertaking a randomized clinical trial in 2013 to determine whether linking a patient‐specific intervention to a PT will result in improved outcomes. The intervention we will be testing is to have the RRT immediately notified about alerted patients so as to formally evaluate them and to determine the need for therapeutic interventions, and to administer such interventions as needed and/or transfer the alerted patients to a higher level of care as deemed necessary. Additionally, we are updating our PT with more temporal data to determine if this will improve its accuracy. One of these updates will include linking the PT to wirelessly obtained continuous oximetry and heart‐rate data, using minimally intrusive sensors, to establish a 2‐tiered EWS.[11]

Our study has several important limitations. First, the PT was developed using local data, and thus the results may not be applicable to other settings. However, our model shares many of the characteristics identified in other clinical‐deterioration PTs.[15, 17] Second, the positive prediction value of 15.2% for ICU transfer may not be clinically useful due to the large number of false‐positive results. Moreover, the large number of false positives could result in alert fatigue, causing alerts to be ignored. Third, although the charge nurses were supposed to call the responsible physicians for the alerted patients, we did not determine whether all these calls occurred or whether they resulted in any meaningful changes in monitoring or patient treatment. This is important because lack of an effective intervention or treatment would make the intervention group much more like our control group. Future studies are needed to assess the impact of an integrated intervention (eg, notification of experienced RRT members with adequate resource access) to determine if patient outcomes can be impacted by the use of an EWS. Finally, we did not compare the performance of our PT to other models such as the modified early warning score (MEWS).

An additional limitation to consider is that our PT offered no new information to the nurse manager, or the PT did not change the opinions of the charge nurses. This is supported by a recent study of 63 serious adverse outcomes in a Belgian teaching hospital where death was the final outcome.[36] Survey results revealed that nurses were often unaware that their patients were deteriorating before the crisis. Nurses also reported threshold levels for concern for abnormal vital signs that suggested they would call for assistance relatively late in clinical crises. The limited ability of nursing staff to identify deteriorating patients is also supported by a recent simulation study demonstrating that nurses did identify that patients were deteriorating, but as each patient deteriorated staff performance declined, with a reduction in all observational records and actions.[37]

In summary, we have demonstrated that a relatively simple hospital‐specific PT could accurately identify patients on general medicine wards who subsequently developed clinical deterioration and the need for ICU transfer, as well as hospital mortality. However, no improvements in patient outcomes were found from reporting this information to nursing wards on a real‐time basis. The low positive predictive value of the alerts, local development of the EWS, and absence of improved outcomes substantially limits the broader application of this system in its current form. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general wards but also intervene to improve their outcomes.

Timely interventions are essential in the management of complex medical conditions such as new‐onset sepsis in order to prevent rapid progression to severe sepsis and septic shock.[1, 2, 3, 4, 5] Similarly, rapid identification and appropriate treatment of other medical and surgical conditions have been associated with improved outcomes.[6, 7, 8] We previously developed a real‐time, computerized prediction tool (PT) using recursive partitioning regression tree analysis for the identification of impending sepsis for use on general hospital wards.[9] We also showed that implementation of a real‐time computerized sepsis alert on hospital wards based on the PT resulted in increased use of early interventions, including antibiotic escalation, intravenous fluids, oxygen therapy, and diagnostics in patients identified as at risk.[10]

The first goal of this study was to develop an updated PT for use on hospital wards that could be used to predict subsequent global clinical deterioration and the need for a higher level of care. The second goal was to determine whether simply providing a real‐time alert to nursing staff based on the updated PT resulted in any demonstrable changes in patient outcomes.

METHODS

RESULTS

DISCUSSION

We have demonstrated that a relatively simple hospital‐specific method for generating a PT derived from routine laboratory and hemodynamic values is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general hospital wards. We also found that the PT identified a sicker patient population as manifest by longer hospital LOS. The methods used in generating this real‐time PT are relatively simple and easily executed with the use of an electronic medical record (EMR) system. However, our data also showed that simply providing an alert to nursing units based on the PT did not result in any demonstrable improvement in patient outcomes. Moreover, our PT and intervention in their current form have substantial limitations, including low sensitivity and positive predictive value, high possibility of alert fatigue, and no clear clinical impact. These limitations suggest that this approach has limited applicability in its current form.

Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[13] Bapoje et al evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[14] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in a PT or early warning system (EWS). Keller et al evaluated 50 consecutive general medical patients with unplanned ICU transfers between 2003 and 2004.[15] Using a case‐control methodology, these investigators found shock index values>0.85 to be the best predictor for subsequent unplanned ICU transfer (P<0.02; odds ratio: 3.0).

Organizations such as the Institute for Healthcare Improvement have called for the development and implementation of EWSs in order to direct the activities of RRTs and improve outcomes.[16] Escobar et al carried out a retrospective case‐control study using as the unit of analysis 12‐hour patient shifts on hospital wards.[17] Using logistic regression and split validation, they developed a PT for ICU transfer from clinical variables available in their EMR. The EMR derived PT had a C‐statistic of 0.845 in the derivation dataset and 0.775 in the validation dataset, concluding that EMR‐based detection of impending deterioration outside the ICU is feasible in integrated healthcare delivery systems.

We found that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our PT. This may have been due to simply relying on the alerted nursing staff to make phone calls to physicians and not linking a specific and effective patient‐directed intervention to the PT. Other investigators have similarly observed that the use of an EWS or PT may not result in outcome improvements.[18] Gao et al performed an analysis of 31 studies describing hospital track and trigger EWSs.[19] They found little evidence of reliability, validity, and utility of these systems. Peebles et al showed that even when high‐risk non‐ICU patients are identified, delays in providing appropriate therapies occur, which may explain the lack of efficacy of EWSs and RRTs.[20] These observations suggest that there is currently a paucity of validated interventions available to improve outcome in deteriorating patients, despite our ability to identify patients who are at risk for such deterioration.

As a result of mandates from quality‐improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[21] However, as noted above, there is limited evidence to suggest that RRTs contribute to improved patient outcomes.[22, 23, 24, 25, 26, 27] The potential importance of this is reflected in a recent report suggesting that 2900 US hospitals now have rapid‐response systems in place without clear demonstration of their overall efficacy.[28] Linking rapid‐response interventions with a validated real‐time alert may represent a way of improving the effectiveness of such interventions.[29, 30, 31, 32, 33, 34] Our data showed that hospital LOS was statistically longer among alerted patients compared with nonalerted patients. This supports the conclusion that the alerts helped identify a sicker group of patients, but the nursing alerts did not appear to change outcomes. This finding also seems to refute the hypothesis that simply linking an intervention to a PT will improve outcomes, albeit the intervention we employed may not have been robust enough to influence patient outcomes.

The development of accurate real‐time EWSs holds the potential to identify patients at risk for clinical deterioration at an earlier point in time when rescue interventions can be implemented in a potentially more effective manner in both adults and children.[35] Unfortunately, the ideal intervention to be applied in this situation is unknown. Our experience suggests that successful interventions will require a more integrated approach than simply providing an alert with general management principles. As a result of our experience, we are undertaking a randomized clinical trial in 2013 to determine whether linking a patient‐specific intervention to a PT will result in improved outcomes. The intervention we will be testing is to have the RRT immediately notified about alerted patients so as to formally evaluate them and to determine the need for therapeutic interventions, and to administer such interventions as needed and/or transfer the alerted patients to a higher level of care as deemed necessary. Additionally, we are updating our PT with more temporal data to determine if this will improve its accuracy. One of these updates will include linking the PT to wirelessly obtained continuous oximetry and heart‐rate data, using minimally intrusive sensors, to establish a 2‐tiered EWS.[11]

Our study has several important limitations. First, the PT was developed using local data, and thus the results may not be applicable to other settings. However, our model shares many of the characteristics identified in other clinical‐deterioration PTs.[15, 17] Second, the positive prediction value of 15.2% for ICU transfer may not be clinically useful due to the large number of false‐positive results. Moreover, the large number of false positives could result in alert fatigue, causing alerts to be ignored. Third, although the charge nurses were supposed to call the responsible physicians for the alerted patients, we did not determine whether all these calls occurred or whether they resulted in any meaningful changes in monitoring or patient treatment. This is important because lack of an effective intervention or treatment would make the intervention group much more like our control group. Future studies are needed to assess the impact of an integrated intervention (eg, notification of experienced RRT members with adequate resource access) to determine if patient outcomes can be impacted by the use of an EWS. Finally, we did not compare the performance of our PT to other models such as the modified early warning score (MEWS).

An additional limitation to consider is that our PT offered no new information to the nurse manager, or the PT did not change the opinions of the charge nurses. This is supported by a recent study of 63 serious adverse outcomes in a Belgian teaching hospital where death was the final outcome.[36] Survey results revealed that nurses were often unaware that their patients were deteriorating before the crisis. Nurses also reported threshold levels for concern for abnormal vital signs that suggested they would call for assistance relatively late in clinical crises. The limited ability of nursing staff to identify deteriorating patients is also supported by a recent simulation study demonstrating that nurses did identify that patients were deteriorating, but as each patient deteriorated staff performance declined, with a reduction in all observational records and actions.[37]

In summary, we have demonstrated that a relatively simple hospital‐specific PT could accurately identify patients on general medicine wards who subsequently developed clinical deterioration and the need for ICU transfer, as well as hospital mortality. However, no improvements in patient outcomes were found from reporting this information to nursing wards on a real‐time basis. The low positive predictive value of the alerts, local development of the EWS, and absence of improved outcomes substantially limits the broader application of this system in its current form. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general wards but also intervene to improve their outcomes.