Marla N. Gardner, BA

Although the prevalence of coronary heart disease and death from acute myocardial infarction (AMI) have declined steadily, about 935,000 heart attacks still occur annually in the United States, with approximately one‐third of these being fatal.[1, 2, 3] Studies have demonstrated decreased 30‐day and longer‐term mortality in AMI patients who receive evidence‐based treatment, including aspirin, ‐blockers, angiotensin‐converting enzyme (ACE) inhibitors or angiotensin receptor blockers (ARBs), anticoagulation therapy, and statins.[4, 5, 6, 7] Despite clinical practice guidelines (CPGs) outlining evidence‐based care and considerable efforts to implement processes that improve patient outcomes, delivery of effective therapy remains suboptimal.[8] For example, the Hospital Quality Alliance Program[9] found that in AMI patients, use of aspirin on admission was only 81% to 92%, ‐blocker on admission 75% to 85%, and ACE inhibitors for left ventricular dysfunction 71% to 74%.

Efforts to increase adherence to CPGs and improve patient outcomes in AMI have resulted in variable degrees of success. They include promotion of CPGs,[4, 5, 6, 7] physician education with feedback, report cards, care paths, registries,[10] Joint Commission standardized measures,[11] and paper checklists or order sets (OS).[12, 13]

In this report, we describe the association between use of an evidence‐based, electronic OS for AMI (AMI‐OS) and better adherence to CPGs. This AMI‐OS was implemented in the inpatient electronic medical records (EMRs) of a large integrated healthcare delivery system, Kaiser Permanente Northern California (KPNC). The purpose of our investigation was to determine (1) whether use of the AMI‐OS was associated with improved AMI processes and patient outcomes, and (2) whether these associations persisted after risk adjustment using a comprehensive severity of illness scoring system.

MATERIALS AND METHODS

This project was approved by the KPNC institutional review board.

Under a mutual exclusivity arrangement, salaried physicians of The Permanente Medical Group, Inc., care for 3.4 million Kaiser Foundation Health Plan, Inc. members at facilities owned by Kaiser Foundation Hospitals, Inc. All KPNC facilities employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere.[14] Our setting consisted of 21 KPNC hospitals described in previous reports,[15, 16, 17, 18] using the same commercially available EMR system that includes computerized physician order entry (CPOE). Deployment of the customized inpatient Epic EMR (www.epicsystems.com), known internally as KP HealthConnect (KPHC), began in 2006 and was completed in 2010.

In this EMR's CPOE, physicians have options to select individual orders (a la carte) or they can utilize an OS, which is a collection of the most appropriate orders associated with specific diagnoses, procedures, or treatments. The evidence‐based AMI‐OS studied in this project was developed by a multidisciplinary team (for detailed components see Supporting Appendix 1Appendix 5 in the online version of this article).

Our study focused on the first set of hospital admission orders for patients with AMI. The study sample consisted of patients meeting these criteria: (1) age 18 years at admission; (2) admitted to a KPNC hospital for an overnight stay between September 28, 2008 and December 31, 2010; (3) principal diagnosis was AMI (International Classification of Diseases, 9th Revision [ICD‐9][19] codes 410.00, 01, 10, 11, 20, 21, 30, 31, 40, 41, 50, 51, 60, 61, 70, 71, 80, 90, and 91); and (4) KPHC had been operational at the hospital for at least 3 months to be included (for assembly descriptions see Supporting Appendices 15 in the online version of this article). At the study hospitals, troponin I was measured using the Beckman Access AccuTnI assay (Beckman Coulter, Inc., Brea, CA), whose upper reference limit (99th percentile) is 0.04 ng/mL. We excluded patients initially hospitalized for AMI at a non‐KPNC site and transferred into a study hospital.

The data processing methods we employed have been detailed elsewhere.[14, 15, 17, 20, 21, 22] The dependent outcome variables were total hospital length of stay, inpatient mortality, 30‐day mortality, and all‐cause rehospitalization within 30 days of discharge. Linked state mortality data were unavailable for the entire study period, so we ascertained 30‐day mortality based on the combination of KPNC patient demographic data and publicly available Social Security Administration decedent files. We ascertained rehospitalization by scanning KPNC hospitalization databases, which also track out‐of‐plan use.

The dependent process variables were use of aspirin within 24 hours of admission, ‐blockers, anticoagulation, ACE inhibitors or ARBs, and statins. The primary independent variable of interest was whether or not the admitting physician employed the AMI‐OS when admission orders were entered. Consequently, this variable is dichotomous (AMI‐OS vs a la carte).

We controlled for acute illness severity and chronic illness burden using a recent modification[22] of an externally validated risk‐adjustment system applicable to all hospitalized patients.[15, 16, 23, 24, 25] Our methodology included vital signs, neurological status checks, and laboratory test results obtained in the 72 hours preceding hospital admission; comorbidities were captured longitudinally using data from the year preceding hospitalization (for comparison purposes, we also assigned a Charlson Comorbidity Index score[26]).

End‐of‐life care directives are mandatory on admission at KPNC hospitals. Physicians have 4 options: full code, partial code, do not resuscitate, and comfort care only. Because of small numbers in some categories, we collapsed these 4 categories into full code and not full code. Because patients' care directives may change, we elected to capture the care directive in effect when a patient first entered a hospital unit other than the emergency department (ED).

Two authors (M.B., P.C.L.), one of whom is a board‐certified cardiologist, reviewed all admission electrocardiograms and made a consensus determination as to whether or not criteria for ST‐segment elevation myocardial infarction (STEMI) were present (ie, new ST‐segment elevation or left bundle branch block); we also reviewed the records of all patients with missing troponin I data to confirm the AMI diagnosis.

RESULTS

Table 1 summarizes the characteristics of the 5879 patients. It shows that AMI‐OS patients were more likely to receive evidence‐based therapies for AMI (aspirin, ‐blockers, ACE inhibitors or ARBs, anticoagulation, and statins) and had a 46% lower mortality rate in hospital (3.51 % vs 6.52%) and 33% lower rate at 30 days (5.66% vs 8.48%). AMI‐OS patients were also found to be at lower risk for an adverse outcome than nonAMI‐OS patients. The AMI‐OS patients had lower peak troponin I values, severity of illness (lower Laboratory‐Based Acute Physiology Score, version 2 [LAPS2] scores), comorbidity burdens (lower Comorbidity Point Score, version 2 [COPS2] and Charlson scores), and global predicted mortality risk. AMI‐OS patients were also less likely to have required intensive care. AMI‐OS patients were at higher risk of death than nonAMI‐OS patients with respect to only 1 variable (being full code at the time of admission), but although this difference was statistically significant, it was of minor clinical impact (86% vs 88%).

Table 2 shows the result of a logistic regression model in which the dependent variable was inpatient mortality and either the 5 evidence‐based therapies or the total number of evidence‐based therapies are the dependent variables. ‐blocker, statin, and ACE inhibitor or ARB therapies all had a protective effect on mortality, with odds ratios ranging from 0.48 (95% confidence interval [CI]: 0.36‐0.64), 0.63 (95% CI: 0.45‐0.89), and 0.40 (95% CI: 0.30‐0.53), respectively. An increased number of therapies also had a beneficial effect on inpatient mortality, with patients having 3 or more of the evidence‐based therapies showing an adjusted odds ratio (AOR) of 0.49 (95% CI: 0.33‐0.73), 4 or more therapies an AOR of 0.29 (95% CI: 0.20‐0.42), and 0.17 (95% CI: 0.11‐0.25) for 5 or more therapies.

Table 3 shows that the use of the AMI‐OS is protective, with an AOR of 0.59 and a 95% CI of 0.45‐0.76. Table 3 also shows that the most potent predictors were comorbidity burden (AOR: 1.07, 95% CI: 1.03‐1.10 per 10 COPS2 points), severity of illness (AOR: 1.09, 95% CI: 1.07‐1.12 per 10 LAPS2 points), STEMI (AOR: 3.86, 95% CI: 2.68‐5.58), and timing of cardiac catheterization referral occurring immediately prior to or during the admission (AOR: 0.37, 95% CI: 0.27‐0.51). The statistical significance of the AMI‐OS effect disappears when both AMI‐OS and the individual therapies are included in the same model (see Supporting Information, Appendices 15, in the online version of this article).

Table 4 shows separately the average treatment effect (ATE) and average treatment effect for the treated (ATT) of AMI‐OS and of increasing number of therapies on other outcomes (30‐day mortality, LOS, and readmission). Both the ATE and ATT show that the use of the AMI‐OS was significantly protective with respect to mortality and total hospital LOS but not significant with respect to readmission. The effect of the number of therapies on mortality is significantly higher with increasing number of therapies. For example, patients who received 5 therapies had an average treatment effect on 30‐day inpatient mortality of 0.23 (95% CI: 0.15‐0.35) compared to 0.64 (95% CI: 0.43‐0.96) for 3 therapies, almost a 3‐fold difference. The effects of increasing number of therapies were not significant for LOS or readmission. A sensitivity analysis in which the 535 STEMI patients were removed showed essentially the same results, so it is not reported here.

To further elucidate possible reasons why physicians did not use the AMI‐OS, the lead author reviewed 105 randomly selected records where the AMI‐OS was not used, 5 records from each of the 21 study hospitals. This review found that in 36% of patients, the AMI‐OS was not used because emergent catheterization or transfer to a facility with percutaneous coronary intervention capability occurred. Presence of other significant medical conditions, including critical illness, was the reason in 17% of these cases, patient or family refusal of treatments in 8%, issues around end‐of‐life care in 3%, and specific medical contraindications in 1%. In the remaining 34%, no reason for not using the AMI‐OS could be identified.

DISCUSSION

We evaluated the use of an evidence‐based electronic AMI‐OS embedded in a comprehensive EMR and found that it was beneficial. Its use was associated with increased adherence to evidence‐based therapies, which in turn were associated with improved outcomes. Using data from a large cohort of hospitalized AMI patients in 21 community hospitals, we were able to use risk adjustment that included physiologic illness severity to adjust for baseline mortality risk. Patients in whom the AMI‐OS was employed tended to be at lower risk; nonetheless, after controlling for confounding variables and adjusting for bias using propensity scores, the AMI‐OS was associated with increased use of evidence‐based therapies and decreased mortality. Most importantly, it appears that the benefits of the OS were not just due to increased receipt of individual recommended therapies, but to increased concurrent receipt of multiple recommended therapies.

Modern EMRs have great potential for significant improvements in the quality, efficiency, and safety of care provided,[36] and our study highlights this potential. However, a number of important limitations to our study must be considered. Although we had access to a very rich dataset, we could not control for all possible confounders, and our risk adjustment cannot match the level of information available to clinicians. In particular, the measurements available to us with respect to cardiac risk are limited. Thus, we have to recognize that the strength of our findings does not approximate that of a randomized trial, and one would expect that the magnitude of the beneficial association would fall under more controlled conditions. Resource limitations also did not permit us to gather more time course data (eg, sequential measurements of patient instability, cardiac damage, or use of recommended therapies), which could provide a better delineation of differences in both processes and outcomes.

Limitations also exist to the generalizability of the use of order sets in other settings that go beyond the availability of a comprehensive EMR. Our study population was cared for in a setting with an unusually high level of integration.[1] For example, KPNC has an elaborate administrative infrastructure for training in the use of the EMR as well as ensuring that order sets are not just evidence‐based, but that they are perceived by clinicians to be of significant value. This infrastructure, established to ensure physician buy‐in, may not be easy to replicate in smaller or less‐integrated settings. Thus, it is conceivable that factors other than the degree of support during the EMR deployments can affect rates of order set use.

Although our use of counterfactual methods included illness severity (LAPS2) and longitudinal comorbidity burden (COPS2), which are not yet available outside highly integrated delivery services employing comprehensive EMRs, it is possible they are insufficient. We cannot exclude the possibility that other biases or patient characteristics were present that led clinicians to preferentially employ the electronic order set in some patients but not in others. One could also argue that future studies should consider using overall adherence to recommended AMI treatment guidelines as a risk adjustment tool that would permit one to analyze what other factors may be playing a role in residual differences in patient outcomes. Last, one could object to our inclusion of STEMI patients; however, this was not a study on optimum treatment strategies for STEMI patients. Rather, it was a study on the impact on AMI outcomes of a specific component of computerized order entry outside the research setting.

Despite these limitations, we believe that our findings provide strong support for the continued use of electronic evidence‐based order sets in the inpatient medical setting. Once the initial implementation of a comprehensive EMR has occurred, deployment of these electronic order sets is a relatively inexpensive but effective method to foster compliance with evidence‐based care.

Future research in healthcare information technology can take a number of directions. One important area, of course, revolves around ways to promote enhanced physician adoption of EMRs. Our audit of records where the AMI‐OS was not used found that specific reasons for not using the order set (eg, treatment refusals, emergent intervention) were present in two‐thirds of the cases. This suggests that future analyses of adherence involving EMRs and CPOE implementation should take a more nuanced look at how order entry is actually enabled. It may be that understanding how order sets affect care enhances clinician acceptance and thus could serve as an incentive to EMR adoption. However, once an EMR is adopted, a need exists to continue evaluations such as this because, ultimately, the gold standard should be improved patient care processes and better outcomes for patients.

Although the prevalence of coronary heart disease and death from acute myocardial infarction (AMI) have declined steadily, about 935,000 heart attacks still occur annually in the United States, with approximately one‐third of these being fatal.[1, 2, 3] Studies have demonstrated decreased 30‐day and longer‐term mortality in AMI patients who receive evidence‐based treatment, including aspirin, ‐blockers, angiotensin‐converting enzyme (ACE) inhibitors or angiotensin receptor blockers (ARBs), anticoagulation therapy, and statins.[4, 5, 6, 7] Despite clinical practice guidelines (CPGs) outlining evidence‐based care and considerable efforts to implement processes that improve patient outcomes, delivery of effective therapy remains suboptimal.[8] For example, the Hospital Quality Alliance Program[9] found that in AMI patients, use of aspirin on admission was only 81% to 92%, ‐blocker on admission 75% to 85%, and ACE inhibitors for left ventricular dysfunction 71% to 74%.

Efforts to increase adherence to CPGs and improve patient outcomes in AMI have resulted in variable degrees of success. They include promotion of CPGs,[4, 5, 6, 7] physician education with feedback, report cards, care paths, registries,[10] Joint Commission standardized measures,[11] and paper checklists or order sets (OS).[12, 13]

In this report, we describe the association between use of an evidence‐based, electronic OS for AMI (AMI‐OS) and better adherence to CPGs. This AMI‐OS was implemented in the inpatient electronic medical records (EMRs) of a large integrated healthcare delivery system, Kaiser Permanente Northern California (KPNC). The purpose of our investigation was to determine (1) whether use of the AMI‐OS was associated with improved AMI processes and patient outcomes, and (2) whether these associations persisted after risk adjustment using a comprehensive severity of illness scoring system.

MATERIALS AND METHODS

This project was approved by the KPNC institutional review board.

Under a mutual exclusivity arrangement, salaried physicians of The Permanente Medical Group, Inc., care for 3.4 million Kaiser Foundation Health Plan, Inc. members at facilities owned by Kaiser Foundation Hospitals, Inc. All KPNC facilities employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere.[14] Our setting consisted of 21 KPNC hospitals described in previous reports,[15, 16, 17, 18] using the same commercially available EMR system that includes computerized physician order entry (CPOE). Deployment of the customized inpatient Epic EMR (www.epicsystems.com), known internally as KP HealthConnect (KPHC), began in 2006 and was completed in 2010.

In this EMR's CPOE, physicians have options to select individual orders (a la carte) or they can utilize an OS, which is a collection of the most appropriate orders associated with specific diagnoses, procedures, or treatments. The evidence‐based AMI‐OS studied in this project was developed by a multidisciplinary team (for detailed components see Supporting Appendix 1Appendix 5 in the online version of this article).

Our study focused on the first set of hospital admission orders for patients with AMI. The study sample consisted of patients meeting these criteria: (1) age 18 years at admission; (2) admitted to a KPNC hospital for an overnight stay between September 28, 2008 and December 31, 2010; (3) principal diagnosis was AMI (International Classification of Diseases, 9th Revision [ICD‐9][19] codes 410.00, 01, 10, 11, 20, 21, 30, 31, 40, 41, 50, 51, 60, 61, 70, 71, 80, 90, and 91); and (4) KPHC had been operational at the hospital for at least 3 months to be included (for assembly descriptions see Supporting Appendices 15 in the online version of this article). At the study hospitals, troponin I was measured using the Beckman Access AccuTnI assay (Beckman Coulter, Inc., Brea, CA), whose upper reference limit (99th percentile) is 0.04 ng/mL. We excluded patients initially hospitalized for AMI at a non‐KPNC site and transferred into a study hospital.

The data processing methods we employed have been detailed elsewhere.[14, 15, 17, 20, 21, 22] The dependent outcome variables were total hospital length of stay, inpatient mortality, 30‐day mortality, and all‐cause rehospitalization within 30 days of discharge. Linked state mortality data were unavailable for the entire study period, so we ascertained 30‐day mortality based on the combination of KPNC patient demographic data and publicly available Social Security Administration decedent files. We ascertained rehospitalization by scanning KPNC hospitalization databases, which also track out‐of‐plan use.

The dependent process variables were use of aspirin within 24 hours of admission, ‐blockers, anticoagulation, ACE inhibitors or ARBs, and statins. The primary independent variable of interest was whether or not the admitting physician employed the AMI‐OS when admission orders were entered. Consequently, this variable is dichotomous (AMI‐OS vs a la carte).

We controlled for acute illness severity and chronic illness burden using a recent modification[22] of an externally validated risk‐adjustment system applicable to all hospitalized patients.[15, 16, 23, 24, 25] Our methodology included vital signs, neurological status checks, and laboratory test results obtained in the 72 hours preceding hospital admission; comorbidities were captured longitudinally using data from the year preceding hospitalization (for comparison purposes, we also assigned a Charlson Comorbidity Index score[26]).

End‐of‐life care directives are mandatory on admission at KPNC hospitals. Physicians have 4 options: full code, partial code, do not resuscitate, and comfort care only. Because of small numbers in some categories, we collapsed these 4 categories into full code and not full code. Because patients' care directives may change, we elected to capture the care directive in effect when a patient first entered a hospital unit other than the emergency department (ED).

Two authors (M.B., P.C.L.), one of whom is a board‐certified cardiologist, reviewed all admission electrocardiograms and made a consensus determination as to whether or not criteria for ST‐segment elevation myocardial infarction (STEMI) were present (ie, new ST‐segment elevation or left bundle branch block); we also reviewed the records of all patients with missing troponin I data to confirm the AMI diagnosis.

RESULTS

Table 1 summarizes the characteristics of the 5879 patients. It shows that AMI‐OS patients were more likely to receive evidence‐based therapies for AMI (aspirin, ‐blockers, ACE inhibitors or ARBs, anticoagulation, and statins) and had a 46% lower mortality rate in hospital (3.51 % vs 6.52%) and 33% lower rate at 30 days (5.66% vs 8.48%). AMI‐OS patients were also found to be at lower risk for an adverse outcome than nonAMI‐OS patients. The AMI‐OS patients had lower peak troponin I values, severity of illness (lower Laboratory‐Based Acute Physiology Score, version 2 [LAPS2] scores), comorbidity burdens (lower Comorbidity Point Score, version 2 [COPS2] and Charlson scores), and global predicted mortality risk. AMI‐OS patients were also less likely to have required intensive care. AMI‐OS patients were at higher risk of death than nonAMI‐OS patients with respect to only 1 variable (being full code at the time of admission), but although this difference was statistically significant, it was of minor clinical impact (86% vs 88%).

Table 2 shows the result of a logistic regression model in which the dependent variable was inpatient mortality and either the 5 evidence‐based therapies or the total number of evidence‐based therapies are the dependent variables. ‐blocker, statin, and ACE inhibitor or ARB therapies all had a protective effect on mortality, with odds ratios ranging from 0.48 (95% confidence interval [CI]: 0.36‐0.64), 0.63 (95% CI: 0.45‐0.89), and 0.40 (95% CI: 0.30‐0.53), respectively. An increased number of therapies also had a beneficial effect on inpatient mortality, with patients having 3 or more of the evidence‐based therapies showing an adjusted odds ratio (AOR) of 0.49 (95% CI: 0.33‐0.73), 4 or more therapies an AOR of 0.29 (95% CI: 0.20‐0.42), and 0.17 (95% CI: 0.11‐0.25) for 5 or more therapies.

Table 3 shows that the use of the AMI‐OS is protective, with an AOR of 0.59 and a 95% CI of 0.45‐0.76. Table 3 also shows that the most potent predictors were comorbidity burden (AOR: 1.07, 95% CI: 1.03‐1.10 per 10 COPS2 points), severity of illness (AOR: 1.09, 95% CI: 1.07‐1.12 per 10 LAPS2 points), STEMI (AOR: 3.86, 95% CI: 2.68‐5.58), and timing of cardiac catheterization referral occurring immediately prior to or during the admission (AOR: 0.37, 95% CI: 0.27‐0.51). The statistical significance of the AMI‐OS effect disappears when both AMI‐OS and the individual therapies are included in the same model (see Supporting Information, Appendices 15, in the online version of this article).

Table 4 shows separately the average treatment effect (ATE) and average treatment effect for the treated (ATT) of AMI‐OS and of increasing number of therapies on other outcomes (30‐day mortality, LOS, and readmission). Both the ATE and ATT show that the use of the AMI‐OS was significantly protective with respect to mortality and total hospital LOS but not significant with respect to readmission. The effect of the number of therapies on mortality is significantly higher with increasing number of therapies. For example, patients who received 5 therapies had an average treatment effect on 30‐day inpatient mortality of 0.23 (95% CI: 0.15‐0.35) compared to 0.64 (95% CI: 0.43‐0.96) for 3 therapies, almost a 3‐fold difference. The effects of increasing number of therapies were not significant for LOS or readmission. A sensitivity analysis in which the 535 STEMI patients were removed showed essentially the same results, so it is not reported here.

To further elucidate possible reasons why physicians did not use the AMI‐OS, the lead author reviewed 105 randomly selected records where the AMI‐OS was not used, 5 records from each of the 21 study hospitals. This review found that in 36% of patients, the AMI‐OS was not used because emergent catheterization or transfer to a facility with percutaneous coronary intervention capability occurred. Presence of other significant medical conditions, including critical illness, was the reason in 17% of these cases, patient or family refusal of treatments in 8%, issues around end‐of‐life care in 3%, and specific medical contraindications in 1%. In the remaining 34%, no reason for not using the AMI‐OS could be identified.

DISCUSSION

We evaluated the use of an evidence‐based electronic AMI‐OS embedded in a comprehensive EMR and found that it was beneficial. Its use was associated with increased adherence to evidence‐based therapies, which in turn were associated with improved outcomes. Using data from a large cohort of hospitalized AMI patients in 21 community hospitals, we were able to use risk adjustment that included physiologic illness severity to adjust for baseline mortality risk. Patients in whom the AMI‐OS was employed tended to be at lower risk; nonetheless, after controlling for confounding variables and adjusting for bias using propensity scores, the AMI‐OS was associated with increased use of evidence‐based therapies and decreased mortality. Most importantly, it appears that the benefits of the OS were not just due to increased receipt of individual recommended therapies, but to increased concurrent receipt of multiple recommended therapies.

Modern EMRs have great potential for significant improvements in the quality, efficiency, and safety of care provided,[36] and our study highlights this potential. However, a number of important limitations to our study must be considered. Although we had access to a very rich dataset, we could not control for all possible confounders, and our risk adjustment cannot match the level of information available to clinicians. In particular, the measurements available to us with respect to cardiac risk are limited. Thus, we have to recognize that the strength of our findings does not approximate that of a randomized trial, and one would expect that the magnitude of the beneficial association would fall under more controlled conditions. Resource limitations also did not permit us to gather more time course data (eg, sequential measurements of patient instability, cardiac damage, or use of recommended therapies), which could provide a better delineation of differences in both processes and outcomes.

Limitations also exist to the generalizability of the use of order sets in other settings that go beyond the availability of a comprehensive EMR. Our study population was cared for in a setting with an unusually high level of integration.[1] For example, KPNC has an elaborate administrative infrastructure for training in the use of the EMR as well as ensuring that order sets are not just evidence‐based, but that they are perceived by clinicians to be of significant value. This infrastructure, established to ensure physician buy‐in, may not be easy to replicate in smaller or less‐integrated settings. Thus, it is conceivable that factors other than the degree of support during the EMR deployments can affect rates of order set use.

Although our use of counterfactual methods included illness severity (LAPS2) and longitudinal comorbidity burden (COPS2), which are not yet available outside highly integrated delivery services employing comprehensive EMRs, it is possible they are insufficient. We cannot exclude the possibility that other biases or patient characteristics were present that led clinicians to preferentially employ the electronic order set in some patients but not in others. One could also argue that future studies should consider using overall adherence to recommended AMI treatment guidelines as a risk adjustment tool that would permit one to analyze what other factors may be playing a role in residual differences in patient outcomes. Last, one could object to our inclusion of STEMI patients; however, this was not a study on optimum treatment strategies for STEMI patients. Rather, it was a study on the impact on AMI outcomes of a specific component of computerized order entry outside the research setting.

Despite these limitations, we believe that our findings provide strong support for the continued use of electronic evidence‐based order sets in the inpatient medical setting. Once the initial implementation of a comprehensive EMR has occurred, deployment of these electronic order sets is a relatively inexpensive but effective method to foster compliance with evidence‐based care.

Future research in healthcare information technology can take a number of directions. One important area, of course, revolves around ways to promote enhanced physician adoption of EMRs. Our audit of records where the AMI‐OS was not used found that specific reasons for not using the order set (eg, treatment refusals, emergent intervention) were present in two‐thirds of the cases. This suggests that future analyses of adherence involving EMRs and CPOE implementation should take a more nuanced look at how order entry is actually enabled. It may be that understanding how order sets affect care enhances clinician acceptance and thus could serve as an incentive to EMR adoption. However, once an EMR is adopted, a need exists to continue evaluations such as this because, ultimately, the gold standard should be improved patient care processes and better outcomes for patients.

Although the prevalence of coronary heart disease and death from acute myocardial infarction (AMI) have declined steadily, about 935,000 heart attacks still occur annually in the United States, with approximately one‐third of these being fatal.[1, 2, 3] Studies have demonstrated decreased 30‐day and longer‐term mortality in AMI patients who receive evidence‐based treatment, including aspirin, ‐blockers, angiotensin‐converting enzyme (ACE) inhibitors or angiotensin receptor blockers (ARBs), anticoagulation therapy, and statins.[4, 5, 6, 7] Despite clinical practice guidelines (CPGs) outlining evidence‐based care and considerable efforts to implement processes that improve patient outcomes, delivery of effective therapy remains suboptimal.[8] For example, the Hospital Quality Alliance Program[9] found that in AMI patients, use of aspirin on admission was only 81% to 92%, ‐blocker on admission 75% to 85%, and ACE inhibitors for left ventricular dysfunction 71% to 74%.

Efforts to increase adherence to CPGs and improve patient outcomes in AMI have resulted in variable degrees of success. They include promotion of CPGs,[4, 5, 6, 7] physician education with feedback, report cards, care paths, registries,[10] Joint Commission standardized measures,[11] and paper checklists or order sets (OS).[12, 13]

In this report, we describe the association between use of an evidence‐based, electronic OS for AMI (AMI‐OS) and better adherence to CPGs. This AMI‐OS was implemented in the inpatient electronic medical records (EMRs) of a large integrated healthcare delivery system, Kaiser Permanente Northern California (KPNC). The purpose of our investigation was to determine (1) whether use of the AMI‐OS was associated with improved AMI processes and patient outcomes, and (2) whether these associations persisted after risk adjustment using a comprehensive severity of illness scoring system.

MATERIALS AND METHODS

This project was approved by the KPNC institutional review board.

Under a mutual exclusivity arrangement, salaried physicians of The Permanente Medical Group, Inc., care for 3.4 million Kaiser Foundation Health Plan, Inc. members at facilities owned by Kaiser Foundation Hospitals, Inc. All KPNC facilities employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere.[14] Our setting consisted of 21 KPNC hospitals described in previous reports,[15, 16, 17, 18] using the same commercially available EMR system that includes computerized physician order entry (CPOE). Deployment of the customized inpatient Epic EMR (www.epicsystems.com), known internally as KP HealthConnect (KPHC), began in 2006 and was completed in 2010.

In this EMR's CPOE, physicians have options to select individual orders (a la carte) or they can utilize an OS, which is a collection of the most appropriate orders associated with specific diagnoses, procedures, or treatments. The evidence‐based AMI‐OS studied in this project was developed by a multidisciplinary team (for detailed components see Supporting Appendix 1Appendix 5 in the online version of this article).

Our study focused on the first set of hospital admission orders for patients with AMI. The study sample consisted of patients meeting these criteria: (1) age 18 years at admission; (2) admitted to a KPNC hospital for an overnight stay between September 28, 2008 and December 31, 2010; (3) principal diagnosis was AMI (International Classification of Diseases, 9th Revision [ICD‐9][19] codes 410.00, 01, 10, 11, 20, 21, 30, 31, 40, 41, 50, 51, 60, 61, 70, 71, 80, 90, and 91); and (4) KPHC had been operational at the hospital for at least 3 months to be included (for assembly descriptions see Supporting Appendices 15 in the online version of this article). At the study hospitals, troponin I was measured using the Beckman Access AccuTnI assay (Beckman Coulter, Inc., Brea, CA), whose upper reference limit (99th percentile) is 0.04 ng/mL. We excluded patients initially hospitalized for AMI at a non‐KPNC site and transferred into a study hospital.

The data processing methods we employed have been detailed elsewhere.[14, 15, 17, 20, 21, 22] The dependent outcome variables were total hospital length of stay, inpatient mortality, 30‐day mortality, and all‐cause rehospitalization within 30 days of discharge. Linked state mortality data were unavailable for the entire study period, so we ascertained 30‐day mortality based on the combination of KPNC patient demographic data and publicly available Social Security Administration decedent files. We ascertained rehospitalization by scanning KPNC hospitalization databases, which also track out‐of‐plan use.

The dependent process variables were use of aspirin within 24 hours of admission, ‐blockers, anticoagulation, ACE inhibitors or ARBs, and statins. The primary independent variable of interest was whether or not the admitting physician employed the AMI‐OS when admission orders were entered. Consequently, this variable is dichotomous (AMI‐OS vs a la carte).

We controlled for acute illness severity and chronic illness burden using a recent modification[22] of an externally validated risk‐adjustment system applicable to all hospitalized patients.[15, 16, 23, 24, 25] Our methodology included vital signs, neurological status checks, and laboratory test results obtained in the 72 hours preceding hospital admission; comorbidities were captured longitudinally using data from the year preceding hospitalization (for comparison purposes, we also assigned a Charlson Comorbidity Index score[26]).

End‐of‐life care directives are mandatory on admission at KPNC hospitals. Physicians have 4 options: full code, partial code, do not resuscitate, and comfort care only. Because of small numbers in some categories, we collapsed these 4 categories into full code and not full code. Because patients' care directives may change, we elected to capture the care directive in effect when a patient first entered a hospital unit other than the emergency department (ED).

Two authors (M.B., P.C.L.), one of whom is a board‐certified cardiologist, reviewed all admission electrocardiograms and made a consensus determination as to whether or not criteria for ST‐segment elevation myocardial infarction (STEMI) were present (ie, new ST‐segment elevation or left bundle branch block); we also reviewed the records of all patients with missing troponin I data to confirm the AMI diagnosis.

RESULTS

Table 1 summarizes the characteristics of the 5879 patients. It shows that AMI‐OS patients were more likely to receive evidence‐based therapies for AMI (aspirin, ‐blockers, ACE inhibitors or ARBs, anticoagulation, and statins) and had a 46% lower mortality rate in hospital (3.51 % vs 6.52%) and 33% lower rate at 30 days (5.66% vs 8.48%). AMI‐OS patients were also found to be at lower risk for an adverse outcome than nonAMI‐OS patients. The AMI‐OS patients had lower peak troponin I values, severity of illness (lower Laboratory‐Based Acute Physiology Score, version 2 [LAPS2] scores), comorbidity burdens (lower Comorbidity Point Score, version 2 [COPS2] and Charlson scores), and global predicted mortality risk. AMI‐OS patients were also less likely to have required intensive care. AMI‐OS patients were at higher risk of death than nonAMI‐OS patients with respect to only 1 variable (being full code at the time of admission), but although this difference was statistically significant, it was of minor clinical impact (86% vs 88%).

Table 2 shows the result of a logistic regression model in which the dependent variable was inpatient mortality and either the 5 evidence‐based therapies or the total number of evidence‐based therapies are the dependent variables. ‐blocker, statin, and ACE inhibitor or ARB therapies all had a protective effect on mortality, with odds ratios ranging from 0.48 (95% confidence interval [CI]: 0.36‐0.64), 0.63 (95% CI: 0.45‐0.89), and 0.40 (95% CI: 0.30‐0.53), respectively. An increased number of therapies also had a beneficial effect on inpatient mortality, with patients having 3 or more of the evidence‐based therapies showing an adjusted odds ratio (AOR) of 0.49 (95% CI: 0.33‐0.73), 4 or more therapies an AOR of 0.29 (95% CI: 0.20‐0.42), and 0.17 (95% CI: 0.11‐0.25) for 5 or more therapies.

Table 3 shows that the use of the AMI‐OS is protective, with an AOR of 0.59 and a 95% CI of 0.45‐0.76. Table 3 also shows that the most potent predictors were comorbidity burden (AOR: 1.07, 95% CI: 1.03‐1.10 per 10 COPS2 points), severity of illness (AOR: 1.09, 95% CI: 1.07‐1.12 per 10 LAPS2 points), STEMI (AOR: 3.86, 95% CI: 2.68‐5.58), and timing of cardiac catheterization referral occurring immediately prior to or during the admission (AOR: 0.37, 95% CI: 0.27‐0.51). The statistical significance of the AMI‐OS effect disappears when both AMI‐OS and the individual therapies are included in the same model (see Supporting Information, Appendices 15, in the online version of this article).

Table 4 shows separately the average treatment effect (ATE) and average treatment effect for the treated (ATT) of AMI‐OS and of increasing number of therapies on other outcomes (30‐day mortality, LOS, and readmission). Both the ATE and ATT show that the use of the AMI‐OS was significantly protective with respect to mortality and total hospital LOS but not significant with respect to readmission. The effect of the number of therapies on mortality is significantly higher with increasing number of therapies. For example, patients who received 5 therapies had an average treatment effect on 30‐day inpatient mortality of 0.23 (95% CI: 0.15‐0.35) compared to 0.64 (95% CI: 0.43‐0.96) for 3 therapies, almost a 3‐fold difference. The effects of increasing number of therapies were not significant for LOS or readmission. A sensitivity analysis in which the 535 STEMI patients were removed showed essentially the same results, so it is not reported here.

To further elucidate possible reasons why physicians did not use the AMI‐OS, the lead author reviewed 105 randomly selected records where the AMI‐OS was not used, 5 records from each of the 21 study hospitals. This review found that in 36% of patients, the AMI‐OS was not used because emergent catheterization or transfer to a facility with percutaneous coronary intervention capability occurred. Presence of other significant medical conditions, including critical illness, was the reason in 17% of these cases, patient or family refusal of treatments in 8%, issues around end‐of‐life care in 3%, and specific medical contraindications in 1%. In the remaining 34%, no reason for not using the AMI‐OS could be identified.

DISCUSSION

We evaluated the use of an evidence‐based electronic AMI‐OS embedded in a comprehensive EMR and found that it was beneficial. Its use was associated with increased adherence to evidence‐based therapies, which in turn were associated with improved outcomes. Using data from a large cohort of hospitalized AMI patients in 21 community hospitals, we were able to use risk adjustment that included physiologic illness severity to adjust for baseline mortality risk. Patients in whom the AMI‐OS was employed tended to be at lower risk; nonetheless, after controlling for confounding variables and adjusting for bias using propensity scores, the AMI‐OS was associated with increased use of evidence‐based therapies and decreased mortality. Most importantly, it appears that the benefits of the OS were not just due to increased receipt of individual recommended therapies, but to increased concurrent receipt of multiple recommended therapies.

Modern EMRs have great potential for significant improvements in the quality, efficiency, and safety of care provided,[36] and our study highlights this potential. However, a number of important limitations to our study must be considered. Although we had access to a very rich dataset, we could not control for all possible confounders, and our risk adjustment cannot match the level of information available to clinicians. In particular, the measurements available to us with respect to cardiac risk are limited. Thus, we have to recognize that the strength of our findings does not approximate that of a randomized trial, and one would expect that the magnitude of the beneficial association would fall under more controlled conditions. Resource limitations also did not permit us to gather more time course data (eg, sequential measurements of patient instability, cardiac damage, or use of recommended therapies), which could provide a better delineation of differences in both processes and outcomes.

Limitations also exist to the generalizability of the use of order sets in other settings that go beyond the availability of a comprehensive EMR. Our study population was cared for in a setting with an unusually high level of integration.[1] For example, KPNC has an elaborate administrative infrastructure for training in the use of the EMR as well as ensuring that order sets are not just evidence‐based, but that they are perceived by clinicians to be of significant value. This infrastructure, established to ensure physician buy‐in, may not be easy to replicate in smaller or less‐integrated settings. Thus, it is conceivable that factors other than the degree of support during the EMR deployments can affect rates of order set use.

Although our use of counterfactual methods included illness severity (LAPS2) and longitudinal comorbidity burden (COPS2), which are not yet available outside highly integrated delivery services employing comprehensive EMRs, it is possible they are insufficient. We cannot exclude the possibility that other biases or patient characteristics were present that led clinicians to preferentially employ the electronic order set in some patients but not in others. One could also argue that future studies should consider using overall adherence to recommended AMI treatment guidelines as a risk adjustment tool that would permit one to analyze what other factors may be playing a role in residual differences in patient outcomes. Last, one could object to our inclusion of STEMI patients; however, this was not a study on optimum treatment strategies for STEMI patients. Rather, it was a study on the impact on AMI outcomes of a specific component of computerized order entry outside the research setting.

Despite these limitations, we believe that our findings provide strong support for the continued use of electronic evidence‐based order sets in the inpatient medical setting. Once the initial implementation of a comprehensive EMR has occurred, deployment of these electronic order sets is a relatively inexpensive but effective method to foster compliance with evidence‐based care.

Future research in healthcare information technology can take a number of directions. One important area, of course, revolves around ways to promote enhanced physician adoption of EMRs. Our audit of records where the AMI‐OS was not used found that specific reasons for not using the order set (eg, treatment refusals, emergent intervention) were present in two‐thirds of the cases. This suggests that future analyses of adherence involving EMRs and CPOE implementation should take a more nuanced look at how order entry is actually enabled. It may be that understanding how order sets affect care enhances clinician acceptance and thus could serve as an incentive to EMR adoption. However, once an EMR is adopted, a need exists to continue evaluations such as this because, ultimately, the gold standard should be improved patient care processes and better outcomes for patients.

Emergency Department (ED) patients who are hospitalized and require unplanned transfer to the intensive care unit (ICU) within 24 hours of arrival on the ward have higher mortality than direct ICU admissions.1, 2 Previous research found that 5% of ED admissions experienced unplanned ICU transfer during their hospitalization, yet these patients account for 25% of in‐hospital deaths and have a longer length of stay than direct ICU admissions.1, 3 For these reasons, inpatient rapid‐response teams and early warning systems have been studied to reduce the mortality of patients who rapidly deteriorate on the hospital ward.410 However, there is little conclusive evidence that these interventions decrease mortality.710 It is possible that with better recognition and intervention in the ED, a portion of these unplanned ICU transfers and their subsequent adverse outcomes could be prevented.11

Previous research on risk factors for unplanned ICU transfers among ED admissions is limited. While 2 previous studies from non‐US hospitals used administrative data to identify some general populations at risk for unplanned ICU transfer,12, 13 these studies did not differentiate between transfers shortly after admission and those that occurred during a prolonged hospital staya critical distinction since the outcomes between these groups differs substantially.1 Another limitation of these studies is the absence of physiologic measures at ED presentation, which have been shown to be highly predictive of mortality.14

In this study, we describe risk factors for unplanned transfer to the ICU within 24 hours of arrival on the ward, among a large cohort of ED hospitalizations across 13 community hospitals. Focusing on admitting diagnoses most at risk, our goal was to inform efforts to improve the triage of ED admissions and determine which patients may benefit from additional interventions, such as improved resuscitation, closer monitoring, or risk stratification tools. We also hypothesized that higher volume hospitals would have lower rates of unplanned ICU transfers, as these hospitals are more likely have more patient care resources on the hospital ward and a higher threshold to transfer to the ICU.

METHODS

RESULTS

Of 178,315 ED non‐ICU hospitalizations meeting inclusion criteria, 4,252 (2.4%) were admitted to the ward and were transferred to the ICU within 24 hours of leaving the ED. There were 122,251 unique patients in our study population. Table 1 compares the characteristics of ED hospitalizations in which an unplanned transfer occurred to those that did not experience an unplanned transfer. Unplanned transfers were more likely to have a higher comorbidity burden, more deranged physiology, and more likely to arrive on the floor during the overnight shift.

Unplanned ICU transfers were more frequent in lower volume hospitals (Table 1). Figure 1 displays the inverse relationship between hospital annual ED inpatient admission volume and unplanned ICU transfers rates. The lowest volume hospital had a crude rate twice as high as the 2 highest volume hospitals (39 vs 20, per 1,000 admissions).

Figure 1

Pneumonia/respiratory infection was the most frequent admitting condition associated with unplanned transfer (16.3%) (Table 1). There was also wide variation in crude rates for unplanned ICU transfer by admitting condition (Figure 2). Patients admitted with sepsis had the highest rate (59 per 1,000 admissions), while patients admitted with renal conditions other than acute renal failure had the lowest rates (14.3 per 1,000 admissions).

Figure 2

We confirmed that almost all diagnoses found to account for a disproportionately high share of unplanned ICU transfers in Table 1 were indeed independently associated with this phenomenon after adjustment for patient and hospital differences (Figure 2). Pneumonia remained the most frequent condition associated with unplanned ICU transfer (odds ratio [OR] 1.50; 95% CI 1.201.86). Although less frequent, sepsis had the strongest association of any condition with unplanned transfer (OR 2.51; 95% CI 1.903.31). However, metabolic, endocrine, and electrolyte conditions were no longer associated with unplanned transfer after adjustment, while arrhythmias and pulmonary embolism were. Other conditions confirmed to be associated with increased risk of unplanned transfer included: myocardial infarction (MI), chronic obstructive pulmonary disease (COPD), stroke, diabetic emergencies, catastrophic conditions (includes aortic catastrophes, all forms of shock except septic shock, and intracranial hemorrhage), and acute renal failure. After taking into account the frequency of admitting diagnoses, respiratory conditions (COPD, pneumonia/acute respiratory infection) comprised nearly half (47%) of all conditions associated with increased risk of unplanned ICU transfer.

Other factors confirmed to be independently associated with unplanned ICU transfer included: male sex (OR 1.20; 95% CI 1.131.28), high comorbidity burden as measured by COPS >145 (OR 1.13; 95% CI 1.031.24), increasingly abnormal physiology compared to a LAPS <7, and arrival on ward during the overnight shift (OR 1.10; 95% CI 1.011.21). After adjustment, we did find that admission to the TCU rather than a medicalsurgical unit was associated with decreased risk of unplanned ICU transfer (OR 0.83; 95% CI 0.770.90). Age 85 was associated with decreased risk of unplanned ICU transfer relative to the youngest age group of 1834‐year‐old patients (OR 0.64; 95% CI 0.530.77).

ED admissions to higher volume hospitals were 6% less likely to experience an unplanned transfer for each additional 1,000 annual ED hospitalizations over a lower volume hospital (OR 0.94; 95% CI 0.910.98). In other words, a patient admitted to a hospital with 8,000 annual ED hospitalizations had 30% decreased odds of unplanned ICU transfer compared to a hospital with only 3,000 annual ED hospitalizations.

DISCUSSION

Patients admitted with respiratory conditions accounted for half of all admitting diagnoses associated with increased risk of unplanned transfer to the ICU within 24 hours of arrival to the ward. We found that 1 in 30 ED ward admissions for pneumonia, and 1 in 33 for COPD, were transferred to the ICU within 24 hours. These findings indicate that there is some room for improvement in early care of respiratory conditions, given the average unplanned transfer rate of 1 in 42, and previous research showing that patients with pneumonia and patients with COPD, who experience unplanned ICU transfer, have substantially worse mortality than those directly admitted to the ICU.1

Although less frequent than hospitalizations for respiratory conditions, patients admitted with sepsis were at the highest risk of unplanned ICU transfer (1 in 17 ED non‐ICU hospitalizations). We also found that MI and stroke ward admissions had a higher risk of unplanned ICU transfer. However, we previously found that unplanned ICU transfers for sepsis, MI, and stroke did not have worse mortality than direct ICU admits for these conditions.1 Therefore, quality improvement efforts to reduce excess mortality related to early decompensation in the hospital and unplanned ICU transfer would be most effective if targeted towards respiratory conditions such as pneumonia and COPD.

This is the only in‐depth study, to our knowledge, to explore the association between a set of mutually exclusive diagnostic categories and risk of unplanned ICU transfer within 24 hours, and it is the first study to identify risk factors for unplanned ICU transfer in a multi‐hospital cohort adjusted for patient‐ and hospital‐level characteristics. We also identified a novel hospital volumeoutcome relationship: Unplanned ICU transfers are up to twice as likely to occur in the smallest volume hospitals compared with highest volume hospitals. Hospital volume has long been proposed as a proxy for hospital resources; there are several studies showing a relationship between low‐volume hospitals and worse outcomes for a number of conditions.19, 20 Possible mechanisms may include decreased ICU capacity, decreased on‐call intensivists in the hospital after hours, and less experience with certain critical care conditions seen more frequently in high‐volume hospitals.21

Patients at risk of unplanned ICU transfer were also more likely to have physiologic derangement identified on laboratory testing, high comorbidity burden, and arrive on the ward between 11 PM and 7 AM. Given the strong correlation between comorbidity burden and physiologic derangement and mortality,14 it is not surprising that the COPS and LAPS were independent predictors of unplanned transfer. It is unclear, however, why arriving on the ward on the overnight shift is associated with higher risk. One possibility is that patients who arrive on the wards during 11 PM to 7 AM are also likely to have been in the ED during evening peak hours most associated with ED crowding.22 High levels of ED crowding have been associated with delays in care, worse quality care, lapses in patient safety, and even increased in‐hospital mortality.22, 23 Other possible reasons include decreased in‐hospital staffing and longer delays in critical diagnostic tests and interventions.2428

Admission to TCUs was associated with decreased risk of unplanned ICU transfer in the first 24 hours of hospitalization. This may be due to the continuous monitoring, decreased nursing‐to‐patient ratios, or the availability to provide some critical care interventions. In our study, age 85 was associated with lower likelihood of unplanned transfer. Unfortunately, we did not have access to data on advanced directives or patient preferences. Data on advanced directives would help to distinguish whether this phenomenon was related to end‐of‐life care goals versus other explanations.

Our study confirms some risk factors identified in previous studies. These include specific diagnoses such as pneumonia and COPD,12, 13, 29 heavy comorbidity burden,12, 13, 29 abnormal labs,29 and male sex.13 Pneumonia has consistently been shown to be a risk factor for unplanned ICU transfer. This may stem from the dynamic nature of this condition and its ability to rapidly progress, and the fact that some ICUs may not accept pneumonia patients unless they demonstrate a need for mechanical ventilation.30 Recently, a prediction rule has been developed to determine which patients with pneumonia are likely to have an unplanned ICU transfer.30 It is possible that with validation and application of this rule, unplanned transfer rates for pneumonia could be reduced. It is unclear whether males have unmeasured factors associated with increased risk of unplanned transfer or whether a true gender disparity exists.

Our findings should be interpreted within the context of this study's limitations. First, this study was not designed to distinguish the underlying cause of the unplanned transfer such as under‐recognition of illness severity in the ED, evolving clinical disease after leaving the ED, or delays in critical interventions on the ward. These are a focus of our ongoing research efforts. Second, while previous studies have demonstrated that our automated risk adjustment variables can accurately predict in‐hospital mortality (0.88 area under curve in external populations),17 additional data on vital signs and mental status could further improve risk adjustment. However, using automated data allowed us to study risk factors for unplanned transfer in a multi‐hospital cohort with a much larger population than has been previously studied. Serial data on vital signs and mental status both in the ED and during hospitalization could also be helpful in determining which unplanned transfers could be prevented with earlier recognition and intervention. Finally, all patient care occurred within an integrated healthcare delivery system. Thus, differences in case‐mix, hospital resources, ICU structure, and geographic location should be considered when applying our results to other healthcare systems.

This study raises several new areas for future research. With access to richer data becoming available in electronic medical records, prediction rules should be developed to enable better triage to appropriate levels of care for ED admissions. Future research should also analyze the comparative effectiveness of intermediate monitored units versus non‐monitored wards for preventing clinical deterioration by admitting diagnosis. Diagnoses that have been shown to have an increased risk of death after unplanned ICU transfer, such as pneumonia/respiratory infection and COPD,1 should be prioritized in this research. Better understanding is needed on the diagnosis‐specific differences and the differences in ED triage process and ICU structure that may explain why high‐volume hospitals have significantly lower rates of early unplanned ICU transfers compared with low‐volume hospitals. In particular, determining the effect of TCU and ICU capacities and census at the time of admission, and comparing patient risk characteristics across hospital‐volume strata would be very useful. Finally, more work is needed to determine whether the higher rate of unplanned transfers during overnight nursing shifts is related to decreased resource availability, preceding ED crowding, or other organizational causes.

In conclusion, patients admitted with respiratory conditions, sepsis, MI, high comorbidity, and abnormal labs are at modestly increased risk of unplanned ICU transfer within 24 hours of admission from the ED. Patients admitted with respiratory conditions (pneumonia/respiratory infections and COPD) accounted for half of the admitting diagnoses that are at increased risk for unplanned ICU transfer. These patients may benefit from better inpatient triage from the ED, earlier intervention, or closer monitoring. More research is needed to determine the specific aspects of care associated with admission to intermediate care units and high‐volume hospitals that reduce the risk of unplanned ICU transfer.

Emergency Department (ED) patients who are hospitalized and require unplanned transfer to the intensive care unit (ICU) within 24 hours of arrival on the ward have higher mortality than direct ICU admissions.1, 2 Previous research found that 5% of ED admissions experienced unplanned ICU transfer during their hospitalization, yet these patients account for 25% of in‐hospital deaths and have a longer length of stay than direct ICU admissions.1, 3 For these reasons, inpatient rapid‐response teams and early warning systems have been studied to reduce the mortality of patients who rapidly deteriorate on the hospital ward.410 However, there is little conclusive evidence that these interventions decrease mortality.710 It is possible that with better recognition and intervention in the ED, a portion of these unplanned ICU transfers and their subsequent adverse outcomes could be prevented.11

Previous research on risk factors for unplanned ICU transfers among ED admissions is limited. While 2 previous studies from non‐US hospitals used administrative data to identify some general populations at risk for unplanned ICU transfer,12, 13 these studies did not differentiate between transfers shortly after admission and those that occurred during a prolonged hospital staya critical distinction since the outcomes between these groups differs substantially.1 Another limitation of these studies is the absence of physiologic measures at ED presentation, which have been shown to be highly predictive of mortality.14

In this study, we describe risk factors for unplanned transfer to the ICU within 24 hours of arrival on the ward, among a large cohort of ED hospitalizations across 13 community hospitals. Focusing on admitting diagnoses most at risk, our goal was to inform efforts to improve the triage of ED admissions and determine which patients may benefit from additional interventions, such as improved resuscitation, closer monitoring, or risk stratification tools. We also hypothesized that higher volume hospitals would have lower rates of unplanned ICU transfers, as these hospitals are more likely have more patient care resources on the hospital ward and a higher threshold to transfer to the ICU.

METHODS

RESULTS

Of 178,315 ED non‐ICU hospitalizations meeting inclusion criteria, 4,252 (2.4%) were admitted to the ward and were transferred to the ICU within 24 hours of leaving the ED. There were 122,251 unique patients in our study population. Table 1 compares the characteristics of ED hospitalizations in which an unplanned transfer occurred to those that did not experience an unplanned transfer. Unplanned transfers were more likely to have a higher comorbidity burden, more deranged physiology, and more likely to arrive on the floor during the overnight shift.

Unplanned ICU transfers were more frequent in lower volume hospitals (Table 1). Figure 1 displays the inverse relationship between hospital annual ED inpatient admission volume and unplanned ICU transfers rates. The lowest volume hospital had a crude rate twice as high as the 2 highest volume hospitals (39 vs 20, per 1,000 admissions).

Figure 1

Pneumonia/respiratory infection was the most frequent admitting condition associated with unplanned transfer (16.3%) (Table 1). There was also wide variation in crude rates for unplanned ICU transfer by admitting condition (Figure 2). Patients admitted with sepsis had the highest rate (59 per 1,000 admissions), while patients admitted with renal conditions other than acute renal failure had the lowest rates (14.3 per 1,000 admissions).

Figure 2

We confirmed that almost all diagnoses found to account for a disproportionately high share of unplanned ICU transfers in Table 1 were indeed independently associated with this phenomenon after adjustment for patient and hospital differences (Figure 2). Pneumonia remained the most frequent condition associated with unplanned ICU transfer (odds ratio [OR] 1.50; 95% CI 1.201.86). Although less frequent, sepsis had the strongest association of any condition with unplanned transfer (OR 2.51; 95% CI 1.903.31). However, metabolic, endocrine, and electrolyte conditions were no longer associated with unplanned transfer after adjustment, while arrhythmias and pulmonary embolism were. Other conditions confirmed to be associated with increased risk of unplanned transfer included: myocardial infarction (MI), chronic obstructive pulmonary disease (COPD), stroke, diabetic emergencies, catastrophic conditions (includes aortic catastrophes, all forms of shock except septic shock, and intracranial hemorrhage), and acute renal failure. After taking into account the frequency of admitting diagnoses, respiratory conditions (COPD, pneumonia/acute respiratory infection) comprised nearly half (47%) of all conditions associated with increased risk of unplanned ICU transfer.

Other factors confirmed to be independently associated with unplanned ICU transfer included: male sex (OR 1.20; 95% CI 1.131.28), high comorbidity burden as measured by COPS >145 (OR 1.13; 95% CI 1.031.24), increasingly abnormal physiology compared to a LAPS <7, and arrival on ward during the overnight shift (OR 1.10; 95% CI 1.011.21). After adjustment, we did find that admission to the TCU rather than a medicalsurgical unit was associated with decreased risk of unplanned ICU transfer (OR 0.83; 95% CI 0.770.90). Age 85 was associated with decreased risk of unplanned ICU transfer relative to the youngest age group of 1834‐year‐old patients (OR 0.64; 95% CI 0.530.77).

ED admissions to higher volume hospitals were 6% less likely to experience an unplanned transfer for each additional 1,000 annual ED hospitalizations over a lower volume hospital (OR 0.94; 95% CI 0.910.98). In other words, a patient admitted to a hospital with 8,000 annual ED hospitalizations had 30% decreased odds of unplanned ICU transfer compared to a hospital with only 3,000 annual ED hospitalizations.

DISCUSSION

Patients admitted with respiratory conditions accounted for half of all admitting diagnoses associated with increased risk of unplanned transfer to the ICU within 24 hours of arrival to the ward. We found that 1 in 30 ED ward admissions for pneumonia, and 1 in 33 for COPD, were transferred to the ICU within 24 hours. These findings indicate that there is some room for improvement in early care of respiratory conditions, given the average unplanned transfer rate of 1 in 42, and previous research showing that patients with pneumonia and patients with COPD, who experience unplanned ICU transfer, have substantially worse mortality than those directly admitted to the ICU.1

Although less frequent than hospitalizations for respiratory conditions, patients admitted with sepsis were at the highest risk of unplanned ICU transfer (1 in 17 ED non‐ICU hospitalizations). We also found that MI and stroke ward admissions had a higher risk of unplanned ICU transfer. However, we previously found that unplanned ICU transfers for sepsis, MI, and stroke did not have worse mortality than direct ICU admits for these conditions.1 Therefore, quality improvement efforts to reduce excess mortality related to early decompensation in the hospital and unplanned ICU transfer would be most effective if targeted towards respiratory conditions such as pneumonia and COPD.

This is the only in‐depth study, to our knowledge, to explore the association between a set of mutually exclusive diagnostic categories and risk of unplanned ICU transfer within 24 hours, and it is the first study to identify risk factors for unplanned ICU transfer in a multi‐hospital cohort adjusted for patient‐ and hospital‐level characteristics. We also identified a novel hospital volumeoutcome relationship: Unplanned ICU transfers are up to twice as likely to occur in the smallest volume hospitals compared with highest volume hospitals. Hospital volume has long been proposed as a proxy for hospital resources; there are several studies showing a relationship between low‐volume hospitals and worse outcomes for a number of conditions.19, 20 Possible mechanisms may include decreased ICU capacity, decreased on‐call intensivists in the hospital after hours, and less experience with certain critical care conditions seen more frequently in high‐volume hospitals.21

Patients at risk of unplanned ICU transfer were also more likely to have physiologic derangement identified on laboratory testing, high comorbidity burden, and arrive on the ward between 11 PM and 7 AM. Given the strong correlation between comorbidity burden and physiologic derangement and mortality,14 it is not surprising that the COPS and LAPS were independent predictors of unplanned transfer. It is unclear, however, why arriving on the ward on the overnight shift is associated with higher risk. One possibility is that patients who arrive on the wards during 11 PM to 7 AM are also likely to have been in the ED during evening peak hours most associated with ED crowding.22 High levels of ED crowding have been associated with delays in care, worse quality care, lapses in patient safety, and even increased in‐hospital mortality.22, 23 Other possible reasons include decreased in‐hospital staffing and longer delays in critical diagnostic tests and interventions.2428

Admission to TCUs was associated with decreased risk of unplanned ICU transfer in the first 24 hours of hospitalization. This may be due to the continuous monitoring, decreased nursing‐to‐patient ratios, or the availability to provide some critical care interventions. In our study, age 85 was associated with lower likelihood of unplanned transfer. Unfortunately, we did not have access to data on advanced directives or patient preferences. Data on advanced directives would help to distinguish whether this phenomenon was related to end‐of‐life care goals versus other explanations.

Our study confirms some risk factors identified in previous studies. These include specific diagnoses such as pneumonia and COPD,12, 13, 29 heavy comorbidity burden,12, 13, 29 abnormal labs,29 and male sex.13 Pneumonia has consistently been shown to be a risk factor for unplanned ICU transfer. This may stem from the dynamic nature of this condition and its ability to rapidly progress, and the fact that some ICUs may not accept pneumonia patients unless they demonstrate a need for mechanical ventilation.30 Recently, a prediction rule has been developed to determine which patients with pneumonia are likely to have an unplanned ICU transfer.30 It is possible that with validation and application of this rule, unplanned transfer rates for pneumonia could be reduced. It is unclear whether males have unmeasured factors associated with increased risk of unplanned transfer or whether a true gender disparity exists.

Our findings should be interpreted within the context of this study's limitations. First, this study was not designed to distinguish the underlying cause of the unplanned transfer such as under‐recognition of illness severity in the ED, evolving clinical disease after leaving the ED, or delays in critical interventions on the ward. These are a focus of our ongoing research efforts. Second, while previous studies have demonstrated that our automated risk adjustment variables can accurately predict in‐hospital mortality (0.88 area under curve in external populations),17 additional data on vital signs and mental status could further improve risk adjustment. However, using automated data allowed us to study risk factors for unplanned transfer in a multi‐hospital cohort with a much larger population than has been previously studied. Serial data on vital signs and mental status both in the ED and during hospitalization could also be helpful in determining which unplanned transfers could be prevented with earlier recognition and intervention. Finally, all patient care occurred within an integrated healthcare delivery system. Thus, differences in case‐mix, hospital resources, ICU structure, and geographic location should be considered when applying our results to other healthcare systems.

This study raises several new areas for future research. With access to richer data becoming available in electronic medical records, prediction rules should be developed to enable better triage to appropriate levels of care for ED admissions. Future research should also analyze the comparative effectiveness of intermediate monitored units versus non‐monitored wards for preventing clinical deterioration by admitting diagnosis. Diagnoses that have been shown to have an increased risk of death after unplanned ICU transfer, such as pneumonia/respiratory infection and COPD,1 should be prioritized in this research. Better understanding is needed on the diagnosis‐specific differences and the differences in ED triage process and ICU structure that may explain why high‐volume hospitals have significantly lower rates of early unplanned ICU transfers compared with low‐volume hospitals. In particular, determining the effect of TCU and ICU capacities and census at the time of admission, and comparing patient risk characteristics across hospital‐volume strata would be very useful. Finally, more work is needed to determine whether the higher rate of unplanned transfers during overnight nursing shifts is related to decreased resource availability, preceding ED crowding, or other organizational causes.

In conclusion, patients admitted with respiratory conditions, sepsis, MI, high comorbidity, and abnormal labs are at modestly increased risk of unplanned ICU transfer within 24 hours of admission from the ED. Patients admitted with respiratory conditions (pneumonia/respiratory infections and COPD) accounted for half of the admitting diagnoses that are at increased risk for unplanned ICU transfer. These patients may benefit from better inpatient triage from the ED, earlier intervention, or closer monitoring. More research is needed to determine the specific aspects of care associated with admission to intermediate care units and high‐volume hospitals that reduce the risk of unplanned ICU transfer.

Emergency Department (ED) patients who are hospitalized and require unplanned transfer to the intensive care unit (ICU) within 24 hours of arrival on the ward have higher mortality than direct ICU admissions.1, 2 Previous research found that 5% of ED admissions experienced unplanned ICU transfer during their hospitalization, yet these patients account for 25% of in‐hospital deaths and have a longer length of stay than direct ICU admissions.1, 3 For these reasons, inpatient rapid‐response teams and early warning systems have been studied to reduce the mortality of patients who rapidly deteriorate on the hospital ward.410 However, there is little conclusive evidence that these interventions decrease mortality.710 It is possible that with better recognition and intervention in the ED, a portion of these unplanned ICU transfers and their subsequent adverse outcomes could be prevented.11

Previous research on risk factors for unplanned ICU transfers among ED admissions is limited. While 2 previous studies from non‐US hospitals used administrative data to identify some general populations at risk for unplanned ICU transfer,12, 13 these studies did not differentiate between transfers shortly after admission and those that occurred during a prolonged hospital staya critical distinction since the outcomes between these groups differs substantially.1 Another limitation of these studies is the absence of physiologic measures at ED presentation, which have been shown to be highly predictive of mortality.14

In this study, we describe risk factors for unplanned transfer to the ICU within 24 hours of arrival on the ward, among a large cohort of ED hospitalizations across 13 community hospitals. Focusing on admitting diagnoses most at risk, our goal was to inform efforts to improve the triage of ED admissions and determine which patients may benefit from additional interventions, such as improved resuscitation, closer monitoring, or risk stratification tools. We also hypothesized that higher volume hospitals would have lower rates of unplanned ICU transfers, as these hospitals are more likely have more patient care resources on the hospital ward and a higher threshold to transfer to the ICU.

METHODS

RESULTS

Of 178,315 ED non‐ICU hospitalizations meeting inclusion criteria, 4,252 (2.4%) were admitted to the ward and were transferred to the ICU within 24 hours of leaving the ED. There were 122,251 unique patients in our study population. Table 1 compares the characteristics of ED hospitalizations in which an unplanned transfer occurred to those that did not experience an unplanned transfer. Unplanned transfers were more likely to have a higher comorbidity burden, more deranged physiology, and more likely to arrive on the floor during the overnight shift.

Unplanned ICU transfers were more frequent in lower volume hospitals (Table 1). Figure 1 displays the inverse relationship between hospital annual ED inpatient admission volume and unplanned ICU transfers rates. The lowest volume hospital had a crude rate twice as high as the 2 highest volume hospitals (39 vs 20, per 1,000 admissions).

Figure 1

Pneumonia/respiratory infection was the most frequent admitting condition associated with unplanned transfer (16.3%) (Table 1). There was also wide variation in crude rates for unplanned ICU transfer by admitting condition (Figure 2). Patients admitted with sepsis had the highest rate (59 per 1,000 admissions), while patients admitted with renal conditions other than acute renal failure had the lowest rates (14.3 per 1,000 admissions).

Figure 2

We confirmed that almost all diagnoses found to account for a disproportionately high share of unplanned ICU transfers in Table 1 were indeed independently associated with this phenomenon after adjustment for patient and hospital differences (Figure 2). Pneumonia remained the most frequent condition associated with unplanned ICU transfer (odds ratio [OR] 1.50; 95% CI 1.201.86). Although less frequent, sepsis had the strongest association of any condition with unplanned transfer (OR 2.51; 95% CI 1.903.31). However, metabolic, endocrine, and electrolyte conditions were no longer associated with unplanned transfer after adjustment, while arrhythmias and pulmonary embolism were. Other conditions confirmed to be associated with increased risk of unplanned transfer included: myocardial infarction (MI), chronic obstructive pulmonary disease (COPD), stroke, diabetic emergencies, catastrophic conditions (includes aortic catastrophes, all forms of shock except septic shock, and intracranial hemorrhage), and acute renal failure. After taking into account the frequency of admitting diagnoses, respiratory conditions (COPD, pneumonia/acute respiratory infection) comprised nearly half (47%) of all conditions associated with increased risk of unplanned ICU transfer.

Other factors confirmed to be independently associated with unplanned ICU transfer included: male sex (OR 1.20; 95% CI 1.131.28), high comorbidity burden as measured by COPS >145 (OR 1.13; 95% CI 1.031.24), increasingly abnormal physiology compared to a LAPS <7, and arrival on ward during the overnight shift (OR 1.10; 95% CI 1.011.21). After adjustment, we did find that admission to the TCU rather than a medicalsurgical unit was associated with decreased risk of unplanned ICU transfer (OR 0.83; 95% CI 0.770.90). Age 85 was associated with decreased risk of unplanned ICU transfer relative to the youngest age group of 1834‐year‐old patients (OR 0.64; 95% CI 0.530.77).

ED admissions to higher volume hospitals were 6% less likely to experience an unplanned transfer for each additional 1,000 annual ED hospitalizations over a lower volume hospital (OR 0.94; 95% CI 0.910.98). In other words, a patient admitted to a hospital with 8,000 annual ED hospitalizations had 30% decreased odds of unplanned ICU transfer compared to a hospital with only 3,000 annual ED hospitalizations.

DISCUSSION

Patients admitted with respiratory conditions accounted for half of all admitting diagnoses associated with increased risk of unplanned transfer to the ICU within 24 hours of arrival to the ward. We found that 1 in 30 ED ward admissions for pneumonia, and 1 in 33 for COPD, were transferred to the ICU within 24 hours. These findings indicate that there is some room for improvement in early care of respiratory conditions, given the average unplanned transfer rate of 1 in 42, and previous research showing that patients with pneumonia and patients with COPD, who experience unplanned ICU transfer, have substantially worse mortality than those directly admitted to the ICU.1

Although less frequent than hospitalizations for respiratory conditions, patients admitted with sepsis were at the highest risk of unplanned ICU transfer (1 in 17 ED non‐ICU hospitalizations). We also found that MI and stroke ward admissions had a higher risk of unplanned ICU transfer. However, we previously found that unplanned ICU transfers for sepsis, MI, and stroke did not have worse mortality than direct ICU admits for these conditions.1 Therefore, quality improvement efforts to reduce excess mortality related to early decompensation in the hospital and unplanned ICU transfer would be most effective if targeted towards respiratory conditions such as pneumonia and COPD.

This is the only in‐depth study, to our knowledge, to explore the association between a set of mutually exclusive diagnostic categories and risk of unplanned ICU transfer within 24 hours, and it is the first study to identify risk factors for unplanned ICU transfer in a multi‐hospital cohort adjusted for patient‐ and hospital‐level characteristics. We also identified a novel hospital volumeoutcome relationship: Unplanned ICU transfers are up to twice as likely to occur in the smallest volume hospitals compared with highest volume hospitals. Hospital volume has long been proposed as a proxy for hospital resources; there are several studies showing a relationship between low‐volume hospitals and worse outcomes for a number of conditions.19, 20 Possible mechanisms may include decreased ICU capacity, decreased on‐call intensivists in the hospital after hours, and less experience with certain critical care conditions seen more frequently in high‐volume hospitals.21

Patients at risk of unplanned ICU transfer were also more likely to have physiologic derangement identified on laboratory testing, high comorbidity burden, and arrive on the ward between 11 PM and 7 AM. Given the strong correlation between comorbidity burden and physiologic derangement and mortality,14 it is not surprising that the COPS and LAPS were independent predictors of unplanned transfer. It is unclear, however, why arriving on the ward on the overnight shift is associated with higher risk. One possibility is that patients who arrive on the wards during 11 PM to 7 AM are also likely to have been in the ED during evening peak hours most associated with ED crowding.22 High levels of ED crowding have been associated with delays in care, worse quality care, lapses in patient safety, and even increased in‐hospital mortality.22, 23 Other possible reasons include decreased in‐hospital staffing and longer delays in critical diagnostic tests and interventions.2428

Admission to TCUs was associated with decreased risk of unplanned ICU transfer in the first 24 hours of hospitalization. This may be due to the continuous monitoring, decreased nursing‐to‐patient ratios, or the availability to provide some critical care interventions. In our study, age 85 was associated with lower likelihood of unplanned transfer. Unfortunately, we did not have access to data on advanced directives or patient preferences. Data on advanced directives would help to distinguish whether this phenomenon was related to end‐of‐life care goals versus other explanations.

Our study confirms some risk factors identified in previous studies. These include specific diagnoses such as pneumonia and COPD,12, 13, 29 heavy comorbidity burden,12, 13, 29 abnormal labs,29 and male sex.13 Pneumonia has consistently been shown to be a risk factor for unplanned ICU transfer. This may stem from the dynamic nature of this condition and its ability to rapidly progress, and the fact that some ICUs may not accept pneumonia patients unless they demonstrate a need for mechanical ventilation.30 Recently, a prediction rule has been developed to determine which patients with pneumonia are likely to have an unplanned ICU transfer.30 It is possible that with validation and application of this rule, unplanned transfer rates for pneumonia could be reduced. It is unclear whether males have unmeasured factors associated with increased risk of unplanned transfer or whether a true gender disparity exists.

Our findings should be interpreted within the context of this study's limitations. First, this study was not designed to distinguish the underlying cause of the unplanned transfer such as under‐recognition of illness severity in the ED, evolving clinical disease after leaving the ED, or delays in critical interventions on the ward. These are a focus of our ongoing research efforts. Second, while previous studies have demonstrated that our automated risk adjustment variables can accurately predict in‐hospital mortality (0.88 area under curve in external populations),17 additional data on vital signs and mental status could further improve risk adjustment. However, using automated data allowed us to study risk factors for unplanned transfer in a multi‐hospital cohort with a much larger population than has been previously studied. Serial data on vital signs and mental status both in the ED and during hospitalization could also be helpful in determining which unplanned transfers could be prevented with earlier recognition and intervention. Finally, all patient care occurred within an integrated healthcare delivery system. Thus, differences in case‐mix, hospital resources, ICU structure, and geographic location should be considered when applying our results to other healthcare systems.

This study raises several new areas for future research. With access to richer data becoming available in electronic medical records, prediction rules should be developed to enable better triage to appropriate levels of care for ED admissions. Future research should also analyze the comparative effectiveness of intermediate monitored units versus non‐monitored wards for preventing clinical deterioration by admitting diagnosis. Diagnoses that have been shown to have an increased risk of death after unplanned ICU transfer, such as pneumonia/respiratory infection and COPD,1 should be prioritized in this research. Better understanding is needed on the diagnosis‐specific differences and the differences in ED triage process and ICU structure that may explain why high‐volume hospitals have significantly lower rates of early unplanned ICU transfers compared with low‐volume hospitals. In particular, determining the effect of TCU and ICU capacities and census at the time of admission, and comparing patient risk characteristics across hospital‐volume strata would be very useful. Finally, more work is needed to determine whether the higher rate of unplanned transfers during overnight nursing shifts is related to decreased resource availability, preceding ED crowding, or other organizational causes.

In conclusion, patients admitted with respiratory conditions, sepsis, MI, high comorbidity, and abnormal labs are at modestly increased risk of unplanned ICU transfer within 24 hours of admission from the ED. Patients admitted with respiratory conditions (pneumonia/respiratory infections and COPD) accounted for half of the admitting diagnoses that are at increased risk for unplanned ICU transfer. These patients may benefit from better inpatient triage from the ED, earlier intervention, or closer monitoring. More research is needed to determine the specific aspects of care associated with admission to intermediate care units and high‐volume hospitals that reduce the risk of unplanned ICU transfer.

If you wish to receive credit for this activity, which begins on the next page, please refer to the website: www.blackwellpublishing.com/cme.

Accreditation and Designation Statement

Blackwell Futura Media Services designates this educational activity for a 1 AMA PRA Category 1 Credit. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Blackwell Futura Media Services is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Educational Objectives

Upon completion of this educational activity, participants will be better able to employ automated bed history data to examine outcomes of intra‐hospital transfers using all hospital admissions as the denominator.

Continuous participation in the Journal of Hospital Medicine CME program will enable learners to be better able to:

• Interpret clinical guidelines and their applications for higher quality and more efficient care for all hospitalized patients.

• Describe the standard of care for common illnesses and conditions treated in the hospital; such as pneumonia, COPD exacerbation, acute coronary syndrome, HF exacerbation, glycemic control, venous thromboembolic disease, stroke, etc.

• Discuss evidence‐based recommendations involving transitions of care, including the hospital discharge process.

• Gain insights into the roles of hospitalists as medical educators, researchers, medical ethicists, palliative care providers, and hospital‐based geriatricians.

• Incorporate best practices for hospitalist administration, including quality improvement, patient safety, practice management, leadership, and demonstrating hospitalist value.

• Identify evidence‐based best practices and trends for both adult and pediatric hospital medicine.

Instructions on Receiving Credit

For information on applicability and acceptance of continuing medical education credit for this activity, please consult your professional licensing board.

This activity is designed to be completed within the time designated on the title page; physicians should claim only those credits that reflect the time actually spent in the activity. To successfully earn credit, participants must complete the activity during the valid credit period that is noted on the title page.

Follow these steps to earn credit:

• Log on to www.blackwellpublishing.com/cme.

• Read the target audience, learning objectives, and author disclosures.

• Read the article in print or online format.

• Reflect on the article.

• Access the CME Exam, and choose the best answer to each question.

• Complete the required evaluation component of the activity.

If you wish to receive credit for this activity, which begins on the next page, please refer to the website: www.blackwellpublishing.com/cme.

Accreditation and Designation Statement

Blackwell Futura Media Services designates this educational activity for a 1 AMA PRA Category 1 Credit. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Blackwell Futura Media Services is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Educational Objectives

Upon completion of this educational activity, participants will be better able to employ automated bed history data to examine outcomes of intra‐hospital transfers using all hospital admissions as the denominator.

Continuous participation in the Journal of Hospital Medicine CME program will enable learners to be better able to:

• Interpret clinical guidelines and their applications for higher quality and more efficient care for all hospitalized patients.

• Describe the standard of care for common illnesses and conditions treated in the hospital; such as pneumonia, COPD exacerbation, acute coronary syndrome, HF exacerbation, glycemic control, venous thromboembolic disease, stroke, etc.

• Discuss evidence‐based recommendations involving transitions of care, including the hospital discharge process.

• Gain insights into the roles of hospitalists as medical educators, researchers, medical ethicists, palliative care providers, and hospital‐based geriatricians.

• Incorporate best practices for hospitalist administration, including quality improvement, patient safety, practice management, leadership, and demonstrating hospitalist value.

• Identify evidence‐based best practices and trends for both adult and pediatric hospital medicine.

Instructions on Receiving Credit

For information on applicability and acceptance of continuing medical education credit for this activity, please consult your professional licensing board.

This activity is designed to be completed within the time designated on the title page; physicians should claim only those credits that reflect the time actually spent in the activity. To successfully earn credit, participants must complete the activity during the valid credit period that is noted on the title page.

Follow these steps to earn credit:

• Log on to www.blackwellpublishing.com/cme.

• Read the target audience, learning objectives, and author disclosures.

• Read the article in print or online format.

• Reflect on the article.

• Access the CME Exam, and choose the best answer to each question.

• Complete the required evaluation component of the activity.

If you wish to receive credit for this activity, which begins on the next page, please refer to the website: www.blackwellpublishing.com/cme.

Accreditation and Designation Statement

Blackwell Futura Media Services designates this educational activity for a 1 AMA PRA Category 1 Credit. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Blackwell Futura Media Services is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Educational Objectives

Upon completion of this educational activity, participants will be better able to employ automated bed history data to examine outcomes of intra‐hospital transfers using all hospital admissions as the denominator.

Continuous participation in the Journal of Hospital Medicine CME program will enable learners to be better able to:

• Interpret clinical guidelines and their applications for higher quality and more efficient care for all hospitalized patients.

• Describe the standard of care for common illnesses and conditions treated in the hospital; such as pneumonia, COPD exacerbation, acute coronary syndrome, HF exacerbation, glycemic control, venous thromboembolic disease, stroke, etc.

• Discuss evidence‐based recommendations involving transitions of care, including the hospital discharge process.

• Gain insights into the roles of hospitalists as medical educators, researchers, medical ethicists, palliative care providers, and hospital‐based geriatricians.

• Incorporate best practices for hospitalist administration, including quality improvement, patient safety, practice management, leadership, and demonstrating hospitalist value.

• Identify evidence‐based best practices and trends for both adult and pediatric hospital medicine.

Instructions on Receiving Credit

For information on applicability and acceptance of continuing medical education credit for this activity, please consult your professional licensing board.

This activity is designed to be completed within the time designated on the title page; physicians should claim only those credits that reflect the time actually spent in the activity. To successfully earn credit, participants must complete the activity during the valid credit period that is noted on the title page.

Follow these steps to earn credit:

• Log on to www.blackwellpublishing.com/cme.

• Read the target audience, learning objectives, and author disclosures.

• Read the article in print or online format.

• Reflect on the article.

• Access the CME Exam, and choose the best answer to each question.

• Complete the required evaluation component of the activity.

Considerable research and public attention is being paid to the quantification, risk adjustment, and reporting of inpatient mortality.15 Inpatient mortality is reported as aggregate mortality (for all hospitalized patients or those with a specific diagnosis3, 6) or intensive care unit (ICU) mortality.7, 8 While reporting aggregate hospital or aggregate ICU mortality rates is useful, it is also important to develop reporting strategies that go beyond simply using data elements found in administrative databases (eg, diagnosis and procedure codes) to quantify practice variation. Ideally, such strategies would permit delineating processes of careparticularly those potentially under the control of hospitalists, not only intensiviststo identify improvement opportunities. One such process, which can be tracked using the bed history component of a patient's electronic medical record, is the transfer of patients between different units within the same hospital.

Several studies have documented that risk of ICU death is highest among patients transferred from general medical‐surgical wards, intermediate among direct admissions from the emergency department, and lowest among surgical admissions.911 Opportunities to reduce subsequent ICU mortality have been studied among ward patients who develop sepsis and are then transferred to the ICU,12 among patients who experience cardiac arrest,13, 14 as well as among patients with any physiological deterioration (eg, through the use of rapid response teams).1517 Most of these studies have been single‐center studies and/or studies reporting only an ICU denominator. While useful in some respects, such studies are less helpful to hospitalists, who would benefit from better understanding of the types of patients transferred and the total impact that transfers to a higher level of care make on general medical‐surgical wards. In addition, entities such as the Institute for Healthcare Improvement recommend the manual review of records of patients who were transferred from the ward to the ICU18 to identify performance improvement opportunities. While laudable, such approaches do not lend themselves to automated reporting strategies.

We recently described a new risk adjustment methodology for inpatient mortality based entirely on automated data preceding hospital admission and not restricted to ICU patients. This methodology, which has been externally validated in Ottawa, Canada, after development in the Kaiser Permanente Medical Care Program (KPMCP), permits quantification of a patient's pre‐existing comorbidity burden, physiologic derangement at the time of admission, and overall inpatient mortality risk.19, 20 The primary purpose of this study was to combine this methodology with bed history analysis to quantify the in‐hospital mortality and length of stay (LOS) of patients who experienced intra‐hospital transfers in a large, multihospital system. As a secondary goal, we also wanted to assess the degree to which these transfers could be predicted based on information available prior to a patient's admission.

Materials and Methods

This project was approved by the Northern California KPMCP Institutional Review Board for the Protection of Human Subjects.

The Northern California KPMCP serves a total population of approximately 3.3 million members. Under a mutual exclusivity arrangement, physicians of The Permanente Medical Group, Inc., care for Kaiser Foundation Health Plan, Inc. members at facilities owned by Kaiser Foundation Hospitals, Inc. All Northern California KPMCP hospitals and clinics employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere. Databases maintained by the KPMCP capture admission and discharge times, admission and discharge diagnoses and procedures (assigned by professional coders), bed histories, inter‐hospital transfers, as well as the results of all inpatient and outpatient laboratory tests. The use of these databases for research has been described in multiple reports.2124

Our setting consisted of all 19 hospitals owned and operated by the KPMCP, whose characteristics are summarized in the Supporting Information Appendix available to interested readers. These include the 17 described in our previous report19 as well as 2 new hospitals (Antioch and Manteca) which are similar in size and type of population served. Our study population consisted of all patients admitted to these 19 hospitals who met these criteria: 1) hospitalization began from November 1st, 2006 through January 31st, 2008; 2) initial hospitalization occurred at a Northern California KPMCP hospital (ie, for inter‐hospital transfers, the first hospital stay occurred within the KPMCP); 3) age 15 years; and 4) hospitalization was not for childbirth.

We defined a linked hospitalization as the time period that began with a patient's admission to the hospital and ended with the patient's discharge (home, to a nursing home, or death). Linked hospitalizations can thus involve more than 1 hospital stay and could include a patient transfer from one hospital to another prior to definitive discharge. For linked hospitalizations, mortality was attributed to the admitting KPMCP hospital (ie, if a patient was admitted to hospital A, transferred to B, and died at hospital B, mortality was attributed to hospital A). We defined total LOS as the exact time in hours from when a patient was first admitted to the hospital until death or final discharge home or to a nursing home, while total ICU or transitional care unit (TCU, referred to as stepdown unit in some hospitals) LOS was calculated for all individual ICU or TCU stays during the hospital stay.

Results

During the study period, a total of 249,129 individual hospital stays involving 170,151 patients occurred at these 19 hospitals. After concatenation of inter‐hospital transfers, we were left with 237,208 linked hospitalizations. We excluded 26,738 linked hospitalizations that began at a non‐KPMCP hospital (ie, they were transported in), leaving a total of 210,470 linked hospitalizations involving 150,495 patients. The overall linked hospitalization mortality rate was 3.30%.

Table 1 summarizes cohort characteristics based on initial hospital location. On admission, ICU patients had the highest degree of physiologic derangement as well as the highest predicted mortality. Considerable inter‐hospital variation was present in both predictors and outcomes; details on these variations are provided in the Supporting Information Appendix.

Table 2 summarizes data from 3 groups of patients: patients initially admitted to the ward, or TCU, who did not experience a transfer to a higher level of care and patients admitted to these 2 units who did experience such a transfer. Patients who experienced a transfer constituted 5.3% (6,484/121,237) of ward patients and 6.7% (1,384/20,556) of TCU patients. Transferred patients tended to be older, have more acute physiologic derangement (higher LAPS), a greater pre‐existing illness burden (higher COPS), and a higher predicted mortality risk. Among ward patients, those with the following admission diagnoses were most likely to experience a transfer to a higher level of care: gastrointestinal bleeding (10.8% of all transfers), pneumonia (8.7%), and other infections (8.2%). The diagnoses most likely to be associated with death following transfer were cancer (death rate among transferred patients, 48%), renal disease (death rate, 36%), and liver disease (33%). Similar distributions were observed for TCU patients.

Table 3 compares outcomes among ward and TCU patients who did and did not experience a transfer to a higher level of care. The table shows that transferred patients were almost 3 times as likely to die, even after controlling for severity of illness, and that their hospital LOS was 9 days higher than expected. This increased risk was seen in all hospitals and among all transfer types (ward to ICU, ward to TCU, and TCU to ICU).

Table 3 also shows that, among decedent patients, those who never left the ward or TCU died much sooner than those who died following transfer. Among direct admits to the ICU, the median LOS at time of death was 3.9 days, with a mean of 9.4 standard deviation of 19.9 days, while the corresponding times for TCU direct admits were a median and mean LOS of 6.5 and 11.7 19.5 days.

Table 4 summarizes outcomes among different patient subgroups that did and did not experience a transfer to a higher level of care. Based on location, patients who experienced a transfer from the TCU to the ICU had the highest crude death rate, but patients transferred from the ward to the ICU had the highest OEMR. On the other hand, if one divides patients by the degree of physiologic derangement, patients with low LAPS who experienced a transfer had the highest OEMR. With respect to LOS, patients transferred from the TCU to the ICU had the highest OMELOS (13.4 extra days).

Transfers to a higher level of care at a different hospital, which in the KPMCP are usually planned, experienced lower mortality than transfers within the same hospital. For ward to TCU transfers, intra‐hospital transfers had a mortality of 12.1% while inter‐hospital transfers had a mortality of 5.7%. Corresponding rates for ward to ICU transfers were 21.7% and 11.2%, and for TCU to ICU transfers the rates were 25.9% and 12.5%, respectively.

Among patients initially admitted to the ward, a model to predict the occurrence of a transfer to a higher level of care (within 48 hours after admission) that included age, sex, admission type, primary condition, LAPS, COPS, and interaction terms had poor discrimination, with an area under the receiver operator characteristic (c statistic) of only 0.64. The c statistic for a model to predict transfer after 48 hours was 0.66. The corresponding models for TCU admits had c statistics of 0.67 and 0.68. All four models had poor calibration.

Discussion

Using automated bed history data permits characterizing a patient population with disproportionate mortality and LOS: intra‐hospital transfers to special care units (ICUs or TCUs). Indeed, the largest subset of these patients (those initially admitted to the ward or TCU) constituted only 3.7% of all admissions, but accounted for 24.2% of all ICU admissions, 21.7% of all hospital deaths, and 13.2% of all hospital days. These patients also had very elevated OEMRs and OMELOS. Models based on age, sex, preadmission laboratory test results, and comorbidities did not predict the occurrence of these transfers.

We performed multivariate analyses to explore the degree to which electronically assigned preadmission severity scores could predict these transfers. These analyses found that, compared to our ability to predict inpatient or 30‐day mortality at the time of admission, which is excellent, our ability to predict the occurrence of transfer after admission is much more limited. These results highlight the limitations of severity scores that rely on automated data, which may not have adequate discrimination when it comes to determining the risk of an adverse outcome within a narrow time frame. For example, among the 121,237 patients initially admitted to the ward who did not experience an intra‐hospital transfer, the mean LAPS was 18.9, while the mean LAPS among the 6,484 ward patients who did experience a transfer was 25.5. Differences between the mean and median LAPS, COPS, and predicted mortality risk among transferred and non‐transferred patients were significant (P < 0.0001 for all comparisons). However, examination of the distribution of LAPS, COPS, and predicted mortality risk between these two groups of patients showed considerable overlap.

Our methodology resembles Silber et al.'s27, 28 concept of failure to rescue in that it focuses on events occurring after hospitalization. Silber et al. argue that a hospital's quality can be measured by quantifying the degree to which patients who experience new problems are successfully rescued. Furthermore, quantification of those situations where rescue attempts are unsuccessful is felt to be superior to simply comparing raw or adjusted mortality rates because these are primarily determined by underlying case mix. The primary difference between Silber et al.'s approach and ours is at the level of detailthey specified a specific set of complications, whereas our measure is more generic and would include patients with many of the complications specified by Silber et al.27, 28

Most of the patients transferred to a higher level of care in our cohort survived (ie, were rescued), indicating that intensive care is beneficial. However, the fact that these patients had elevated OEMRs and OMELOS indicates that the real challenge facing hospitalists involves the timing of provision of a beneficial intervention. In theory, improved timing could result from earlier detection of problems, which is the underlying rationale for employing rapid response teams. However, the fact that our electronic tools (LAPS, COPS) cannot predict patient deteriorations within a narrow time frame suggests that early detection will remain a major challenge. Manually assigned vital signs scores designed for this purpose do not have good discrimination either.29, 30 This raises the possibility that, though patient groups may differ in terms of overall illness severity and mortality risk, differences at the individual patient level may be too subtle for clinicians to detect. Future research may thus need to focus on scores that combine laboratory data, vital signs, trends in data,31, 32 and newer proteomic markers (eg, procalcitonin).33 We also found that most transfers occurred early (within <72 hours), raising the possibility that at least some of these transfers may involve issues around triage rather than sudden deterioration.

Our study has important limitations. Due to resource constraints and limited data availability, we could not characterize the patients as well as might be desirable; in particular, we could not make full determinations of the actual reasons for patients' transfer for all patients. Broadly speaking, transfer to a higher level of care could be due to inappropriate triage, appropriate (preventive) transfer (which could include transfer to a more richly staffed unit for a specific procedure), relentless progression of disease despite maximal therapy, the occurrence of management errors, patient and family uncertainty about goals of care or inadequate understanding of treatment options and prognoses, or a combination of these factors. We could not make these distinctions with currently available electronic data. This is also true of postsurgical patients, in whom it is difficult to determine which transfers to intensive care might be planned (eg, in the case of surgical procedures where ICU care is anticipated) as opposed to the occurrence of a deterioration during or following surgery. Another major limitation of this study is our inability to identify code or no code status electronically. The elapsed LOS at time of death among patients who experienced a transfer to a higher level of care (as compared to patients who died in the ward without ever experiencing intra‐hospital transfer) suggests, but does not prove, that prolonged efforts were being made to keep them alive. We were also limited in terms of having access to other process data (eg, physician staffing levels, provision and timing of palliative care). Having ICU severity of illness scores would have permitted us to compare our cohort to those of other recent studies showing elevated mortality rates among transfer patients,911 but we have not yet developed that capability.

Consideration of our study findings suggests a possible research agenda that could be implemented by hospitalist researchers. This agenda should emphasize three areas: detection, intervention, and reflection.

With respect to detection, attention needs to be paid to better tools for quantifying patient risk at the time a decision to admit to the ward is made. It is likely that such tools will need to combine the attributes of our severity score (LAPS) with those of the manually assigned scores.30, 34 In some cases, use of these tools could lead a physician to change the locus of admission from the ward to the TCU or ICU, which could improve outcomes by ensuring more timely provision of intensive care. Since problems with initial triage could be due to factors other than the failure to suspect or anticipate impending instability, future research should also include a cognitive component (eg, quantifying what proportion of subsequent patient deteriorations could be ascribed to missed diagnoses35). Additional work also needs to be done on developing mathematical models that can inform electronic monitoring of ward (not just ICU) patients.

Research on interventions that hospitalists can use to prevent the need for intensive care or to improve the rescue rate should take two routes. The first is a disease‐specific route, which builds on the fact that a relatively small set of conditions (pneumonia, sepsis, gastrointestinal bleeding) account for most transfers to a higher level of care. Condition‐specific protocols, checklists, and bundles36 tailored to a ward environment (as opposed to the ICU or to the entire hospital) might prevent deteriorations in these patients, as has been reported for sepsis.37 The second route is to improve the overall capabilities of rapid response and code blue teams. Such research would need to include a more careful assessment of what commonalities exist among patients who were and were not successfully rescued by these teams. This approach would probably yield more insights than the current literature, which focuses on whether rapid response teams are a good thing or not.

Finally, research also needs to be performed on how hospitalists reflect on adverse outcomes among ward patients. Greater emphasis needs to be placed on moving beyond trigger tool approaches that rely on manual chart review. In an era of expanding use of electronic medical record systems, more work needs to be done on how to harness these to provide hospitalists with better quantitative and risk‐adjusted information. This information should not be limited to simply reporting rates of transfers and deaths. Rather, finer distinctions must be provided with respect of the type of patients (ie, more diagnostic detail), the clinical status of patients (ie, more physiologic detail), as well as the effects of including or excluding patients in whom therapeutic options may be limited (ie, do not resuscitate and comfort care patients) on reported rates. Ideally, researchers should develop better process and outcomes measures that could be tested in collaborative networks that include multiple nonacademic general medical‐surgical wards.