Mark E. Cowen, MD, SM

Few today deny the importance of the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey.[1, 2] The Centers for Medicare and Medicaid Services' (CMS) Value Based Purchasing incentive, sympathy for the ill, and relationships between the patient experience and quality of care provide sufficient justification.[3, 4] How to improve the experience scores is not well understood. The national scores have improved only modestly over the past 3 years.[5, 6]

Clinicians may not typically compartmentalize what they do to improve outcomes versus the patient experience. A possible source for new improvement strategies is to understand the types of patients in which both adverse outcomes and suboptimal experiences are likely to occur, then redesign the multidisciplinary care processes to address both concurrently.[7] Previous studies support the existence of a relationship between a higher mortality risk on admission and subsequent worse outcomes, as well as a relationship between worse outcomes and lower HCAHPS scores.[8, 9, 10, 11, 12, 13] We hypothesized the mortality risk on admission, patient experience, and outcomes might share a triad relationship (Figure 1). In this article we explore the third edge of this triangle, the association between the mortality risk on admission and the subsequent patient experience.

Figure 1

METHODS

We studied HCAHPS from 5 midwestern US hospitals having 113, 136, 304, 443, and 537 licensed beds, affiliated with the same multistate healthcare system. HCAHPS telephone surveys were administered via a vendor to a random sample of inpatients 18 years of age or older discharged from January 1, 2012 through June 30, 2014. Per CMS guidelines, surveyed patients must have been discharged alive after a hospital stay of at least 1 night.[14] Patients ineligible to be surveyed included those discharged to skilled nursing facilities or hospice care.[14] Because not all study hospitals provided obstetrical services, we restricted the analyses to medical and surgical respondents. With the permission of the local institutional review board, subjects' survey responses were linked confidentially to their clinical data.

We focused on the 8 dimensions of the care experience used in the CMS Value Based Purchasing program: communication with doctors, communication with nurses, responsiveness of hospital staff, pain management, communication about medicines, discharge information, hospital environment, and an overall rating of the hospital.[2] Following the scoring convention for publicly reported results, we dichotomized the 4‐level Likert scales into the most favorable response possible (always) versus all other responses.[15] Similarly we dichotomized the hospital rating scale at 9 and above for the most favorable response.

Our unit of analysis was an individual hospitalization. Our primary outcome of interest was whether or not the respondent provided the most favorable response for all questions answered within a given domain. For example, for the physician communication domain, the patient must have answered always to each of the 3 questions answered within the domain. This approach is appropriate for learning which patient‐level factors influence the survey responses, but differs from that used for the publically reported domain scores for which the relative performance of hospitals is the focus.[16] For the latter, the hospital was the unit of analysis, and the domain score was basically the average of the percentages of top box scores for the questions within a domain. For example, if 90% respondents from a hospital provided a top box response for courtesy, 80% for listening, and 70% for explanation, the hospital's physician communication score would be (90 + 80 + 70)/3 = 80%.[17]

Our primary explanatory variable was a binary high versus low mortality‐risk status of the respondent on admission based on age, gender, prior hospitalizations, clinical laboratory values, and diagnoses present on admission.[12] The calculated mortality risk was then dichotomized prior to the analysis at a probability of dying equal to 0.07 or higher. This corresponded roughly to the top quintile of predicted risk found in prior studies.[12, 13] During the study period, only 2 of the hospitals had the capability of generating mortality scores in real time, so for this study the mortality risk was calculated retrospectively, using information deemed present on admission.[12]

To estimate the sample size, we assumed that the high‐risk strata contained approximately 13% of respondents, and that the true percent of top box responses from patients in the lower‐risk stratum was approximately 80% for each domain. A meaningful difference in the proportion of most favorable responses was considered as an odds ratio (OR) of 0.75 for high risk versus low risk. A significance level of P < 0.003 was set to control study‐wide type I error due to multiple comparisons. We determined that for each dimension, approximately 8583 survey responses would be required for low‐risk patients and approximately 1116 responses for high‐risk patients to achieve 80% power under these assumptions. We were able to accrue the target number of surveys for all but 3 domains (pain management, communication about medicines, and hospital environment) because of data availability, and because patients are allowed to skip questions that do not apply. Univariate relationships were examined with ², t test, and Fisher exact tests where indicated. Generalized linear mixed regression models with a logit link were fit to determine the association between patient mortality risk and the top box experience for each of the HCAHPS domains and for the overall rating. The patient's hospital was considered a random intercept to account for the patient‐hospital hierarchy and the unmeasured hospital‐specific practices. The multivariable models controlled for gender plus the HCAHPS patient‐mix adjustment variables of age, education, self‐rated health, language spoken at home, service line, and the number of days elapsed between the date of discharge and date of the survey.[18, 19, 20, 21] In keeping with the industry analyses, a second order interaction variable was included between surgery patients and age.[19] We considered the potential collinearity between the mortality risk status, age, and patient self‐reported health. We found the variance inflation factors were small, so we drew inference from the full multivariable model.

We also performed a post hoc sensitivity analysis to determine if our conclusions were biased due to missing patient responses for the risk‐adjustment variables. Accordingly, we imputed the response level most negatively associated with most HCAHPS domains as previously reported and reran the multivariable models.[19] We did not find a meaningful change in our conclusions (see Supporting Figure 1 in the online version of this article).

RESULTS

The hospitals discharged 152,333 patients during the study period, 39,905 of whom (26.2 %) had a predicted 30‐day mortality risk greater or equal to 0.07 (Table 1). Of the 36,280 high‐risk patients discharged alive, 5901 (16.3%) died in the ensuing 30 days, and 7951 (22%) were readmitted.

The high‐risk subset was under‐represented in those who completed the HCAHPS survey with 7% (2513/36,280) completing surveys compared to 13% of low‐risk patients (14,996/111,600) (P < 0.0001). Moreover, compared to high‐risk patients who were alive at discharge but did not complete surveys, high‐risk survey respondents were less likely to die within 30 days (16/2513 = 0.64% vs 5885/33,767 = 17.4%, P < 0.0001), and less likely to be readmitted (424/2513 = 16.9% vs 7527/33,767 = 22.3%, P < 0.0001).

On average, high‐risk respondents (compared to low risk) were slightly less likely to be female (52.4% vs 57.9%), less educated (30.6% with some college vs 42.3%), less likely to have been on a surgical service (13.8% vs 42.5%), and less likely to report good or better health (49.0% vs 68.0%, all P < 0.0001). High‐risk respondents were also older (76.6 vs 63.1 years), stayed in the hospital longer (4.6 vs 3.5 days), and received care from more specialties (2.2 vs 1.7 specialties) (all P < 0.0001). High‐risk respondents experienced more 30‐day readmissions (16.9% vs 8.1%) and deaths within 30 days (0.6 % vs 0.1 %, all P < 0.0001) than their low‐risk counterparts.

High‐risk respondents were less likely to provide the most favorable response (unadjusted) for all HCAHPS domains compared to low‐risk respondents, although the difference was not significant for discharge information (Table 1, Figure 2A). The gradient between high‐risk and low‐risk patients was seen for all domains within each hospital except for pain management, hospital environment, and overall rating (Figure 3).

Figure 2

Figure 3

The multivariable regression models examined whether the mortality risk on admission simply represented older medical patients and/or those who considered themselves unhealthy (Figure 2B) (see Supporting Table 1 in the online version of this article). Accounting for hospital, age, gender, language, self‐reported health, educational level, service line, and days elapsed from discharge, respondents in the high‐mortality‐risk stratum were still less likely to report an always experience for doctor communication (OR: 0.85; 95% confidence interval [CI]: 0.77‐0.94) and responsiveness of hospital staff (OR: 0.77; 95% CI: 0.69‐0.85). Higher‐risk patients also tended to have less favorable experiences with nursing communication, although the CI crossed 1 (OR: 0.91; 95% CI: 0.82‐1.01). In contrast, higher‐risk patients were more likely to provide top box responses for having received discharge information (OR: 1.30; 95% CI: 1.14‐1.48). We did not find independent associations between mortality risk and the other domains when the patient risk‐adjustment factors were considered.[18, 19, 20, 21]

DISCUSSION

The high‐mortality‐risk stratum on admission contained a subset of patients who provided less favorable responses for almost all incentivized HCAHPS domains when other risk‐adjustment variables were not taken into consideration (Figure 2A). These univariate relationships weakened when we controlled for gender, the standard HCAHPS risk‐adjustment variables, and individual hospital influences (Figure 2B).[18, 19, 20, 21] After multivariable adjustment, survey respondents in the high‐risk category remained less likely to report their physicians always communicated well and to experience hospital staff responding quickly, but were more likely to report receiving discharge information. We did not find an independent association between the underlying mortality risk and the other incentivized HCAHPS domains after risk adjustment.

We are cautious with initial interpretations of our findings in light of the relatively small number of hospitals studied and the substantial survey response bias of healthier patients. Undoubtedly, the CMS exclusions of patients discharged to hospice or skilled nursing facilities provide a partial explanation for the selection bias, but the experience of those at high risk who did not complete surveys remains conjecture at this point.[14] Previous evidence suggests sicker patients and those with worse experiences are less likely to respond to the HCAHPS survey.[18, 22] On the other hand, it is possible that high‐risk nonrespondents who died could have received better communication and staff responsiveness.[23, 24] We were unable to find a previous, patient‐level study that explicitly tested the association between the admission mortality risk and the subsequent patient experience, yet our findings are consistent with a previous single‐site study of a surgical population showing lower overall ratings from patients with higher Injury Severity Scores.[25]

Our findings provide evidence of complex relationships among admission mortality risk, the 3 domains of the patient experience, and adverse outcomes, at least within the study hospitals (Figure 1). The developing field of palliative care has found very ill patients have special communication needs regarding goals of care, as well as physical symptoms, anxiety, and depression that might prompt more calls for help.[26] If these needs were more important for high‐risk compared to low‐risk patients, and were either not recognized or adequately addressed by the clinical teams at the study hospitals, then the high‐risk patients may have been less likely to perceive their physicians listened and explained things well, or that staff responded promptly to their requests for help.[27] On the other hand, the higher ratings for discharge information suggest the needs of the high‐risk patients were relatively easier to address by current practices at these hospitals. The lack of association between the mortality risk and the other HCAHPS domains may reflect the relatively stronger influence of age, gender, educational level, provider variability, and other unmeasured influences within the study sites, or that the level of patient need was similar among high‐risk and low‐risk patients within these domains.[27]

There are several possible confounders of our observed relationship between mortality risk and HCAHPS scores. The first category of confounders represents patient level variables that might impact the communication scores, some of which are part of the formula of our mortality prediction rule, for example, cognitive impairment and emergent admission.[18, 22, 27] The effect of the mortality risk could also be confounded by unmeasured patient‐level factors such as lower socioeconomic status.[28] A second category of confounders pertains to clinical outcomes and processes of care associated with serious illness irrespective of the risk of dying. More physicians involved in the care of the seriously ill (Table 1) may impact the communication scores, due to the larger opportunity for conflicting or confusing information presented to patients and their families.[29] The longer hospital stays, readmissions, and adverse events of the seriously ill may also underlie the apparent association between mortality risk and HCAHPS scores.[8, 9, 10]

Even if we do not understand precisely if and how the mortality risk might be associated with suboptimal physician communication and staff responsiveness, there may still be some value in considering how these possible relationships could be leveraged to improve patient care. We recall Berwick's insight that every system is perfectly designed to achieve the results it achieves.[7] We have previously argued for the use of mortality‐risk strata to initiate concurrent, multidisciplinary care processes to reduce adverse outcomes.[12, 13] Others have used risk‐based approaches for anticipating clinical deterioration of surgical patients, and determining the intensity of individualized case management services.[30, 31] In this framework, all patients receive a standard set of care processes, but higher‐risk patients receive additional efforts to promote better outcomes. An efficient extension of this approach is to assume patients at risk for adverse outcomes also have additional needs for communication, coordination of specialty care, and timely response to the call button. The admission mortality risk could be used as a determinant for the level of nurse staffing to reduce deaths plus shorten response time to the call button.[32, 33] Hospitalists and specialists could work together on a standard way to conference among themselves for high‐risk patients above that needed for less‐complex cases. Patients in the high‐risk strata could be screened early to see if they might benefit from the involvement of the palliative care team.[26]

Our study has limitations in addition to those already noted. First, our use of the top box as the formulation of the outcome of interest could be challenged. We chose this to be relevant to the Value‐Based Purchasing environment, but other formulations or use of other survey instruments may be needed to tease out the complex relationships we hypothesize. Next, we do not know the extent to which the patients and care processes reflected in our study represent other settings. The literature indicates some hospitals are more effective in providing care for certain subgroups of patients than for others, and that there is substantial regional variation in care intensity that is in turn associated with the patient experience.[29, 34] The mortality‐risk experience relationship for nonstudy hospitals could be weaker or stronger than what we found. Third, many hospitals may not have the capability to generate mortality scores on admission, although more hospitals may be able to do so in the future.[35] Explicit risk strata have the benefit of providing members of the multidisciplinary team with a quick preview of the clinical needs and prognoses of patients in much the way that the term baroque alerts the audience to the genre of music. Still, clinicians in any hospital could attempt to improve outcomes and experience through the use of informal risk assessment during interdisciplinary care rounds or by simply asking the team if they would be surprised if this patient died in the next year.[30, 36] Finally, we do not know if awareness of an experience risk will identify remediable practices that actually improve the experience. Clearly, future studies are needed to answer all of these concerns.

We have provided evidence that a group of patients who were at elevated risk for dying at the time of admission were more likely to have issues with physician communication and staff responsiveness than their lower‐risk counterparts. While we await future studies to confirm these findings, clinical teams can consider whether or not their patients' HCAHPS scores reflect how their system of care addresses the needs of these vulnerable people.

Few today deny the importance of the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey.[1, 2] The Centers for Medicare and Medicaid Services' (CMS) Value Based Purchasing incentive, sympathy for the ill, and relationships between the patient experience and quality of care provide sufficient justification.[3, 4] How to improve the experience scores is not well understood. The national scores have improved only modestly over the past 3 years.[5, 6]

Clinicians may not typically compartmentalize what they do to improve outcomes versus the patient experience. A possible source for new improvement strategies is to understand the types of patients in which both adverse outcomes and suboptimal experiences are likely to occur, then redesign the multidisciplinary care processes to address both concurrently.[7] Previous studies support the existence of a relationship between a higher mortality risk on admission and subsequent worse outcomes, as well as a relationship between worse outcomes and lower HCAHPS scores.[8, 9, 10, 11, 12, 13] We hypothesized the mortality risk on admission, patient experience, and outcomes might share a triad relationship (Figure 1). In this article we explore the third edge of this triangle, the association between the mortality risk on admission and the subsequent patient experience.

Figure 1

METHODS

We studied HCAHPS from 5 midwestern US hospitals having 113, 136, 304, 443, and 537 licensed beds, affiliated with the same multistate healthcare system. HCAHPS telephone surveys were administered via a vendor to a random sample of inpatients 18 years of age or older discharged from January 1, 2012 through June 30, 2014. Per CMS guidelines, surveyed patients must have been discharged alive after a hospital stay of at least 1 night.[14] Patients ineligible to be surveyed included those discharged to skilled nursing facilities or hospice care.[14] Because not all study hospitals provided obstetrical services, we restricted the analyses to medical and surgical respondents. With the permission of the local institutional review board, subjects' survey responses were linked confidentially to their clinical data.

We focused on the 8 dimensions of the care experience used in the CMS Value Based Purchasing program: communication with doctors, communication with nurses, responsiveness of hospital staff, pain management, communication about medicines, discharge information, hospital environment, and an overall rating of the hospital.[2] Following the scoring convention for publicly reported results, we dichotomized the 4‐level Likert scales into the most favorable response possible (always) versus all other responses.[15] Similarly we dichotomized the hospital rating scale at 9 and above for the most favorable response.

Our unit of analysis was an individual hospitalization. Our primary outcome of interest was whether or not the respondent provided the most favorable response for all questions answered within a given domain. For example, for the physician communication domain, the patient must have answered always to each of the 3 questions answered within the domain. This approach is appropriate for learning which patient‐level factors influence the survey responses, but differs from that used for the publically reported domain scores for which the relative performance of hospitals is the focus.[16] For the latter, the hospital was the unit of analysis, and the domain score was basically the average of the percentages of top box scores for the questions within a domain. For example, if 90% respondents from a hospital provided a top box response for courtesy, 80% for listening, and 70% for explanation, the hospital's physician communication score would be (90 + 80 + 70)/3 = 80%.[17]

Our primary explanatory variable was a binary high versus low mortality‐risk status of the respondent on admission based on age, gender, prior hospitalizations, clinical laboratory values, and diagnoses present on admission.[12] The calculated mortality risk was then dichotomized prior to the analysis at a probability of dying equal to 0.07 or higher. This corresponded roughly to the top quintile of predicted risk found in prior studies.[12, 13] During the study period, only 2 of the hospitals had the capability of generating mortality scores in real time, so for this study the mortality risk was calculated retrospectively, using information deemed present on admission.[12]

To estimate the sample size, we assumed that the high‐risk strata contained approximately 13% of respondents, and that the true percent of top box responses from patients in the lower‐risk stratum was approximately 80% for each domain. A meaningful difference in the proportion of most favorable responses was considered as an odds ratio (OR) of 0.75 for high risk versus low risk. A significance level of P < 0.003 was set to control study‐wide type I error due to multiple comparisons. We determined that for each dimension, approximately 8583 survey responses would be required for low‐risk patients and approximately 1116 responses for high‐risk patients to achieve 80% power under these assumptions. We were able to accrue the target number of surveys for all but 3 domains (pain management, communication about medicines, and hospital environment) because of data availability, and because patients are allowed to skip questions that do not apply. Univariate relationships were examined with ², t test, and Fisher exact tests where indicated. Generalized linear mixed regression models with a logit link were fit to determine the association between patient mortality risk and the top box experience for each of the HCAHPS domains and for the overall rating. The patient's hospital was considered a random intercept to account for the patient‐hospital hierarchy and the unmeasured hospital‐specific practices. The multivariable models controlled for gender plus the HCAHPS patient‐mix adjustment variables of age, education, self‐rated health, language spoken at home, service line, and the number of days elapsed between the date of discharge and date of the survey.[18, 19, 20, 21] In keeping with the industry analyses, a second order interaction variable was included between surgery patients and age.[19] We considered the potential collinearity between the mortality risk status, age, and patient self‐reported health. We found the variance inflation factors were small, so we drew inference from the full multivariable model.

We also performed a post hoc sensitivity analysis to determine if our conclusions were biased due to missing patient responses for the risk‐adjustment variables. Accordingly, we imputed the response level most negatively associated with most HCAHPS domains as previously reported and reran the multivariable models.[19] We did not find a meaningful change in our conclusions (see Supporting Figure 1 in the online version of this article).

RESULTS

The hospitals discharged 152,333 patients during the study period, 39,905 of whom (26.2 %) had a predicted 30‐day mortality risk greater or equal to 0.07 (Table 1). Of the 36,280 high‐risk patients discharged alive, 5901 (16.3%) died in the ensuing 30 days, and 7951 (22%) were readmitted.

The high‐risk subset was under‐represented in those who completed the HCAHPS survey with 7% (2513/36,280) completing surveys compared to 13% of low‐risk patients (14,996/111,600) (P < 0.0001). Moreover, compared to high‐risk patients who were alive at discharge but did not complete surveys, high‐risk survey respondents were less likely to die within 30 days (16/2513 = 0.64% vs 5885/33,767 = 17.4%, P < 0.0001), and less likely to be readmitted (424/2513 = 16.9% vs 7527/33,767 = 22.3%, P < 0.0001).

On average, high‐risk respondents (compared to low risk) were slightly less likely to be female (52.4% vs 57.9%), less educated (30.6% with some college vs 42.3%), less likely to have been on a surgical service (13.8% vs 42.5%), and less likely to report good or better health (49.0% vs 68.0%, all P < 0.0001). High‐risk respondents were also older (76.6 vs 63.1 years), stayed in the hospital longer (4.6 vs 3.5 days), and received care from more specialties (2.2 vs 1.7 specialties) (all P < 0.0001). High‐risk respondents experienced more 30‐day readmissions (16.9% vs 8.1%) and deaths within 30 days (0.6 % vs 0.1 %, all P < 0.0001) than their low‐risk counterparts.

High‐risk respondents were less likely to provide the most favorable response (unadjusted) for all HCAHPS domains compared to low‐risk respondents, although the difference was not significant for discharge information (Table 1, Figure 2A). The gradient between high‐risk and low‐risk patients was seen for all domains within each hospital except for pain management, hospital environment, and overall rating (Figure 3).

Figure 2

Figure 3

The multivariable regression models examined whether the mortality risk on admission simply represented older medical patients and/or those who considered themselves unhealthy (Figure 2B) (see Supporting Table 1 in the online version of this article). Accounting for hospital, age, gender, language, self‐reported health, educational level, service line, and days elapsed from discharge, respondents in the high‐mortality‐risk stratum were still less likely to report an always experience for doctor communication (OR: 0.85; 95% confidence interval [CI]: 0.77‐0.94) and responsiveness of hospital staff (OR: 0.77; 95% CI: 0.69‐0.85). Higher‐risk patients also tended to have less favorable experiences with nursing communication, although the CI crossed 1 (OR: 0.91; 95% CI: 0.82‐1.01). In contrast, higher‐risk patients were more likely to provide top box responses for having received discharge information (OR: 1.30; 95% CI: 1.14‐1.48). We did not find independent associations between mortality risk and the other domains when the patient risk‐adjustment factors were considered.[18, 19, 20, 21]

DISCUSSION

The high‐mortality‐risk stratum on admission contained a subset of patients who provided less favorable responses for almost all incentivized HCAHPS domains when other risk‐adjustment variables were not taken into consideration (Figure 2A). These univariate relationships weakened when we controlled for gender, the standard HCAHPS risk‐adjustment variables, and individual hospital influences (Figure 2B).[18, 19, 20, 21] After multivariable adjustment, survey respondents in the high‐risk category remained less likely to report their physicians always communicated well and to experience hospital staff responding quickly, but were more likely to report receiving discharge information. We did not find an independent association between the underlying mortality risk and the other incentivized HCAHPS domains after risk adjustment.

We are cautious with initial interpretations of our findings in light of the relatively small number of hospitals studied and the substantial survey response bias of healthier patients. Undoubtedly, the CMS exclusions of patients discharged to hospice or skilled nursing facilities provide a partial explanation for the selection bias, but the experience of those at high risk who did not complete surveys remains conjecture at this point.[14] Previous evidence suggests sicker patients and those with worse experiences are less likely to respond to the HCAHPS survey.[18, 22] On the other hand, it is possible that high‐risk nonrespondents who died could have received better communication and staff responsiveness.[23, 24] We were unable to find a previous, patient‐level study that explicitly tested the association between the admission mortality risk and the subsequent patient experience, yet our findings are consistent with a previous single‐site study of a surgical population showing lower overall ratings from patients with higher Injury Severity Scores.[25]

Our findings provide evidence of complex relationships among admission mortality risk, the 3 domains of the patient experience, and adverse outcomes, at least within the study hospitals (Figure 1). The developing field of palliative care has found very ill patients have special communication needs regarding goals of care, as well as physical symptoms, anxiety, and depression that might prompt more calls for help.[26] If these needs were more important for high‐risk compared to low‐risk patients, and were either not recognized or adequately addressed by the clinical teams at the study hospitals, then the high‐risk patients may have been less likely to perceive their physicians listened and explained things well, or that staff responded promptly to their requests for help.[27] On the other hand, the higher ratings for discharge information suggest the needs of the high‐risk patients were relatively easier to address by current practices at these hospitals. The lack of association between the mortality risk and the other HCAHPS domains may reflect the relatively stronger influence of age, gender, educational level, provider variability, and other unmeasured influences within the study sites, or that the level of patient need was similar among high‐risk and low‐risk patients within these domains.[27]

There are several possible confounders of our observed relationship between mortality risk and HCAHPS scores. The first category of confounders represents patient level variables that might impact the communication scores, some of which are part of the formula of our mortality prediction rule, for example, cognitive impairment and emergent admission.[18, 22, 27] The effect of the mortality risk could also be confounded by unmeasured patient‐level factors such as lower socioeconomic status.[28] A second category of confounders pertains to clinical outcomes and processes of care associated with serious illness irrespective of the risk of dying. More physicians involved in the care of the seriously ill (Table 1) may impact the communication scores, due to the larger opportunity for conflicting or confusing information presented to patients and their families.[29] The longer hospital stays, readmissions, and adverse events of the seriously ill may also underlie the apparent association between mortality risk and HCAHPS scores.[8, 9, 10]

Even if we do not understand precisely if and how the mortality risk might be associated with suboptimal physician communication and staff responsiveness, there may still be some value in considering how these possible relationships could be leveraged to improve patient care. We recall Berwick's insight that every system is perfectly designed to achieve the results it achieves.[7] We have previously argued for the use of mortality‐risk strata to initiate concurrent, multidisciplinary care processes to reduce adverse outcomes.[12, 13] Others have used risk‐based approaches for anticipating clinical deterioration of surgical patients, and determining the intensity of individualized case management services.[30, 31] In this framework, all patients receive a standard set of care processes, but higher‐risk patients receive additional efforts to promote better outcomes. An efficient extension of this approach is to assume patients at risk for adverse outcomes also have additional needs for communication, coordination of specialty care, and timely response to the call button. The admission mortality risk could be used as a determinant for the level of nurse staffing to reduce deaths plus shorten response time to the call button.[32, 33] Hospitalists and specialists could work together on a standard way to conference among themselves for high‐risk patients above that needed for less‐complex cases. Patients in the high‐risk strata could be screened early to see if they might benefit from the involvement of the palliative care team.[26]

Our study has limitations in addition to those already noted. First, our use of the top box as the formulation of the outcome of interest could be challenged. We chose this to be relevant to the Value‐Based Purchasing environment, but other formulations or use of other survey instruments may be needed to tease out the complex relationships we hypothesize. Next, we do not know the extent to which the patients and care processes reflected in our study represent other settings. The literature indicates some hospitals are more effective in providing care for certain subgroups of patients than for others, and that there is substantial regional variation in care intensity that is in turn associated with the patient experience.[29, 34] The mortality‐risk experience relationship for nonstudy hospitals could be weaker or stronger than what we found. Third, many hospitals may not have the capability to generate mortality scores on admission, although more hospitals may be able to do so in the future.[35] Explicit risk strata have the benefit of providing members of the multidisciplinary team with a quick preview of the clinical needs and prognoses of patients in much the way that the term baroque alerts the audience to the genre of music. Still, clinicians in any hospital could attempt to improve outcomes and experience through the use of informal risk assessment during interdisciplinary care rounds or by simply asking the team if they would be surprised if this patient died in the next year.[30, 36] Finally, we do not know if awareness of an experience risk will identify remediable practices that actually improve the experience. Clearly, future studies are needed to answer all of these concerns.

We have provided evidence that a group of patients who were at elevated risk for dying at the time of admission were more likely to have issues with physician communication and staff responsiveness than their lower‐risk counterparts. While we await future studies to confirm these findings, clinical teams can consider whether or not their patients' HCAHPS scores reflect how their system of care addresses the needs of these vulnerable people.

Few today deny the importance of the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey.[1, 2] The Centers for Medicare and Medicaid Services' (CMS) Value Based Purchasing incentive, sympathy for the ill, and relationships between the patient experience and quality of care provide sufficient justification.[3, 4] How to improve the experience scores is not well understood. The national scores have improved only modestly over the past 3 years.[5, 6]

Clinicians may not typically compartmentalize what they do to improve outcomes versus the patient experience. A possible source for new improvement strategies is to understand the types of patients in which both adverse outcomes and suboptimal experiences are likely to occur, then redesign the multidisciplinary care processes to address both concurrently.[7] Previous studies support the existence of a relationship between a higher mortality risk on admission and subsequent worse outcomes, as well as a relationship between worse outcomes and lower HCAHPS scores.[8, 9, 10, 11, 12, 13] We hypothesized the mortality risk on admission, patient experience, and outcomes might share a triad relationship (Figure 1). In this article we explore the third edge of this triangle, the association between the mortality risk on admission and the subsequent patient experience.

Figure 1

METHODS

We studied HCAHPS from 5 midwestern US hospitals having 113, 136, 304, 443, and 537 licensed beds, affiliated with the same multistate healthcare system. HCAHPS telephone surveys were administered via a vendor to a random sample of inpatients 18 years of age or older discharged from January 1, 2012 through June 30, 2014. Per CMS guidelines, surveyed patients must have been discharged alive after a hospital stay of at least 1 night.[14] Patients ineligible to be surveyed included those discharged to skilled nursing facilities or hospice care.[14] Because not all study hospitals provided obstetrical services, we restricted the analyses to medical and surgical respondents. With the permission of the local institutional review board, subjects' survey responses were linked confidentially to their clinical data.

We focused on the 8 dimensions of the care experience used in the CMS Value Based Purchasing program: communication with doctors, communication with nurses, responsiveness of hospital staff, pain management, communication about medicines, discharge information, hospital environment, and an overall rating of the hospital.[2] Following the scoring convention for publicly reported results, we dichotomized the 4‐level Likert scales into the most favorable response possible (always) versus all other responses.[15] Similarly we dichotomized the hospital rating scale at 9 and above for the most favorable response.

Our unit of analysis was an individual hospitalization. Our primary outcome of interest was whether or not the respondent provided the most favorable response for all questions answered within a given domain. For example, for the physician communication domain, the patient must have answered always to each of the 3 questions answered within the domain. This approach is appropriate for learning which patient‐level factors influence the survey responses, but differs from that used for the publically reported domain scores for which the relative performance of hospitals is the focus.[16] For the latter, the hospital was the unit of analysis, and the domain score was basically the average of the percentages of top box scores for the questions within a domain. For example, if 90% respondents from a hospital provided a top box response for courtesy, 80% for listening, and 70% for explanation, the hospital's physician communication score would be (90 + 80 + 70)/3 = 80%.[17]

Our primary explanatory variable was a binary high versus low mortality‐risk status of the respondent on admission based on age, gender, prior hospitalizations, clinical laboratory values, and diagnoses present on admission.[12] The calculated mortality risk was then dichotomized prior to the analysis at a probability of dying equal to 0.07 or higher. This corresponded roughly to the top quintile of predicted risk found in prior studies.[12, 13] During the study period, only 2 of the hospitals had the capability of generating mortality scores in real time, so for this study the mortality risk was calculated retrospectively, using information deemed present on admission.[12]

To estimate the sample size, we assumed that the high‐risk strata contained approximately 13% of respondents, and that the true percent of top box responses from patients in the lower‐risk stratum was approximately 80% for each domain. A meaningful difference in the proportion of most favorable responses was considered as an odds ratio (OR) of 0.75 for high risk versus low risk. A significance level of P < 0.003 was set to control study‐wide type I error due to multiple comparisons. We determined that for each dimension, approximately 8583 survey responses would be required for low‐risk patients and approximately 1116 responses for high‐risk patients to achieve 80% power under these assumptions. We were able to accrue the target number of surveys for all but 3 domains (pain management, communication about medicines, and hospital environment) because of data availability, and because patients are allowed to skip questions that do not apply. Univariate relationships were examined with ², t test, and Fisher exact tests where indicated. Generalized linear mixed regression models with a logit link were fit to determine the association between patient mortality risk and the top box experience for each of the HCAHPS domains and for the overall rating. The patient's hospital was considered a random intercept to account for the patient‐hospital hierarchy and the unmeasured hospital‐specific practices. The multivariable models controlled for gender plus the HCAHPS patient‐mix adjustment variables of age, education, self‐rated health, language spoken at home, service line, and the number of days elapsed between the date of discharge and date of the survey.[18, 19, 20, 21] In keeping with the industry analyses, a second order interaction variable was included between surgery patients and age.[19] We considered the potential collinearity between the mortality risk status, age, and patient self‐reported health. We found the variance inflation factors were small, so we drew inference from the full multivariable model.

We also performed a post hoc sensitivity analysis to determine if our conclusions were biased due to missing patient responses for the risk‐adjustment variables. Accordingly, we imputed the response level most negatively associated with most HCAHPS domains as previously reported and reran the multivariable models.[19] We did not find a meaningful change in our conclusions (see Supporting Figure 1 in the online version of this article).

RESULTS

The hospitals discharged 152,333 patients during the study period, 39,905 of whom (26.2 %) had a predicted 30‐day mortality risk greater or equal to 0.07 (Table 1). Of the 36,280 high‐risk patients discharged alive, 5901 (16.3%) died in the ensuing 30 days, and 7951 (22%) were readmitted.

The high‐risk subset was under‐represented in those who completed the HCAHPS survey with 7% (2513/36,280) completing surveys compared to 13% of low‐risk patients (14,996/111,600) (P < 0.0001). Moreover, compared to high‐risk patients who were alive at discharge but did not complete surveys, high‐risk survey respondents were less likely to die within 30 days (16/2513 = 0.64% vs 5885/33,767 = 17.4%, P < 0.0001), and less likely to be readmitted (424/2513 = 16.9% vs 7527/33,767 = 22.3%, P < 0.0001).

On average, high‐risk respondents (compared to low risk) were slightly less likely to be female (52.4% vs 57.9%), less educated (30.6% with some college vs 42.3%), less likely to have been on a surgical service (13.8% vs 42.5%), and less likely to report good or better health (49.0% vs 68.0%, all P < 0.0001). High‐risk respondents were also older (76.6 vs 63.1 years), stayed in the hospital longer (4.6 vs 3.5 days), and received care from more specialties (2.2 vs 1.7 specialties) (all P < 0.0001). High‐risk respondents experienced more 30‐day readmissions (16.9% vs 8.1%) and deaths within 30 days (0.6 % vs 0.1 %, all P < 0.0001) than their low‐risk counterparts.

High‐risk respondents were less likely to provide the most favorable response (unadjusted) for all HCAHPS domains compared to low‐risk respondents, although the difference was not significant for discharge information (Table 1, Figure 2A). The gradient between high‐risk and low‐risk patients was seen for all domains within each hospital except for pain management, hospital environment, and overall rating (Figure 3).

Figure 2

Figure 3

The multivariable regression models examined whether the mortality risk on admission simply represented older medical patients and/or those who considered themselves unhealthy (Figure 2B) (see Supporting Table 1 in the online version of this article). Accounting for hospital, age, gender, language, self‐reported health, educational level, service line, and days elapsed from discharge, respondents in the high‐mortality‐risk stratum were still less likely to report an always experience for doctor communication (OR: 0.85; 95% confidence interval [CI]: 0.77‐0.94) and responsiveness of hospital staff (OR: 0.77; 95% CI: 0.69‐0.85). Higher‐risk patients also tended to have less favorable experiences with nursing communication, although the CI crossed 1 (OR: 0.91; 95% CI: 0.82‐1.01). In contrast, higher‐risk patients were more likely to provide top box responses for having received discharge information (OR: 1.30; 95% CI: 1.14‐1.48). We did not find independent associations between mortality risk and the other domains when the patient risk‐adjustment factors were considered.[18, 19, 20, 21]

DISCUSSION

The high‐mortality‐risk stratum on admission contained a subset of patients who provided less favorable responses for almost all incentivized HCAHPS domains when other risk‐adjustment variables were not taken into consideration (Figure 2A). These univariate relationships weakened when we controlled for gender, the standard HCAHPS risk‐adjustment variables, and individual hospital influences (Figure 2B).[18, 19, 20, 21] After multivariable adjustment, survey respondents in the high‐risk category remained less likely to report their physicians always communicated well and to experience hospital staff responding quickly, but were more likely to report receiving discharge information. We did not find an independent association between the underlying mortality risk and the other incentivized HCAHPS domains after risk adjustment.

We are cautious with initial interpretations of our findings in light of the relatively small number of hospitals studied and the substantial survey response bias of healthier patients. Undoubtedly, the CMS exclusions of patients discharged to hospice or skilled nursing facilities provide a partial explanation for the selection bias, but the experience of those at high risk who did not complete surveys remains conjecture at this point.[14] Previous evidence suggests sicker patients and those with worse experiences are less likely to respond to the HCAHPS survey.[18, 22] On the other hand, it is possible that high‐risk nonrespondents who died could have received better communication and staff responsiveness.[23, 24] We were unable to find a previous, patient‐level study that explicitly tested the association between the admission mortality risk and the subsequent patient experience, yet our findings are consistent with a previous single‐site study of a surgical population showing lower overall ratings from patients with higher Injury Severity Scores.[25]

Our findings provide evidence of complex relationships among admission mortality risk, the 3 domains of the patient experience, and adverse outcomes, at least within the study hospitals (Figure 1). The developing field of palliative care has found very ill patients have special communication needs regarding goals of care, as well as physical symptoms, anxiety, and depression that might prompt more calls for help.[26] If these needs were more important for high‐risk compared to low‐risk patients, and were either not recognized or adequately addressed by the clinical teams at the study hospitals, then the high‐risk patients may have been less likely to perceive their physicians listened and explained things well, or that staff responded promptly to their requests for help.[27] On the other hand, the higher ratings for discharge information suggest the needs of the high‐risk patients were relatively easier to address by current practices at these hospitals. The lack of association between the mortality risk and the other HCAHPS domains may reflect the relatively stronger influence of age, gender, educational level, provider variability, and other unmeasured influences within the study sites, or that the level of patient need was similar among high‐risk and low‐risk patients within these domains.[27]

There are several possible confounders of our observed relationship between mortality risk and HCAHPS scores. The first category of confounders represents patient level variables that might impact the communication scores, some of which are part of the formula of our mortality prediction rule, for example, cognitive impairment and emergent admission.[18, 22, 27] The effect of the mortality risk could also be confounded by unmeasured patient‐level factors such as lower socioeconomic status.[28] A second category of confounders pertains to clinical outcomes and processes of care associated with serious illness irrespective of the risk of dying. More physicians involved in the care of the seriously ill (Table 1) may impact the communication scores, due to the larger opportunity for conflicting or confusing information presented to patients and their families.[29] The longer hospital stays, readmissions, and adverse events of the seriously ill may also underlie the apparent association between mortality risk and HCAHPS scores.[8, 9, 10]

Even if we do not understand precisely if and how the mortality risk might be associated with suboptimal physician communication and staff responsiveness, there may still be some value in considering how these possible relationships could be leveraged to improve patient care. We recall Berwick's insight that every system is perfectly designed to achieve the results it achieves.[7] We have previously argued for the use of mortality‐risk strata to initiate concurrent, multidisciplinary care processes to reduce adverse outcomes.[12, 13] Others have used risk‐based approaches for anticipating clinical deterioration of surgical patients, and determining the intensity of individualized case management services.[30, 31] In this framework, all patients receive a standard set of care processes, but higher‐risk patients receive additional efforts to promote better outcomes. An efficient extension of this approach is to assume patients at risk for adverse outcomes also have additional needs for communication, coordination of specialty care, and timely response to the call button. The admission mortality risk could be used as a determinant for the level of nurse staffing to reduce deaths plus shorten response time to the call button.[32, 33] Hospitalists and specialists could work together on a standard way to conference among themselves for high‐risk patients above that needed for less‐complex cases. Patients in the high‐risk strata could be screened early to see if they might benefit from the involvement of the palliative care team.[26]

Our study has limitations in addition to those already noted. First, our use of the top box as the formulation of the outcome of interest could be challenged. We chose this to be relevant to the Value‐Based Purchasing environment, but other formulations or use of other survey instruments may be needed to tease out the complex relationships we hypothesize. Next, we do not know the extent to which the patients and care processes reflected in our study represent other settings. The literature indicates some hospitals are more effective in providing care for certain subgroups of patients than for others, and that there is substantial regional variation in care intensity that is in turn associated with the patient experience.[29, 34] The mortality‐risk experience relationship for nonstudy hospitals could be weaker or stronger than what we found. Third, many hospitals may not have the capability to generate mortality scores on admission, although more hospitals may be able to do so in the future.[35] Explicit risk strata have the benefit of providing members of the multidisciplinary team with a quick preview of the clinical needs and prognoses of patients in much the way that the term baroque alerts the audience to the genre of music. Still, clinicians in any hospital could attempt to improve outcomes and experience through the use of informal risk assessment during interdisciplinary care rounds or by simply asking the team if they would be surprised if this patient died in the next year.[30, 36] Finally, we do not know if awareness of an experience risk will identify remediable practices that actually improve the experience. Clearly, future studies are needed to answer all of these concerns.

We have provided evidence that a group of patients who were at elevated risk for dying at the time of admission were more likely to have issues with physician communication and staff responsiveness than their lower‐risk counterparts. While we await future studies to confirm these findings, clinical teams can consider whether or not their patients' HCAHPS scores reflect how their system of care addresses the needs of these vulnerable people.

The systematic deployment of prediction rules within health systems remains a challenge, although such decision aids have been available for decades.[1, 2] We previously developed and validated a prediction rule for 30‐day mortality in a retrospective cohort, noting that the mortality risk is associated with a number of other clinical events.[3] These relationships suggest risk strata, defined by the predicted probability of 30‐day mortality, and could trigger a number of coordinated care processes proportional to the level of risk.[4] For example, patients within the higher‐risk strata could be considered for placement into an intermediate or intensive care unit (ICU), be monitored more closely by physician and nurse team members for clinical deterioration, be seen by a physician within a few days of hospital discharge, and be considered for advance care planning discussions.[3, 4, 5, 6, 7] Patients within the lower‐risk strata might not need the same intensity of these processes routinely unless some other indication were present.

However attractive this conceptual framework may be, its realization is dependent on the willingness of clinical staff to generate predictions consistently on a substantial portion of the patient population, and on the accuracy of the predictions when the risk factors are determined with some level of uncertainty at the beginning of the hospitalization.[2, 8] Skepticism is justified, because the work involved in completing the prediction rule might be incompatible with existing workflow. A patient might not be scored if the emergency physician lacks time or if technical issues arise with the information system and computation process.[9] There is also a generic concern that the predictions will prove to be less accurate outside of the original study population.[8, 9, 10] A more specific concern for our rule is how well present on admission diagnoses can be determined during the relatively short emergency department or presurgery evaluation period. For example, a final diagnosis of heart failure might not be established until later in the hospitalization, after the results of diagnostic testing and clinical response to treatment are known. Moreover, our retrospective prediction rule requires an assessment of the presence or absence of sepsis and respiratory failure. These diagnoses appear to be susceptible to secular trends in medical record coding practices, suggesting the rule's accuracy might not be stable over time.[11]

We report the feasibility of having emergency physicians and the surgical preparation center team generate mortality predictions before an inpatient bed is assigned. We evaluate and report the accuracy of these prospective predictions.

METHODS

The study population consisted of all patients 18 years of age or less than 100 years who were admitted from the emergency department or assigned an inpatient bed following elective surgery at a tertiary, community teaching hospital in the Midwestern United States from September 1, 2012 through February 15, 2013. Although patients entering the hospital from these 2 pathways would be expected to have different levels of mortality risk, we used the original prediction rule for both because such distinctions were not made in its derivation and validation. Patients were not considered if they were admitted for childbirth or other obstetrical reasons, admitted directly from physician offices, the cardiac catheterization laboratory, hemodialysis unit, or from another hospital. The site institutional review board approved this study.

The implementation process began with presentations to the administrative and medical staff leadership on the accuracy of the retrospectively generated mortality predictions and risk of other adverse events.[3] The chief medical and nursing officers became project champions, secured internal funding for the technical components, and arranged to have 2 project comanagers available. A multidisciplinary task force endorsed the implementation details at biweekly meetings throughout the planning year. The leadership of the emergency department and surgical preparation center committed their colleagues to generate the predictions. The support of the emergency leadership was contingent on the completion of the entire prediction generating process in a very short time (within the time a physician could hold his/her breath). The chief medical officer, with the support of the leadership of the hospitalists and emergency physicians, made the administrative decision that a prediction must be generated prior to the assignment of a hospital room.

During the consensus‐building phase, a Web‐based application was developed to generate the predictions. Emergency physicians and surgical preparation staff were trained on the definitions of the risk factors (see Supporting Information, Appendix, in the online version of this article) and how to use the Web application. Three supporting databases were created. Each midnight, a past medical history database was updated, identifying those who had been discharged from the study hospital in the previous 365 days, and whether or not their diagnoses included atrial fibrillation, leukemia/lymphoma, metastatic cancer, cancer other than leukemia, lymphoma, cognitive disorder, or other neurological conditions (eg, Parkinson's, multiple sclerosis, epilepsy, coma, and stupor). Similarly, a clinical laboratory results database was created and updated real time through an HL7 (Health Level Seven, a standard data exchange format[12]) interface with the laboratory information system for the following tests performed in the preceding 30 days at a hospital‐affiliated facility: hemoglobin, platelet count, white blood count, serum troponin, blood urea nitrogen, serum albumin, serum lactate, arterial pH, arterial partial pressure of oxygen values. The third database, admission‐discharge‐transfer, was created and updated every 15 minutes to identify patients currently in the emergency room or scheduled for surgery. When a patient registration event was added to this database, the Web application created a record, retrieved all relevant data, and displayed the patient name for scoring. When the decision for hospitalization was made, the clinician selected the patient's name and reviewed the pre‐populated medical diagnoses of interest, which could be overwritten based on his/her own assessment (Figure 1A,B). The clinician then indicated (yes, no, or unknown) if the patient currently had or was being treated for each of the following: injury, heart failure, sepsis, respiratory failure, and whether or not the admitting service would be medicine (ie, nonsurgical, nonobstetrical). We considered unknown status to indicate the patient did not have the condition. When laboratory values were not available, a normal value was imputed using a previously developed algorithm.[3] Two additional questions, not used in the current prediction process, were answered to provide data for a future analysis: 1 concerning the change in the patient's condition while in the emergency department and the other concerning the presence of abnormal vital signs. The probability of 30‐day mortality was calculated via the Web application using the risk information supplied and the scoring weights (ie, parameter estimates) provided in the Appendices of our original publication.[3] Predictions were updated every minute as new laboratory values became available, and flagged with an alert if a more severe score resulted.

Figure 1

For the analyses of this study, the last prospective prediction viewed by emergency department personnel, a hospital bed manager, or surgical suite staff prior to arrival on the nursing unit is the one referenced as prospective. Once the patient had been discharged from the hospital, we generated a second mortality prediction based on previously published parameter estimates applied to risk factor data ascertained retrospectively as was done in the original article[3]; we subsequently refer to this prediction as retrospective. We will report on the group of patients who had both prospective and retrospective scores (1 patient had a prospective but not retrospective score available).

The prediction scores were made available to the clinical teams gradually during the study period. All scores were viewable by the midpoint of the study for emergency department admissions and near the end of the study for elective‐surgery patients. Only 2 changes in care processes based on level of risk were introduced during the study period. The first required initial placement of patients having a probability of dying of 0.3 or greater into an intensive or intermediate care unit unless the patient or family requested a less aggressive approach. The second occurred in the final 2 months of the study when a large multispecialty practice began routinely arranging for high‐risk patients to be seen within 3 or 7 days of hospital discharge.

RESULTS

Mortality predictions were generated on demand for 7291 out of 7777 (93.8%) eligible patients admitted from the emergency department, and for 2021 out of 2250 (89.8%) eligible elective surgical cases, for a total of 9312 predictions generated out of a possible 10,027 hospitalizations (92.9%). Table 1 displays the characteristics of the study population. The mean age was 65.2 years and 53.8% were women. The most common risk factors were atrial fibrillation (16.4%) and cancer (14.6%). Orders for a comfort care approach (rather than curative) were entered within 4 hours of admission for 32/9312 patients (0.34%), and 9/9312 (0.1%) were hospice patients on admission.

Evaluation of Prediction Accuracy

The AROC for 30‐day mortality was 0.850 (95% confidence interval [CI]: 0.833‐0.866) for prospectively collected covariates, and 0.870 (95% CI: 0.855‐0.885) for retrospectively determined risk factors. These AROCs are not substantively different from each other, demonstrating comparable prediction performance. Calibration was excellent, as indicated in Figure 2, in which the predicted level of risk lay within the 95% confidence limits of the actual 30‐day mortality for 19 out of 20 intervals of 5 percentile increments.

Figure 2

DISCUSSION

Emergency physicians and surgical preparation center nurses generated predictions by the time of hospital admission for over 90% of the target population during usual workflow, without the addition of staff or resources. The discrimination of the prospectively generated predictions was very good to excellent, with an AROC of 0.850 (95% CI: 0.833‐0.866), similar to that obtained from the retrospective version. Calibration was excellent. The prospectively calculated mortality risk was associated with a number of other events. As shown in Table 3, the differing frequency of events within the risk strata support the development of differing intensities of multidisciplinary strategies according to the level of risk.[5] Our study provides useful experience for others who anticipate generating real‐time predictions. We consider the key reasons for success to be the considerable time spent achieving consensus, the technical development of the Web application, the brief clinician time required for the scoring process, the leadership of the chief medical and nursing officers, and the requirement that a prediction be generated before assignment of a hospital room.

Our study has a number of limitations, some of which were noted in our original publication, and although still relevant, will not be repeated here for space considerations. This is a single‐site study that used a prediction rule developed by the same site, albeit on a patient population 4 to 5 years earlier. It is not known how well the specific rule might perform in other hospital populations; any such use should therefore be accompanied by independent validation studies prior to implementation. Our successful experience should motivate future validation studies. Second, because the prognoses of patients scored from the emergency department are likely to be worse than those of elective surgery patients, our rule should be recalibrated for each subgroup separately. We plan to do this in the near future, as well as consider additional risk factors. Third, the other events of interest might be predicted more accurately if rules specifically developed for each were deployed. The mortality risk by itself is unlikely to be a sufficiently accurate predictor, particularly for complications and intensive care use, for reasons outlined in our original publication.[3] However, the varying levels of events within the higher versus lower strata should be noted by the clinical team as they design their team‐based processes. A follow‐up visit with a physician within a few days of discharge could address the concurrent risk of dying as well as readmission, for example. Finally, it is too early to determine if the availability of mortality predictions from this rule will benefit patients.[2, 8, 10] During the study period, we implemented only 2 new care processes based on the level of risk. This lack of interventions allowed us to evaluate the prediction accuracy with minimal additional confounding, but at the expense of not yet knowing the clinical impact of this work. After the study period, we implemented a number of other interventions and plan on evaluating their effectiveness in the future. We are also considering an evaluation of the potential information gained by updating the predictions throughout the course of the hospitalization.[14]

In conclusion, it is feasible to have a reasonably accurate prediction of mortality risk for most adult patients at the beginning of their hospitalizations. The availability of this prognostic information provides an opportunity to develop proactive care plans for high‐ and low‐risk subsets of patients.

The systematic deployment of prediction rules within health systems remains a challenge, although such decision aids have been available for decades.[1, 2] We previously developed and validated a prediction rule for 30‐day mortality in a retrospective cohort, noting that the mortality risk is associated with a number of other clinical events.[3] These relationships suggest risk strata, defined by the predicted probability of 30‐day mortality, and could trigger a number of coordinated care processes proportional to the level of risk.[4] For example, patients within the higher‐risk strata could be considered for placement into an intermediate or intensive care unit (ICU), be monitored more closely by physician and nurse team members for clinical deterioration, be seen by a physician within a few days of hospital discharge, and be considered for advance care planning discussions.[3, 4, 5, 6, 7] Patients within the lower‐risk strata might not need the same intensity of these processes routinely unless some other indication were present.

However attractive this conceptual framework may be, its realization is dependent on the willingness of clinical staff to generate predictions consistently on a substantial portion of the patient population, and on the accuracy of the predictions when the risk factors are determined with some level of uncertainty at the beginning of the hospitalization.[2, 8] Skepticism is justified, because the work involved in completing the prediction rule might be incompatible with existing workflow. A patient might not be scored if the emergency physician lacks time or if technical issues arise with the information system and computation process.[9] There is also a generic concern that the predictions will prove to be less accurate outside of the original study population.[8, 9, 10] A more specific concern for our rule is how well present on admission diagnoses can be determined during the relatively short emergency department or presurgery evaluation period. For example, a final diagnosis of heart failure might not be established until later in the hospitalization, after the results of diagnostic testing and clinical response to treatment are known. Moreover, our retrospective prediction rule requires an assessment of the presence or absence of sepsis and respiratory failure. These diagnoses appear to be susceptible to secular trends in medical record coding practices, suggesting the rule's accuracy might not be stable over time.[11]

We report the feasibility of having emergency physicians and the surgical preparation center team generate mortality predictions before an inpatient bed is assigned. We evaluate and report the accuracy of these prospective predictions.

METHODS

The study population consisted of all patients 18 years of age or less than 100 years who were admitted from the emergency department or assigned an inpatient bed following elective surgery at a tertiary, community teaching hospital in the Midwestern United States from September 1, 2012 through February 15, 2013. Although patients entering the hospital from these 2 pathways would be expected to have different levels of mortality risk, we used the original prediction rule for both because such distinctions were not made in its derivation and validation. Patients were not considered if they were admitted for childbirth or other obstetrical reasons, admitted directly from physician offices, the cardiac catheterization laboratory, hemodialysis unit, or from another hospital. The site institutional review board approved this study.

The implementation process began with presentations to the administrative and medical staff leadership on the accuracy of the retrospectively generated mortality predictions and risk of other adverse events.[3] The chief medical and nursing officers became project champions, secured internal funding for the technical components, and arranged to have 2 project comanagers available. A multidisciplinary task force endorsed the implementation details at biweekly meetings throughout the planning year. The leadership of the emergency department and surgical preparation center committed their colleagues to generate the predictions. The support of the emergency leadership was contingent on the completion of the entire prediction generating process in a very short time (within the time a physician could hold his/her breath). The chief medical officer, with the support of the leadership of the hospitalists and emergency physicians, made the administrative decision that a prediction must be generated prior to the assignment of a hospital room.

During the consensus‐building phase, a Web‐based application was developed to generate the predictions. Emergency physicians and surgical preparation staff were trained on the definitions of the risk factors (see Supporting Information, Appendix, in the online version of this article) and how to use the Web application. Three supporting databases were created. Each midnight, a past medical history database was updated, identifying those who had been discharged from the study hospital in the previous 365 days, and whether or not their diagnoses included atrial fibrillation, leukemia/lymphoma, metastatic cancer, cancer other than leukemia, lymphoma, cognitive disorder, or other neurological conditions (eg, Parkinson's, multiple sclerosis, epilepsy, coma, and stupor). Similarly, a clinical laboratory results database was created and updated real time through an HL7 (Health Level Seven, a standard data exchange format[12]) interface with the laboratory information system for the following tests performed in the preceding 30 days at a hospital‐affiliated facility: hemoglobin, platelet count, white blood count, serum troponin, blood urea nitrogen, serum albumin, serum lactate, arterial pH, arterial partial pressure of oxygen values. The third database, admission‐discharge‐transfer, was created and updated every 15 minutes to identify patients currently in the emergency room or scheduled for surgery. When a patient registration event was added to this database, the Web application created a record, retrieved all relevant data, and displayed the patient name for scoring. When the decision for hospitalization was made, the clinician selected the patient's name and reviewed the pre‐populated medical diagnoses of interest, which could be overwritten based on his/her own assessment (Figure 1A,B). The clinician then indicated (yes, no, or unknown) if the patient currently had or was being treated for each of the following: injury, heart failure, sepsis, respiratory failure, and whether or not the admitting service would be medicine (ie, nonsurgical, nonobstetrical). We considered unknown status to indicate the patient did not have the condition. When laboratory values were not available, a normal value was imputed using a previously developed algorithm.[3] Two additional questions, not used in the current prediction process, were answered to provide data for a future analysis: 1 concerning the change in the patient's condition while in the emergency department and the other concerning the presence of abnormal vital signs. The probability of 30‐day mortality was calculated via the Web application using the risk information supplied and the scoring weights (ie, parameter estimates) provided in the Appendices of our original publication.[3] Predictions were updated every minute as new laboratory values became available, and flagged with an alert if a more severe score resulted.

Figure 1

For the analyses of this study, the last prospective prediction viewed by emergency department personnel, a hospital bed manager, or surgical suite staff prior to arrival on the nursing unit is the one referenced as prospective. Once the patient had been discharged from the hospital, we generated a second mortality prediction based on previously published parameter estimates applied to risk factor data ascertained retrospectively as was done in the original article[3]; we subsequently refer to this prediction as retrospective. We will report on the group of patients who had both prospective and retrospective scores (1 patient had a prospective but not retrospective score available).

The prediction scores were made available to the clinical teams gradually during the study period. All scores were viewable by the midpoint of the study for emergency department admissions and near the end of the study for elective‐surgery patients. Only 2 changes in care processes based on level of risk were introduced during the study period. The first required initial placement of patients having a probability of dying of 0.3 or greater into an intensive or intermediate care unit unless the patient or family requested a less aggressive approach. The second occurred in the final 2 months of the study when a large multispecialty practice began routinely arranging for high‐risk patients to be seen within 3 or 7 days of hospital discharge.

RESULTS

Mortality predictions were generated on demand for 7291 out of 7777 (93.8%) eligible patients admitted from the emergency department, and for 2021 out of 2250 (89.8%) eligible elective surgical cases, for a total of 9312 predictions generated out of a possible 10,027 hospitalizations (92.9%). Table 1 displays the characteristics of the study population. The mean age was 65.2 years and 53.8% were women. The most common risk factors were atrial fibrillation (16.4%) and cancer (14.6%). Orders for a comfort care approach (rather than curative) were entered within 4 hours of admission for 32/9312 patients (0.34%), and 9/9312 (0.1%) were hospice patients on admission.

Evaluation of Prediction Accuracy

The AROC for 30‐day mortality was 0.850 (95% confidence interval [CI]: 0.833‐0.866) for prospectively collected covariates, and 0.870 (95% CI: 0.855‐0.885) for retrospectively determined risk factors. These AROCs are not substantively different from each other, demonstrating comparable prediction performance. Calibration was excellent, as indicated in Figure 2, in which the predicted level of risk lay within the 95% confidence limits of the actual 30‐day mortality for 19 out of 20 intervals of 5 percentile increments.

Figure 2

DISCUSSION

Emergency physicians and surgical preparation center nurses generated predictions by the time of hospital admission for over 90% of the target population during usual workflow, without the addition of staff or resources. The discrimination of the prospectively generated predictions was very good to excellent, with an AROC of 0.850 (95% CI: 0.833‐0.866), similar to that obtained from the retrospective version. Calibration was excellent. The prospectively calculated mortality risk was associated with a number of other events. As shown in Table 3, the differing frequency of events within the risk strata support the development of differing intensities of multidisciplinary strategies according to the level of risk.[5] Our study provides useful experience for others who anticipate generating real‐time predictions. We consider the key reasons for success to be the considerable time spent achieving consensus, the technical development of the Web application, the brief clinician time required for the scoring process, the leadership of the chief medical and nursing officers, and the requirement that a prediction be generated before assignment of a hospital room.

Our study has a number of limitations, some of which were noted in our original publication, and although still relevant, will not be repeated here for space considerations. This is a single‐site study that used a prediction rule developed by the same site, albeit on a patient population 4 to 5 years earlier. It is not known how well the specific rule might perform in other hospital populations; any such use should therefore be accompanied by independent validation studies prior to implementation. Our successful experience should motivate future validation studies. Second, because the prognoses of patients scored from the emergency department are likely to be worse than those of elective surgery patients, our rule should be recalibrated for each subgroup separately. We plan to do this in the near future, as well as consider additional risk factors. Third, the other events of interest might be predicted more accurately if rules specifically developed for each were deployed. The mortality risk by itself is unlikely to be a sufficiently accurate predictor, particularly for complications and intensive care use, for reasons outlined in our original publication.[3] However, the varying levels of events within the higher versus lower strata should be noted by the clinical team as they design their team‐based processes. A follow‐up visit with a physician within a few days of discharge could address the concurrent risk of dying as well as readmission, for example. Finally, it is too early to determine if the availability of mortality predictions from this rule will benefit patients.[2, 8, 10] During the study period, we implemented only 2 new care processes based on the level of risk. This lack of interventions allowed us to evaluate the prediction accuracy with minimal additional confounding, but at the expense of not yet knowing the clinical impact of this work. After the study period, we implemented a number of other interventions and plan on evaluating their effectiveness in the future. We are also considering an evaluation of the potential information gained by updating the predictions throughout the course of the hospitalization.[14]

In conclusion, it is feasible to have a reasonably accurate prediction of mortality risk for most adult patients at the beginning of their hospitalizations. The availability of this prognostic information provides an opportunity to develop proactive care plans for high‐ and low‐risk subsets of patients.

The systematic deployment of prediction rules within health systems remains a challenge, although such decision aids have been available for decades.[1, 2] We previously developed and validated a prediction rule for 30‐day mortality in a retrospective cohort, noting that the mortality risk is associated with a number of other clinical events.[3] These relationships suggest risk strata, defined by the predicted probability of 30‐day mortality, and could trigger a number of coordinated care processes proportional to the level of risk.[4] For example, patients within the higher‐risk strata could be considered for placement into an intermediate or intensive care unit (ICU), be monitored more closely by physician and nurse team members for clinical deterioration, be seen by a physician within a few days of hospital discharge, and be considered for advance care planning discussions.[3, 4, 5, 6, 7] Patients within the lower‐risk strata might not need the same intensity of these processes routinely unless some other indication were present.

However attractive this conceptual framework may be, its realization is dependent on the willingness of clinical staff to generate predictions consistently on a substantial portion of the patient population, and on the accuracy of the predictions when the risk factors are determined with some level of uncertainty at the beginning of the hospitalization.[2, 8] Skepticism is justified, because the work involved in completing the prediction rule might be incompatible with existing workflow. A patient might not be scored if the emergency physician lacks time or if technical issues arise with the information system and computation process.[9] There is also a generic concern that the predictions will prove to be less accurate outside of the original study population.[8, 9, 10] A more specific concern for our rule is how well present on admission diagnoses can be determined during the relatively short emergency department or presurgery evaluation period. For example, a final diagnosis of heart failure might not be established until later in the hospitalization, after the results of diagnostic testing and clinical response to treatment are known. Moreover, our retrospective prediction rule requires an assessment of the presence or absence of sepsis and respiratory failure. These diagnoses appear to be susceptible to secular trends in medical record coding practices, suggesting the rule's accuracy might not be stable over time.[11]

We report the feasibility of having emergency physicians and the surgical preparation center team generate mortality predictions before an inpatient bed is assigned. We evaluate and report the accuracy of these prospective predictions.

METHODS

The study population consisted of all patients 18 years of age or less than 100 years who were admitted from the emergency department or assigned an inpatient bed following elective surgery at a tertiary, community teaching hospital in the Midwestern United States from September 1, 2012 through February 15, 2013. Although patients entering the hospital from these 2 pathways would be expected to have different levels of mortality risk, we used the original prediction rule for both because such distinctions were not made in its derivation and validation. Patients were not considered if they were admitted for childbirth or other obstetrical reasons, admitted directly from physician offices, the cardiac catheterization laboratory, hemodialysis unit, or from another hospital. The site institutional review board approved this study.

The implementation process began with presentations to the administrative and medical staff leadership on the accuracy of the retrospectively generated mortality predictions and risk of other adverse events.[3] The chief medical and nursing officers became project champions, secured internal funding for the technical components, and arranged to have 2 project comanagers available. A multidisciplinary task force endorsed the implementation details at biweekly meetings throughout the planning year. The leadership of the emergency department and surgical preparation center committed their colleagues to generate the predictions. The support of the emergency leadership was contingent on the completion of the entire prediction generating process in a very short time (within the time a physician could hold his/her breath). The chief medical officer, with the support of the leadership of the hospitalists and emergency physicians, made the administrative decision that a prediction must be generated prior to the assignment of a hospital room.

During the consensus‐building phase, a Web‐based application was developed to generate the predictions. Emergency physicians and surgical preparation staff were trained on the definitions of the risk factors (see Supporting Information, Appendix, in the online version of this article) and how to use the Web application. Three supporting databases were created. Each midnight, a past medical history database was updated, identifying those who had been discharged from the study hospital in the previous 365 days, and whether or not their diagnoses included atrial fibrillation, leukemia/lymphoma, metastatic cancer, cancer other than leukemia, lymphoma, cognitive disorder, or other neurological conditions (eg, Parkinson's, multiple sclerosis, epilepsy, coma, and stupor). Similarly, a clinical laboratory results database was created and updated real time through an HL7 (Health Level Seven, a standard data exchange format[12]) interface with the laboratory information system for the following tests performed in the preceding 30 days at a hospital‐affiliated facility: hemoglobin, platelet count, white blood count, serum troponin, blood urea nitrogen, serum albumin, serum lactate, arterial pH, arterial partial pressure of oxygen values. The third database, admission‐discharge‐transfer, was created and updated every 15 minutes to identify patients currently in the emergency room or scheduled for surgery. When a patient registration event was added to this database, the Web application created a record, retrieved all relevant data, and displayed the patient name for scoring. When the decision for hospitalization was made, the clinician selected the patient's name and reviewed the pre‐populated medical diagnoses of interest, which could be overwritten based on his/her own assessment (Figure 1A,B). The clinician then indicated (yes, no, or unknown) if the patient currently had or was being treated for each of the following: injury, heart failure, sepsis, respiratory failure, and whether or not the admitting service would be medicine (ie, nonsurgical, nonobstetrical). We considered unknown status to indicate the patient did not have the condition. When laboratory values were not available, a normal value was imputed using a previously developed algorithm.[3] Two additional questions, not used in the current prediction process, were answered to provide data for a future analysis: 1 concerning the change in the patient's condition while in the emergency department and the other concerning the presence of abnormal vital signs. The probability of 30‐day mortality was calculated via the Web application using the risk information supplied and the scoring weights (ie, parameter estimates) provided in the Appendices of our original publication.[3] Predictions were updated every minute as new laboratory values became available, and flagged with an alert if a more severe score resulted.

Figure 1

For the analyses of this study, the last prospective prediction viewed by emergency department personnel, a hospital bed manager, or surgical suite staff prior to arrival on the nursing unit is the one referenced as prospective. Once the patient had been discharged from the hospital, we generated a second mortality prediction based on previously published parameter estimates applied to risk factor data ascertained retrospectively as was done in the original article[3]; we subsequently refer to this prediction as retrospective. We will report on the group of patients who had both prospective and retrospective scores (1 patient had a prospective but not retrospective score available).

The prediction scores were made available to the clinical teams gradually during the study period. All scores were viewable by the midpoint of the study for emergency department admissions and near the end of the study for elective‐surgery patients. Only 2 changes in care processes based on level of risk were introduced during the study period. The first required initial placement of patients having a probability of dying of 0.3 or greater into an intensive or intermediate care unit unless the patient or family requested a less aggressive approach. The second occurred in the final 2 months of the study when a large multispecialty practice began routinely arranging for high‐risk patients to be seen within 3 or 7 days of hospital discharge.

RESULTS

Mortality predictions were generated on demand for 7291 out of 7777 (93.8%) eligible patients admitted from the emergency department, and for 2021 out of 2250 (89.8%) eligible elective surgical cases, for a total of 9312 predictions generated out of a possible 10,027 hospitalizations (92.9%). Table 1 displays the characteristics of the study population. The mean age was 65.2 years and 53.8% were women. The most common risk factors were atrial fibrillation (16.4%) and cancer (14.6%). Orders for a comfort care approach (rather than curative) were entered within 4 hours of admission for 32/9312 patients (0.34%), and 9/9312 (0.1%) were hospice patients on admission.

Evaluation of Prediction Accuracy

The AROC for 30‐day mortality was 0.850 (95% confidence interval [CI]: 0.833‐0.866) for prospectively collected covariates, and 0.870 (95% CI: 0.855‐0.885) for retrospectively determined risk factors. These AROCs are not substantively different from each other, demonstrating comparable prediction performance. Calibration was excellent, as indicated in Figure 2, in which the predicted level of risk lay within the 95% confidence limits of the actual 30‐day mortality for 19 out of 20 intervals of 5 percentile increments.

Figure 2

DISCUSSION

Emergency physicians and surgical preparation center nurses generated predictions by the time of hospital admission for over 90% of the target population during usual workflow, without the addition of staff or resources. The discrimination of the prospectively generated predictions was very good to excellent, with an AROC of 0.850 (95% CI: 0.833‐0.866), similar to that obtained from the retrospective version. Calibration was excellent. The prospectively calculated mortality risk was associated with a number of other events. As shown in Table 3, the differing frequency of events within the risk strata support the development of differing intensities of multidisciplinary strategies according to the level of risk.[5] Our study provides useful experience for others who anticipate generating real‐time predictions. We consider the key reasons for success to be the considerable time spent achieving consensus, the technical development of the Web application, the brief clinician time required for the scoring process, the leadership of the chief medical and nursing officers, and the requirement that a prediction be generated before assignment of a hospital room.

Our study has a number of limitations, some of which were noted in our original publication, and although still relevant, will not be repeated here for space considerations. This is a single‐site study that used a prediction rule developed by the same site, albeit on a patient population 4 to 5 years earlier. It is not known how well the specific rule might perform in other hospital populations; any such use should therefore be accompanied by independent validation studies prior to implementation. Our successful experience should motivate future validation studies. Second, because the prognoses of patients scored from the emergency department are likely to be worse than those of elective surgery patients, our rule should be recalibrated for each subgroup separately. We plan to do this in the near future, as well as consider additional risk factors. Third, the other events of interest might be predicted more accurately if rules specifically developed for each were deployed. The mortality risk by itself is unlikely to be a sufficiently accurate predictor, particularly for complications and intensive care use, for reasons outlined in our original publication.[3] However, the varying levels of events within the higher versus lower strata should be noted by the clinical team as they design their team‐based processes. A follow‐up visit with a physician within a few days of discharge could address the concurrent risk of dying as well as readmission, for example. Finally, it is too early to determine if the availability of mortality predictions from this rule will benefit patients.[2, 8, 10] During the study period, we implemented only 2 new care processes based on the level of risk. This lack of interventions allowed us to evaluate the prediction accuracy with minimal additional confounding, but at the expense of not yet knowing the clinical impact of this work. After the study period, we implemented a number of other interventions and plan on evaluating their effectiveness in the future. We are also considering an evaluation of the potential information gained by updating the predictions throughout the course of the hospitalization.[14]

In conclusion, it is feasible to have a reasonably accurate prediction of mortality risk for most adult patients at the beginning of their hospitalizations. The availability of this prognostic information provides an opportunity to develop proactive care plans for high‐ and low‐risk subsets of patients.

Favorable health outcomes are more likely to occur when the healthcare team quickly identifies and responds to patients at risk.[1, 2, 3] However, the treatment process can break down during handoffs if the clinical condition and active issues are not well communicated.[4] Patients whose decline cannot be reversed also challenge the health team. Many are referred to hospice late,[5] and some do not receive the type of end‐of‐life care matching their preferences.[6]

Progress toward the elusive goal of more effective and efficient care might be made via an industrial engineering approach, mass customization, in which bundles of services are delivered based on the anticipated needs of subsets of patients.[7, 8] An underlying rationale is the frequent finding that a small proportion of individuals experiences the majority of the events of interest, commonly referenced as the Pareto principle.[7] Clinical prediction rules can help identify these high‐risk subsets.[9] However, as more condition‐specific rules become available, the clinical team faces logistical challenges when attempting to incorporate these into practice. For example, which team member will be responsible for generating the prediction and communicating the level of risk? What actions should follow for a given level of risk? What should be done for patients with conditions not addressed by an existing rule?

In this study, we present our rationale for health systems to implement a process for generating mortality predictions at the time of admission on most, if not all, adult patients as a context for the activities of the various clinical team members. Recent studies demonstrate that in‐hospital or 30‐day mortality can be predicted with substantial accuracy using information available at the time of admission.[10, 11, 12, 13, 14, 15, 16, 17, 18, 19] Relationships are beginning to be explored among the risk factors for mortality and other outcomes such as length of stay, unplanned transfers to intensive care units, 30‐day readmissions, and extended care facility placement.[10, 20, 21, 22] We extend this work by examining how a number of adverse events can be understood through their relationship with the risk of dying. We begin by deriving and validating a new mortality prediction rule using information feasible for our institution to use in its implementation.

METHODS

The prediction rule was derived from data on all inpatients (n = 56,003) 18 to 99 years old from St. Joseph Mercy Hospital, Ann Arbor from 2008 to 2009. This is a community‐based, tertiary‐care center. We reference derivation cases as D1, validation cases from the same hospital in the following year (2010) as V1, and data from a second hospital in 2010 as V2. The V2 hospital belonged to the same parent health corporation and shared some physician specialists with D1 and V1 but had separate medical and nursing staff.

The primary outcome predicted is 30‐day mortality from the time of admission. We chose 30‐day rather than in‐hospital mortality to address concerns of potential confounding of duration of hospital stay and likelihood of dying in the hospital.[23] Risk factors were considered for inclusion into the prediction rule based on their prevalence, conceptual, and univariable association with death (details provided in the Supporting information, Appendix I and II, in the online version of this article). The types of risk factors considered were patient diagnoses as of the time of admission obtained from hospital administrative data and grouped by the 2011 Clinical Classification Software (http://www.hcupus.ahrq.gov/toolssoftware/ccs/ccs.jsp#download, accessed June 6, 2012), administrative data from previous hospitalizations within the health system in the preceding 12 months, and the worst value of clinical laboratory blood tests obtained within 30 days prior to the time of admission. When a given patient had missing values for the laboratory tests of interest, we imputed a normal value, assuming the clinician had not ordered these tests because he/she expected the patient would have normal results. The imputed normal values were derived from available results from patients discharged alive with short hospital stays (3 days) in 2007 to 2008. The datasets were built and analyzed using SAS version 9.1, 9.2 (SAS Institute, Inc., Cary, NC) and R (R Foundation for Statistical Computing, Vienna, Austria; http://www.R‐project.org).

RESULTS

Table 1 displays the risk factors used in the 30‐day mortality prediction rule, their distribution in the populations of interest, and the frequency of the outcomes of interest. The derivation (D1) and validation (V1) populations were clinically similar; the patients of hospital V2 differed in the proportion of risk factors and outcomes. The scoring weights or parameter estimates for the risk factors are given in the Appendix (see Supporting Information, Appendix I, in the online version of this article).

Predicting 30‐Day Mortality

The areas under the ROC (95% confidence interval [CI]) for the D1, V1, and V2 populations were 0.876 (95% CI, 0.870‐0.882), 0.885 (95% CI, 0.877‐0.893), and 0.883 (95% CI, 0.875‐0.892), respectively. The calibration curves for all 3 populations are shown in Figure 1. The overlap of symbols indicates that the level of predicted risk matched actual mortality for most intervals, with slight underprediction for those in the highest risk percentiles.

Figure 1

Example of Risk Strata

Figure 2 displays the relationship between the predicted probability of dying within 30 days and the outcomes of interest for V1, and illustrates the Pareto principle for defining high‐ and low‐risk subgroups. Most of the 30‐day deaths (74.7% of D1, 74.2% of V1, and 85.3% of V2) occurred in the small subset of patients with a predicted probability of death exceeding 0.067 (the top quintile of risk of D1, the top 18 % of V1, and the top 29.8% of V2). In contrast, the mortality rate for those with a predicted risk of 0.0033 was 0.02% for the lowest quintile of risk in D1, 0.07% for the 19.3% having the lowest risk in V1, and 0% for the 9.7% of patients with the lowest risk in V2. Figure 3 indicates that the risk for dying peaks within the first few days of the hospitalization. Moreover, those in the high‐risk group remained at elevated risk relative to the lower risk strata for at least 100 days.

Figure 2

Figure 3

DISCUSSION

The primary contribution of our work concerns the number and strength of associations between the probability of dying within 30 days and other events, and the implications for organizing the healthcare delivery model. We also add to the growing evidence that death within 30 days can be accurately predicted at the time of admission from demographic information, modest levels of diagnostic information, and clinical laboratory values. We developed a new prediction rule with excellent accuracy that compares well to a rule recently developed by the Kaiser Permanente system.[13, 14] Feasibility considerations are likely to be the ultimate determinant of which prediction rule a health system chooses.[13, 14, 29] An independent evaluation of the candidate rules applied to the same data is required to compare their accuracy.

These results suggest a context for the coordination of clinical care processes, although mortality risk is not the only domain health systems must address. For illustrative purposes, we will refer to the risk strata shown in Figure 2. After the decisions to admit the patient to the hospital and whether or not surgical intervention is needed, the next decision concerns the level and type of nursing care needed.[10] Recent studies continue to show challenges both with unplanned transfers to intensive care units[21] and care delivered that is consistently concordant with patient wishes.[6, 30] The level of risk for multiple adverse outcomes suggests stratum 1 patients would be the priority group for perfecting the placement and preference assessment process. Our institution is currently piloting an internal placement guideline recommending that nonpalliative patients in the top 2.5 percentile of mortality risk be placed initially in either an intensive or intermediate care unit to receive the potential benefit of higher nursing staffing levels.[31] However, mortality risk cannot be the only criterion used for placement, as demonstrated by its relatively weak association with overall ICU utilization. Our findings may reflect the role of unmeasured factors such as the need for mechanical ventilation, patient preference for comfort care, bed availability, change in patient condition after admission, and inconsistent application of admission criteria.[17, 21, 32, 33, 34]

After the placement decision, the team could decide if the usual level of monitoring, physician rounding, and care coordination would be adequate for the level of risk or whether an additional anticipatory approach is needed. The weak relationship between the risk of death and incidence of complications, although not a new finding,[35, 36] suggests routine surveillance activities need to be conducted on all patients regardless of risk to detect a complication, but that a rescue plan be developed in advance for high mortality risk patients, for example strata 1 and 2, in the event they should develop a complication.[36] Inclusion of the patient's risk strata as part of the routine hand‐off communication among hospitalists, nurses, and other team members could provide a succinct common alert for the likelihood of adverse events.

The 30‐day mortality risk also informs the transition care plan following hospitalization, given the strong association with death in 180 days and the persistent level of this risk (Figure 3). Again, communication of the risk status (stratum 1) to the team caring for the patient after the hospitalization provides a common reference for prognosis and level of attention needed. However, the prediction accuracy is not sufficient to refer high‐risk patients into hospice, but rather, to identify the high‐risk subset having the most urgent need to have their preferences for future end‐of‐life care understood and addressed. The weak relationship of mortality risk with 30‐day readmissions indicates that our rule would have a limited role in identifying readmission risk per se. Others have noted the difficulty in accurately predicting readmissions, most likely because the underlying causes are multifactorial.[37] Our results suggest that 1 dynamic for readmission is the risk of dying, and so the underlying causes of this risk should be addressed in the transition plan.

There are a number of limitations with our study. First, this rule was developed and validated on data from only 2 institutions, assembled retrospectively, with diagnostic information determined from administrative data. One cannot assume the accuracy will carry over to other institutions[29] or when there is diagnostic uncertainty at the time of admission. Second, the 30‐day mortality risk should not be used as the sole criterion for determining the service intensity for individual patients because of issues with calibration, interpretation of risk, and confounding. The calibration curves (Figure 2) show the slight underprediction of the risk of dying for high‐risk groups. Other studies have also noted problems with precise calibration in validation datasets.[13, 14] Caution is also needed in the interpretation of what it means to be at high risk. Most patients in stratum 1 were alive at 30 days; therefore, being at high risk is not a death sentence. Furthermore, the relative weights of the risk factors reflect (ie, are confounded by) the level of treatment rendered. Some deaths within the higher‐risk percentiles undoubtedly occurred in patients choosing a palliative rather than a curative approach, perhaps partially explaining the slight underprediction of deaths. Conversely, the low mortality experienced by patients within the lower‐risk strata may indicate the treatment provided was effective. Low mortality risk does not imply less care is needed.

A third limitation is that we have not defined the thresholds of risk that should trigger placement and care intensity, although we provide examples on how this could be done. Each institution will need to calibrate the thresholds and associated decision‐making processes according to its own environment.[14] Interested readers can explore the sensitivity and specificity of various thresholds\ by using the tables in the Appendix (see the Supporting information, Appendix II, in the online version of this article). Finally, we do not know if identifying the mortality risk on admission will lead to better outcomes[19, 29]

CONCLUSIONS

Death within 30 days can be predicted with information known at the time of admission, and is associated with the risk of having other adverse events. We believe the probability of death can be used to define strata of risk that provide a succinct common reference point for the multidisciplinary team to anticipate the clinical course of subsets of patients and intervene with proportional intensity.

Favorable health outcomes are more likely to occur when the healthcare team quickly identifies and responds to patients at risk.[1, 2, 3] However, the treatment process can break down during handoffs if the clinical condition and active issues are not well communicated.[4] Patients whose decline cannot be reversed also challenge the health team. Many are referred to hospice late,[5] and some do not receive the type of end‐of‐life care matching their preferences.[6]

Progress toward the elusive goal of more effective and efficient care might be made via an industrial engineering approach, mass customization, in which bundles of services are delivered based on the anticipated needs of subsets of patients.[7, 8] An underlying rationale is the frequent finding that a small proportion of individuals experiences the majority of the events of interest, commonly referenced as the Pareto principle.[7] Clinical prediction rules can help identify these high‐risk subsets.[9] However, as more condition‐specific rules become available, the clinical team faces logistical challenges when attempting to incorporate these into practice. For example, which team member will be responsible for generating the prediction and communicating the level of risk? What actions should follow for a given level of risk? What should be done for patients with conditions not addressed by an existing rule?

In this study, we present our rationale for health systems to implement a process for generating mortality predictions at the time of admission on most, if not all, adult patients as a context for the activities of the various clinical team members. Recent studies demonstrate that in‐hospital or 30‐day mortality can be predicted with substantial accuracy using information available at the time of admission.[10, 11, 12, 13, 14, 15, 16, 17, 18, 19] Relationships are beginning to be explored among the risk factors for mortality and other outcomes such as length of stay, unplanned transfers to intensive care units, 30‐day readmissions, and extended care facility placement.[10, 20, 21, 22] We extend this work by examining how a number of adverse events can be understood through their relationship with the risk of dying. We begin by deriving and validating a new mortality prediction rule using information feasible for our institution to use in its implementation.

METHODS

The prediction rule was derived from data on all inpatients (n = 56,003) 18 to 99 years old from St. Joseph Mercy Hospital, Ann Arbor from 2008 to 2009. This is a community‐based, tertiary‐care center. We reference derivation cases as D1, validation cases from the same hospital in the following year (2010) as V1, and data from a second hospital in 2010 as V2. The V2 hospital belonged to the same parent health corporation and shared some physician specialists with D1 and V1 but had separate medical and nursing staff.

The primary outcome predicted is 30‐day mortality from the time of admission. We chose 30‐day rather than in‐hospital mortality to address concerns of potential confounding of duration of hospital stay and likelihood of dying in the hospital.[23] Risk factors were considered for inclusion into the prediction rule based on their prevalence, conceptual, and univariable association with death (details provided in the Supporting information, Appendix I and II, in the online version of this article). The types of risk factors considered were patient diagnoses as of the time of admission obtained from hospital administrative data and grouped by the 2011 Clinical Classification Software (http://www.hcupus.ahrq.gov/toolssoftware/ccs/ccs.jsp#download, accessed June 6, 2012), administrative data from previous hospitalizations within the health system in the preceding 12 months, and the worst value of clinical laboratory blood tests obtained within 30 days prior to the time of admission. When a given patient had missing values for the laboratory tests of interest, we imputed a normal value, assuming the clinician had not ordered these tests because he/she expected the patient would have normal results. The imputed normal values were derived from available results from patients discharged alive with short hospital stays (3 days) in 2007 to 2008. The datasets were built and analyzed using SAS version 9.1, 9.2 (SAS Institute, Inc., Cary, NC) and R (R Foundation for Statistical Computing, Vienna, Austria; http://www.R‐project.org).

RESULTS

Table 1 displays the risk factors used in the 30‐day mortality prediction rule, their distribution in the populations of interest, and the frequency of the outcomes of interest. The derivation (D1) and validation (V1) populations were clinically similar; the patients of hospital V2 differed in the proportion of risk factors and outcomes. The scoring weights or parameter estimates for the risk factors are given in the Appendix (see Supporting Information, Appendix I, in the online version of this article).

Predicting 30‐Day Mortality

The areas under the ROC (95% confidence interval [CI]) for the D1, V1, and V2 populations were 0.876 (95% CI, 0.870‐0.882), 0.885 (95% CI, 0.877‐0.893), and 0.883 (95% CI, 0.875‐0.892), respectively. The calibration curves for all 3 populations are shown in Figure 1. The overlap of symbols indicates that the level of predicted risk matched actual mortality for most intervals, with slight underprediction for those in the highest risk percentiles.

Figure 1

Example of Risk Strata

Figure 2 displays the relationship between the predicted probability of dying within 30 days and the outcomes of interest for V1, and illustrates the Pareto principle for defining high‐ and low‐risk subgroups. Most of the 30‐day deaths (74.7% of D1, 74.2% of V1, and 85.3% of V2) occurred in the small subset of patients with a predicted probability of death exceeding 0.067 (the top quintile of risk of D1, the top 18 % of V1, and the top 29.8% of V2). In contrast, the mortality rate for those with a predicted risk of 0.0033 was 0.02% for the lowest quintile of risk in D1, 0.07% for the 19.3% having the lowest risk in V1, and 0% for the 9.7% of patients with the lowest risk in V2. Figure 3 indicates that the risk for dying peaks within the first few days of the hospitalization. Moreover, those in the high‐risk group remained at elevated risk relative to the lower risk strata for at least 100 days.

Figure 2

Figure 3

DISCUSSION

The primary contribution of our work concerns the number and strength of associations between the probability of dying within 30 days and other events, and the implications for organizing the healthcare delivery model. We also add to the growing evidence that death within 30 days can be accurately predicted at the time of admission from demographic information, modest levels of diagnostic information, and clinical laboratory values. We developed a new prediction rule with excellent accuracy that compares well to a rule recently developed by the Kaiser Permanente system.[13, 14] Feasibility considerations are likely to be the ultimate determinant of which prediction rule a health system chooses.[13, 14, 29] An independent evaluation of the candidate rules applied to the same data is required to compare their accuracy.

These results suggest a context for the coordination of clinical care processes, although mortality risk is not the only domain health systems must address. For illustrative purposes, we will refer to the risk strata shown in Figure 2. After the decisions to admit the patient to the hospital and whether or not surgical intervention is needed, the next decision concerns the level and type of nursing care needed.[10] Recent studies continue to show challenges both with unplanned transfers to intensive care units[21] and care delivered that is consistently concordant with patient wishes.[6, 30] The level of risk for multiple adverse outcomes suggests stratum 1 patients would be the priority group for perfecting the placement and preference assessment process. Our institution is currently piloting an internal placement guideline recommending that nonpalliative patients in the top 2.5 percentile of mortality risk be placed initially in either an intensive or intermediate care unit to receive the potential benefit of higher nursing staffing levels.[31] However, mortality risk cannot be the only criterion used for placement, as demonstrated by its relatively weak association with overall ICU utilization. Our findings may reflect the role of unmeasured factors such as the need for mechanical ventilation, patient preference for comfort care, bed availability, change in patient condition after admission, and inconsistent application of admission criteria.[17, 21, 32, 33, 34]

After the placement decision, the team could decide if the usual level of monitoring, physician rounding, and care coordination would be adequate for the level of risk or whether an additional anticipatory approach is needed. The weak relationship between the risk of death and incidence of complications, although not a new finding,[35, 36] suggests routine surveillance activities need to be conducted on all patients regardless of risk to detect a complication, but that a rescue plan be developed in advance for high mortality risk patients, for example strata 1 and 2, in the event they should develop a complication.[36] Inclusion of the patient's risk strata as part of the routine hand‐off communication among hospitalists, nurses, and other team members could provide a succinct common alert for the likelihood of adverse events.

The 30‐day mortality risk also informs the transition care plan following hospitalization, given the strong association with death in 180 days and the persistent level of this risk (Figure 3). Again, communication of the risk status (stratum 1) to the team caring for the patient after the hospitalization provides a common reference for prognosis and level of attention needed. However, the prediction accuracy is not sufficient to refer high‐risk patients into hospice, but rather, to identify the high‐risk subset having the most urgent need to have their preferences for future end‐of‐life care understood and addressed. The weak relationship of mortality risk with 30‐day readmissions indicates that our rule would have a limited role in identifying readmission risk per se. Others have noted the difficulty in accurately predicting readmissions, most likely because the underlying causes are multifactorial.[37] Our results suggest that 1 dynamic for readmission is the risk of dying, and so the underlying causes of this risk should be addressed in the transition plan.

There are a number of limitations with our study. First, this rule was developed and validated on data from only 2 institutions, assembled retrospectively, with diagnostic information determined from administrative data. One cannot assume the accuracy will carry over to other institutions[29] or when there is diagnostic uncertainty at the time of admission. Second, the 30‐day mortality risk should not be used as the sole criterion for determining the service intensity for individual patients because of issues with calibration, interpretation of risk, and confounding. The calibration curves (Figure 2) show the slight underprediction of the risk of dying for high‐risk groups. Other studies have also noted problems with precise calibration in validation datasets.[13, 14] Caution is also needed in the interpretation of what it means to be at high risk. Most patients in stratum 1 were alive at 30 days; therefore, being at high risk is not a death sentence. Furthermore, the relative weights of the risk factors reflect (ie, are confounded by) the level of treatment rendered. Some deaths within the higher‐risk percentiles undoubtedly occurred in patients choosing a palliative rather than a curative approach, perhaps partially explaining the slight underprediction of deaths. Conversely, the low mortality experienced by patients within the lower‐risk strata may indicate the treatment provided was effective. Low mortality risk does not imply less care is needed.

A third limitation is that we have not defined the thresholds of risk that should trigger placement and care intensity, although we provide examples on how this could be done. Each institution will need to calibrate the thresholds and associated decision‐making processes according to its own environment.[14] Interested readers can explore the sensitivity and specificity of various thresholds\ by using the tables in the Appendix (see the Supporting information, Appendix II, in the online version of this article). Finally, we do not know if identifying the mortality risk on admission will lead to better outcomes[19, 29]

CONCLUSIONS

Death within 30 days can be predicted with information known at the time of admission, and is associated with the risk of having other adverse events. We believe the probability of death can be used to define strata of risk that provide a succinct common reference point for the multidisciplinary team to anticipate the clinical course of subsets of patients and intervene with proportional intensity.

Favorable health outcomes are more likely to occur when the healthcare team quickly identifies and responds to patients at risk.[1, 2, 3] However, the treatment process can break down during handoffs if the clinical condition and active issues are not well communicated.[4] Patients whose decline cannot be reversed also challenge the health team. Many are referred to hospice late,[5] and some do not receive the type of end‐of‐life care matching their preferences.[6]

Progress toward the elusive goal of more effective and efficient care might be made via an industrial engineering approach, mass customization, in which bundles of services are delivered based on the anticipated needs of subsets of patients.[7, 8] An underlying rationale is the frequent finding that a small proportion of individuals experiences the majority of the events of interest, commonly referenced as the Pareto principle.[7] Clinical prediction rules can help identify these high‐risk subsets.[9] However, as more condition‐specific rules become available, the clinical team faces logistical challenges when attempting to incorporate these into practice. For example, which team member will be responsible for generating the prediction and communicating the level of risk? What actions should follow for a given level of risk? What should be done for patients with conditions not addressed by an existing rule?

In this study, we present our rationale for health systems to implement a process for generating mortality predictions at the time of admission on most, if not all, adult patients as a context for the activities of the various clinical team members. Recent studies demonstrate that in‐hospital or 30‐day mortality can be predicted with substantial accuracy using information available at the time of admission.[10, 11, 12, 13, 14, 15, 16, 17, 18, 19] Relationships are beginning to be explored among the risk factors for mortality and other outcomes such as length of stay, unplanned transfers to intensive care units, 30‐day readmissions, and extended care facility placement.[10, 20, 21, 22] We extend this work by examining how a number of adverse events can be understood through their relationship with the risk of dying. We begin by deriving and validating a new mortality prediction rule using information feasible for our institution to use in its implementation.

METHODS

The prediction rule was derived from data on all inpatients (n = 56,003) 18 to 99 years old from St. Joseph Mercy Hospital, Ann Arbor from 2008 to 2009. This is a community‐based, tertiary‐care center. We reference derivation cases as D1, validation cases from the same hospital in the following year (2010) as V1, and data from a second hospital in 2010 as V2. The V2 hospital belonged to the same parent health corporation and shared some physician specialists with D1 and V1 but had separate medical and nursing staff.

The primary outcome predicted is 30‐day mortality from the time of admission. We chose 30‐day rather than in‐hospital mortality to address concerns of potential confounding of duration of hospital stay and likelihood of dying in the hospital.[23] Risk factors were considered for inclusion into the prediction rule based on their prevalence, conceptual, and univariable association with death (details provided in the Supporting information, Appendix I and II, in the online version of this article). The types of risk factors considered were patient diagnoses as of the time of admission obtained from hospital administrative data and grouped by the 2011 Clinical Classification Software (http://www.hcupus.ahrq.gov/toolssoftware/ccs/ccs.jsp#download, accessed June 6, 2012), administrative data from previous hospitalizations within the health system in the preceding 12 months, and the worst value of clinical laboratory blood tests obtained within 30 days prior to the time of admission. When a given patient had missing values for the laboratory tests of interest, we imputed a normal value, assuming the clinician had not ordered these tests because he/she expected the patient would have normal results. The imputed normal values were derived from available results from patients discharged alive with short hospital stays (3 days) in 2007 to 2008. The datasets were built and analyzed using SAS version 9.1, 9.2 (SAS Institute, Inc., Cary, NC) and R (R Foundation for Statistical Computing, Vienna, Austria; http://www.R‐project.org).

RESULTS

Table 1 displays the risk factors used in the 30‐day mortality prediction rule, their distribution in the populations of interest, and the frequency of the outcomes of interest. The derivation (D1) and validation (V1) populations were clinically similar; the patients of hospital V2 differed in the proportion of risk factors and outcomes. The scoring weights or parameter estimates for the risk factors are given in the Appendix (see Supporting Information, Appendix I, in the online version of this article).

Predicting 30‐Day Mortality

The areas under the ROC (95% confidence interval [CI]) for the D1, V1, and V2 populations were 0.876 (95% CI, 0.870‐0.882), 0.885 (95% CI, 0.877‐0.893), and 0.883 (95% CI, 0.875‐0.892), respectively. The calibration curves for all 3 populations are shown in Figure 1. The overlap of symbols indicates that the level of predicted risk matched actual mortality for most intervals, with slight underprediction for those in the highest risk percentiles.

Figure 1

Example of Risk Strata

Figure 2 displays the relationship between the predicted probability of dying within 30 days and the outcomes of interest for V1, and illustrates the Pareto principle for defining high‐ and low‐risk subgroups. Most of the 30‐day deaths (74.7% of D1, 74.2% of V1, and 85.3% of V2) occurred in the small subset of patients with a predicted probability of death exceeding 0.067 (the top quintile of risk of D1, the top 18 % of V1, and the top 29.8% of V2). In contrast, the mortality rate for those with a predicted risk of 0.0033 was 0.02% for the lowest quintile of risk in D1, 0.07% for the 19.3% having the lowest risk in V1, and 0% for the 9.7% of patients with the lowest risk in V2. Figure 3 indicates that the risk for dying peaks within the first few days of the hospitalization. Moreover, those in the high‐risk group remained at elevated risk relative to the lower risk strata for at least 100 days.

Figure 2

Figure 3

DISCUSSION

The primary contribution of our work concerns the number and strength of associations between the probability of dying within 30 days and other events, and the implications for organizing the healthcare delivery model. We also add to the growing evidence that death within 30 days can be accurately predicted at the time of admission from demographic information, modest levels of diagnostic information, and clinical laboratory values. We developed a new prediction rule with excellent accuracy that compares well to a rule recently developed by the Kaiser Permanente system.[13, 14] Feasibility considerations are likely to be the ultimate determinant of which prediction rule a health system chooses.[13, 14, 29] An independent evaluation of the candidate rules applied to the same data is required to compare their accuracy.

These results suggest a context for the coordination of clinical care processes, although mortality risk is not the only domain health systems must address. For illustrative purposes, we will refer to the risk strata shown in Figure 2. After the decisions to admit the patient to the hospital and whether or not surgical intervention is needed, the next decision concerns the level and type of nursing care needed.[10] Recent studies continue to show challenges both with unplanned transfers to intensive care units[21] and care delivered that is consistently concordant with patient wishes.[6, 30] The level of risk for multiple adverse outcomes suggests stratum 1 patients would be the priority group for perfecting the placement and preference assessment process. Our institution is currently piloting an internal placement guideline recommending that nonpalliative patients in the top 2.5 percentile of mortality risk be placed initially in either an intensive or intermediate care unit to receive the potential benefit of higher nursing staffing levels.[31] However, mortality risk cannot be the only criterion used for placement, as demonstrated by its relatively weak association with overall ICU utilization. Our findings may reflect the role of unmeasured factors such as the need for mechanical ventilation, patient preference for comfort care, bed availability, change in patient condition after admission, and inconsistent application of admission criteria.[17, 21, 32, 33, 34]

After the placement decision, the team could decide if the usual level of monitoring, physician rounding, and care coordination would be adequate for the level of risk or whether an additional anticipatory approach is needed. The weak relationship between the risk of death and incidence of complications, although not a new finding,[35, 36] suggests routine surveillance activities need to be conducted on all patients regardless of risk to detect a complication, but that a rescue plan be developed in advance for high mortality risk patients, for example strata 1 and 2, in the event they should develop a complication.[36] Inclusion of the patient's risk strata as part of the routine hand‐off communication among hospitalists, nurses, and other team members could provide a succinct common alert for the likelihood of adverse events.

The 30‐day mortality risk also informs the transition care plan following hospitalization, given the strong association with death in 180 days and the persistent level of this risk (Figure 3). Again, communication of the risk status (stratum 1) to the team caring for the patient after the hospitalization provides a common reference for prognosis and level of attention needed. However, the prediction accuracy is not sufficient to refer high‐risk patients into hospice, but rather, to identify the high‐risk subset having the most urgent need to have their preferences for future end‐of‐life care understood and addressed. The weak relationship of mortality risk with 30‐day readmissions indicates that our rule would have a limited role in identifying readmission risk per se. Others have noted the difficulty in accurately predicting readmissions, most likely because the underlying causes are multifactorial.[37] Our results suggest that 1 dynamic for readmission is the risk of dying, and so the underlying causes of this risk should be addressed in the transition plan.

There are a number of limitations with our study. First, this rule was developed and validated on data from only 2 institutions, assembled retrospectively, with diagnostic information determined from administrative data. One cannot assume the accuracy will carry over to other institutions[29] or when there is diagnostic uncertainty at the time of admission. Second, the 30‐day mortality risk should not be used as the sole criterion for determining the service intensity for individual patients because of issues with calibration, interpretation of risk, and confounding. The calibration curves (Figure 2) show the slight underprediction of the risk of dying for high‐risk groups. Other studies have also noted problems with precise calibration in validation datasets.[13, 14] Caution is also needed in the interpretation of what it means to be at high risk. Most patients in stratum 1 were alive at 30 days; therefore, being at high risk is not a death sentence. Furthermore, the relative weights of the risk factors reflect (ie, are confounded by) the level of treatment rendered. Some deaths within the higher‐risk percentiles undoubtedly occurred in patients choosing a palliative rather than a curative approach, perhaps partially explaining the slight underprediction of deaths. Conversely, the low mortality experienced by patients within the lower‐risk strata may indicate the treatment provided was effective. Low mortality risk does not imply less care is needed.

A third limitation is that we have not defined the thresholds of risk that should trigger placement and care intensity, although we provide examples on how this could be done. Each institution will need to calibrate the thresholds and associated decision‐making processes according to its own environment.[14] Interested readers can explore the sensitivity and specificity of various thresholds\ by using the tables in the Appendix (see the Supporting information, Appendix II, in the online version of this article). Finally, we do not know if identifying the mortality risk on admission will lead to better outcomes[19, 29]

CONCLUSIONS

Death within 30 days can be predicted with information known at the time of admission, and is associated with the risk of having other adverse events. We believe the probability of death can be used to define strata of risk that provide a succinct common reference point for the multidisciplinary team to anticipate the clinical course of subsets of patients and intervene with proportional intensity.