The Centers for Medicare & Medicaid Services (CMS) publicly reports hospital‐specific, 30‐day risk‐standardized mortality and readmission rates for Medicare fee‐for‐service patients admitted with acute myocardial infarction (AMI), heart failure (HF), and pneumonia.1 These measures are intended to reflect hospital performance on quality of care provided to patients during and after hospitalization.2, 3

Quality‐of‐care measures for a given disease are often assumed to reflect the quality of care for that particular condition. However, studies have found limited association between condition‐specific process measures and either mortality or readmission rates for those conditions.46 Mortality and readmission rates may instead reflect broader hospital‐wide or specialty‐wide structure, culture, and practice. For example, studies have previously found that hospitals differ in mortality or readmission rates according to organizational structure,7 financial structure,8 culture,9, 10 information technology,11 patient volume,1214 academic status,12 and other institution‐wide factors.12 There is now a strong policy push towards developing hospital‐wide (all‐condition) measures, beginning with readmission.15

It is not clear how much of the quality of care for a given condition is attributable to hospital‐wide influences that affect all conditions rather than disease‐specific factors. If readmission or mortality performance for a particular condition reflects, in large part, broader institutional characteristics, then improvement efforts might better be focused on hospital‐wide activities, such as team training or implementing electronic medical records. On the other hand, if the disease‐specific measures reflect quality strictly for those conditions, then improvement efforts would be better focused on disease‐specific care, such as early identification of the relevant patient population or standardizing disease‐specific care. As hospitals work to improve performance across an increasingly wide variety of conditions, it is becoming more important for hospitals to prioritize and focus their activities effectively and efficiently.

One means of determining the relative contribution of hospital versus disease factors is to explore whether outcome rates are consistent among different conditions cared for in the same hospital. If mortality (or readmission) rates across different conditions are highly correlated, it would suggest that hospital‐wide factors may play a substantive role in outcomes. Some studies have found that mortality for a particular surgical condition is a useful proxy for mortality for other surgical conditions,16, 17 while other studies have found little correlation among mortality rates for various medical conditions.18, 19 It is also possible that correlation varies according to hospital characteristics; for example, smaller or nonteaching hospitals might be more homogenous in their care than larger, less homogeneous institutions. No studies have been performed using publicly reported estimates of risk‐standardized mortality or readmission rates. In this study we use the publicly reported measures of 30‐day mortality and 30‐day readmission for AMI, HF, and pneumonia to examine whether, and to what degree, mortality rates track together within US hospitals, and separately, to what degree readmission rates track together within US hospitals.

METHODS

RESULTS

The mortality cohort included 4559 hospitals, and the readmission cohort included 4468 hospitals. The majority of hospitals was small, nonteaching, and did not have advanced cardiac capabilities such as cardiac surgery or cardiac catheterization (Table 1).

For mortality measures, the smallest median number of cases per hospital was for AMI (48; interquartile range [IQR], 13,171), and the greatest number was for pneumonia (178; IQR, 87, 336). The same pattern held for readmission measures (AMI median 33; IQR; 9, 150; pneumonia median 191; IQR, 95, 352.5). With respect to mortality measures, AMI had the highest rate and HF the lowest rate; however, for readmission measures, HF had the highest rate and pneumonia the lowest rate (Table 2).

Every mortality measure was significantly correlated with every other mortality measure (range of correlation coefficients, 0.270.41, P < 0.0001 for all 3 correlations). For example, the correlation between risk‐standardized mortality rates (RSMR) for HF and pneumonia was 0.41. Similarly, every readmission measure was significantly correlated with every other readmission measure (range of correlation coefficients, 0.320.47; P < 0.0001 for all 3 correlations). Overall, the lowest correlation was between risk‐standardized mortality rates for AMI and pneumonia (r = 0.27), and the highest correlation was between risk‐standardized readmission rates (RSRR) for HF and pneumonia (r = 0.47) (Table 3).

Both the factor analysis for the mortality measures and the factor analysis for the readmission measures yielded only one factor with an eigenvalue >1. In each factor analysis, this single common factor kept more than half of the data based on the cumulative eigenvalue (55% for mortality measures and 60% for readmission measures). For the mortality measures, the pattern of RSMR for myocardial infarction (MI), heart failure (HF), and pneumonia (PN) in the factor was high (0.68 for MI, 0.78 for HF, and 0.76 for PN); the same was true of the RSRR in the readmission measures (0.72 for MI, 0.81 for HF, and 0.78 for PN).

For all condition pairs and both outcomes, a third or more of hospitals were in the same quartile of performance for both conditions of the pair (Table 4). Hospitals were more likely to be in the same quartile of performance if they were in the top or bottom quartile than if they were in the middle. Less than 10% of hospitals were in the top quartile for one condition in the mortality or readmission pair and in the bottom quartile for the other condition in the pair. Kappa scores for same quartile of performance between pairs of outcomes ranged from 0.16 to 0.27, and were highest for HF and pneumonia for both mortality and readmission rates.

In subgroup analyses, the highest mortality correlation was between HF and pneumonia in hospitals with more than 600 beds (r = 0.51, P = 0.0009), and the highest readmission correlation was between AMI and HF in hospitals with more than 600 beds (r = 0.67, P < 0.0001). Across both measures and all 3 condition pairs, correlations between conditions increased with increasing hospital bed size, presence of cardiac surgery capability, and increasing population of the hospital's Census Bureau statistical area. Furthermore, for most measures and condition pairs, correlations between conditions were highest in not‐for‐profit hospitals, hospitals belonging to the Council of Teaching Hospitals, and non‐safety net hospitals (Table 3).

For all condition pairs, the correlation between readmission rates was significantly higher than the correlation between mortality rates (P < 0.01). In subgroup analyses, readmission correlations were also significantly higher than mortality correlations for all pairs of conditions among moderate‐sized hospitals, among nonprofit hospitals, among teaching hospitals that did not belong to the Council of Teaching Hospitals, and among non‐safety net hospitals (Table 5).

DISCUSSION

In this study, we found that risk‐standardized mortality rates for 3 common medical conditions were moderately correlated within institutions, as were risk‐standardized readmission rates. Readmission rates were more strongly correlated than mortality rates, and all rates tracked closest together in large, urban, and/or teaching hospitals. Very few hospitals were in the top quartile of performance for one condition and in the bottom quartile for a different condition.

Our findings are consistent with the hypothesis that 30‐day risk‐standardized mortality and 30‐day risk‐standardized readmission rates, in part, capture broad aspects of hospital quality that transcend condition‐specific activities. In this study, readmission rates tracked better together than mortality rates for every pair of conditions, suggesting that there may be a greater contribution of hospital‐wide environment, structure, and processes to readmission rates than to mortality rates. This difference is plausible because services specific to readmission, such as discharge planning, care coordination, medication reconciliation, and discharge communication with patients and outpatient clinicians, are typically hospital‐wide processes.

Our study differs from earlier studies of medical conditions in that the correlations we found were higher.18, 19 There are several possible explanations for this difference. First, during the intervening 1525 years since those studies were performed, care for these conditions has evolved substantially, such that there are now more standardized protocols available for all 3 of these diseases. Hospitals that are sufficiently organized or acculturated to systematically implement care protocols may have the infrastructure or culture to do so for all conditions, increasing correlation of performance among conditions. In addition, there are now more technologies and systems available that span care for multiple conditions, such as electronic medical records and quality committees, than were available in previous generations. Second, one of these studies utilized less robust risk‐adjustment,18 and neither used the same methodology of risk standardization. Nonetheless, it is interesting to note that Rosenthal and colleagues identified the same increase in correlation with higher volumes than we did.19 Studies investigating mortality correlations among surgical procedures, on the other hand, have generally found higher correlations than we found in these medical conditions.16, 17

Accountable care organizations will be assessed using an all‐condition readmission measure,31 several states track all‐condition readmission rates,3234 and several countries measure all‐condition mortality.35 An all‐condition measure for quality assessment first requires that there be a hospital‐wide quality signal above and beyond disease‐specific care. This study suggests that a moderate signal exists for readmission and, to a slightly lesser extent, for mortality, across 3 common conditions. There are other considerations, however, in developing all‐condition measures. There must be adequate risk adjustment for the wide variety of conditions that are included, and there must be a means of accounting for the variation in types of conditions and procedures cared for by different hospitals. Our study does not address these challenges, which have been described to be substantial for mortality measures.35

We were surprised by the finding that risk‐standardized rates correlated more strongly within larger institutions than smaller ones, because one might assume that care within smaller hospitals might be more homogenous. It may be easier, however, to detect a quality signal in hospitals with higher volumes of patients for all 3 conditions, because estimates for these hospitals are more precise. Consequently, we have greater confidence in results for larger volumes, and suspect a similar quality signal may be present but more difficult to detect statistically in smaller hospitals. Overall correlations were higher when we restricted the sample to hospitals with at least 25 cases, as is used for public reporting. It is also possible that the finding is real given that large‐volume hospitals have been demonstrated to provide better care for these conditions and are more likely to adopt systems of care that affect multiple conditions, such as electronic medical records.14, 36

The kappa scores comparing quartile of national performance for pairs of conditions were only in the fair range. There are several possible explanations for this fact: 1) outcomes for these 3 conditions are not measuring the same constructs; 2) they are all measuring the same construct, but they are unreliable in doing so; and/or 3) hospitals have similar latent quality for all 3 conditions, but the national quality of performance differs by condition, yielding variable relative performance per hospital for each condition. Based solely on our findings, we cannot distinguish which, if any, of these explanations may be true.31

Our study has several limitations. First, all 3 conditions currently publicly reported by CMS are medical diagnoses, although AMI patients may be cared for in distinct cardiology units and often undergo procedures; therefore, we cannot determine the degree to which correlations reflect hospital‐wide quality versus medicine‐wide quality. An institution may have a weak medicine department but a strong surgical department or vice versa. Second, it is possible that the correlations among conditions for readmission and among conditions for mortality are attributable to patient characteristics that are not adequately adjusted for in the risk‐adjustment model, such as socioeconomic factors, or to hospital characteristics not related to quality, such as coding practices or inter‐hospital transfer rates. For this to be true, these unmeasured characteristics would have to be consistent across different conditions within each hospital and have a consistent influence on outcomes. Third, it is possible that public reporting may have prompted disease‐specific focus on these conditions. We do not have data from non‐publicly reported conditions to test this hypothesis. Fourth, there are many small‐volume hospitals in this study; their estimates for readmission and mortality are less reliable than for large‐volume hospitals, potentially limiting our ability to detect correlations in this group of hospitals.

This study lends credence to the hypothesis that 30‐day risk‐standardized mortality and readmission rates for individual conditions may reflect aspects of hospital‐wide quality or at least medicine‐wide quality, although the correlations are not large enough to conclude that hospital‐wide factors play a dominant role, and there are other possible explanations for the correlations. Further work is warranted to better understand the causes of the correlations, and to better specify the nature of hospital factors that contribute to correlations among outcomes.

The Centers for Medicare & Medicaid Services (CMS) publicly reports hospital‐specific, 30‐day risk‐standardized mortality and readmission rates for Medicare fee‐for‐service patients admitted with acute myocardial infarction (AMI), heart failure (HF), and pneumonia.1 These measures are intended to reflect hospital performance on quality of care provided to patients during and after hospitalization.2, 3

Quality‐of‐care measures for a given disease are often assumed to reflect the quality of care for that particular condition. However, studies have found limited association between condition‐specific process measures and either mortality or readmission rates for those conditions.46 Mortality and readmission rates may instead reflect broader hospital‐wide or specialty‐wide structure, culture, and practice. For example, studies have previously found that hospitals differ in mortality or readmission rates according to organizational structure,7 financial structure,8 culture,9, 10 information technology,11 patient volume,1214 academic status,12 and other institution‐wide factors.12 There is now a strong policy push towards developing hospital‐wide (all‐condition) measures, beginning with readmission.15

It is not clear how much of the quality of care for a given condition is attributable to hospital‐wide influences that affect all conditions rather than disease‐specific factors. If readmission or mortality performance for a particular condition reflects, in large part, broader institutional characteristics, then improvement efforts might better be focused on hospital‐wide activities, such as team training or implementing electronic medical records. On the other hand, if the disease‐specific measures reflect quality strictly for those conditions, then improvement efforts would be better focused on disease‐specific care, such as early identification of the relevant patient population or standardizing disease‐specific care. As hospitals work to improve performance across an increasingly wide variety of conditions, it is becoming more important for hospitals to prioritize and focus their activities effectively and efficiently.

One means of determining the relative contribution of hospital versus disease factors is to explore whether outcome rates are consistent among different conditions cared for in the same hospital. If mortality (or readmission) rates across different conditions are highly correlated, it would suggest that hospital‐wide factors may play a substantive role in outcomes. Some studies have found that mortality for a particular surgical condition is a useful proxy for mortality for other surgical conditions,16, 17 while other studies have found little correlation among mortality rates for various medical conditions.18, 19 It is also possible that correlation varies according to hospital characteristics; for example, smaller or nonteaching hospitals might be more homogenous in their care than larger, less homogeneous institutions. No studies have been performed using publicly reported estimates of risk‐standardized mortality or readmission rates. In this study we use the publicly reported measures of 30‐day mortality and 30‐day readmission for AMI, HF, and pneumonia to examine whether, and to what degree, mortality rates track together within US hospitals, and separately, to what degree readmission rates track together within US hospitals.

METHODS

RESULTS

The mortality cohort included 4559 hospitals, and the readmission cohort included 4468 hospitals. The majority of hospitals was small, nonteaching, and did not have advanced cardiac capabilities such as cardiac surgery or cardiac catheterization (Table 1).

For mortality measures, the smallest median number of cases per hospital was for AMI (48; interquartile range [IQR], 13,171), and the greatest number was for pneumonia (178; IQR, 87, 336). The same pattern held for readmission measures (AMI median 33; IQR; 9, 150; pneumonia median 191; IQR, 95, 352.5). With respect to mortality measures, AMI had the highest rate and HF the lowest rate; however, for readmission measures, HF had the highest rate and pneumonia the lowest rate (Table 2).

Every mortality measure was significantly correlated with every other mortality measure (range of correlation coefficients, 0.270.41, P < 0.0001 for all 3 correlations). For example, the correlation between risk‐standardized mortality rates (RSMR) for HF and pneumonia was 0.41. Similarly, every readmission measure was significantly correlated with every other readmission measure (range of correlation coefficients, 0.320.47; P < 0.0001 for all 3 correlations). Overall, the lowest correlation was between risk‐standardized mortality rates for AMI and pneumonia (r = 0.27), and the highest correlation was between risk‐standardized readmission rates (RSRR) for HF and pneumonia (r = 0.47) (Table 3).

Both the factor analysis for the mortality measures and the factor analysis for the readmission measures yielded only one factor with an eigenvalue >1. In each factor analysis, this single common factor kept more than half of the data based on the cumulative eigenvalue (55% for mortality measures and 60% for readmission measures). For the mortality measures, the pattern of RSMR for myocardial infarction (MI), heart failure (HF), and pneumonia (PN) in the factor was high (0.68 for MI, 0.78 for HF, and 0.76 for PN); the same was true of the RSRR in the readmission measures (0.72 for MI, 0.81 for HF, and 0.78 for PN).

For all condition pairs and both outcomes, a third or more of hospitals were in the same quartile of performance for both conditions of the pair (Table 4). Hospitals were more likely to be in the same quartile of performance if they were in the top or bottom quartile than if they were in the middle. Less than 10% of hospitals were in the top quartile for one condition in the mortality or readmission pair and in the bottom quartile for the other condition in the pair. Kappa scores for same quartile of performance between pairs of outcomes ranged from 0.16 to 0.27, and were highest for HF and pneumonia for both mortality and readmission rates.

In subgroup analyses, the highest mortality correlation was between HF and pneumonia in hospitals with more than 600 beds (r = 0.51, P = 0.0009), and the highest readmission correlation was between AMI and HF in hospitals with more than 600 beds (r = 0.67, P < 0.0001). Across both measures and all 3 condition pairs, correlations between conditions increased with increasing hospital bed size, presence of cardiac surgery capability, and increasing population of the hospital's Census Bureau statistical area. Furthermore, for most measures and condition pairs, correlations between conditions were highest in not‐for‐profit hospitals, hospitals belonging to the Council of Teaching Hospitals, and non‐safety net hospitals (Table 3).

For all condition pairs, the correlation between readmission rates was significantly higher than the correlation between mortality rates (P < 0.01). In subgroup analyses, readmission correlations were also significantly higher than mortality correlations for all pairs of conditions among moderate‐sized hospitals, among nonprofit hospitals, among teaching hospitals that did not belong to the Council of Teaching Hospitals, and among non‐safety net hospitals (Table 5).

DISCUSSION

In this study, we found that risk‐standardized mortality rates for 3 common medical conditions were moderately correlated within institutions, as were risk‐standardized readmission rates. Readmission rates were more strongly correlated than mortality rates, and all rates tracked closest together in large, urban, and/or teaching hospitals. Very few hospitals were in the top quartile of performance for one condition and in the bottom quartile for a different condition.

Our findings are consistent with the hypothesis that 30‐day risk‐standardized mortality and 30‐day risk‐standardized readmission rates, in part, capture broad aspects of hospital quality that transcend condition‐specific activities. In this study, readmission rates tracked better together than mortality rates for every pair of conditions, suggesting that there may be a greater contribution of hospital‐wide environment, structure, and processes to readmission rates than to mortality rates. This difference is plausible because services specific to readmission, such as discharge planning, care coordination, medication reconciliation, and discharge communication with patients and outpatient clinicians, are typically hospital‐wide processes.

Our study differs from earlier studies of medical conditions in that the correlations we found were higher.18, 19 There are several possible explanations for this difference. First, during the intervening 1525 years since those studies were performed, care for these conditions has evolved substantially, such that there are now more standardized protocols available for all 3 of these diseases. Hospitals that are sufficiently organized or acculturated to systematically implement care protocols may have the infrastructure or culture to do so for all conditions, increasing correlation of performance among conditions. In addition, there are now more technologies and systems available that span care for multiple conditions, such as electronic medical records and quality committees, than were available in previous generations. Second, one of these studies utilized less robust risk‐adjustment,18 and neither used the same methodology of risk standardization. Nonetheless, it is interesting to note that Rosenthal and colleagues identified the same increase in correlation with higher volumes than we did.19 Studies investigating mortality correlations among surgical procedures, on the other hand, have generally found higher correlations than we found in these medical conditions.16, 17

Accountable care organizations will be assessed using an all‐condition readmission measure,31 several states track all‐condition readmission rates,3234 and several countries measure all‐condition mortality.35 An all‐condition measure for quality assessment first requires that there be a hospital‐wide quality signal above and beyond disease‐specific care. This study suggests that a moderate signal exists for readmission and, to a slightly lesser extent, for mortality, across 3 common conditions. There are other considerations, however, in developing all‐condition measures. There must be adequate risk adjustment for the wide variety of conditions that are included, and there must be a means of accounting for the variation in types of conditions and procedures cared for by different hospitals. Our study does not address these challenges, which have been described to be substantial for mortality measures.35

We were surprised by the finding that risk‐standardized rates correlated more strongly within larger institutions than smaller ones, because one might assume that care within smaller hospitals might be more homogenous. It may be easier, however, to detect a quality signal in hospitals with higher volumes of patients for all 3 conditions, because estimates for these hospitals are more precise. Consequently, we have greater confidence in results for larger volumes, and suspect a similar quality signal may be present but more difficult to detect statistically in smaller hospitals. Overall correlations were higher when we restricted the sample to hospitals with at least 25 cases, as is used for public reporting. It is also possible that the finding is real given that large‐volume hospitals have been demonstrated to provide better care for these conditions and are more likely to adopt systems of care that affect multiple conditions, such as electronic medical records.14, 36

The kappa scores comparing quartile of national performance for pairs of conditions were only in the fair range. There are several possible explanations for this fact: 1) outcomes for these 3 conditions are not measuring the same constructs; 2) they are all measuring the same construct, but they are unreliable in doing so; and/or 3) hospitals have similar latent quality for all 3 conditions, but the national quality of performance differs by condition, yielding variable relative performance per hospital for each condition. Based solely on our findings, we cannot distinguish which, if any, of these explanations may be true.31

Our study has several limitations. First, all 3 conditions currently publicly reported by CMS are medical diagnoses, although AMI patients may be cared for in distinct cardiology units and often undergo procedures; therefore, we cannot determine the degree to which correlations reflect hospital‐wide quality versus medicine‐wide quality. An institution may have a weak medicine department but a strong surgical department or vice versa. Second, it is possible that the correlations among conditions for readmission and among conditions for mortality are attributable to patient characteristics that are not adequately adjusted for in the risk‐adjustment model, such as socioeconomic factors, or to hospital characteristics not related to quality, such as coding practices or inter‐hospital transfer rates. For this to be true, these unmeasured characteristics would have to be consistent across different conditions within each hospital and have a consistent influence on outcomes. Third, it is possible that public reporting may have prompted disease‐specific focus on these conditions. We do not have data from non‐publicly reported conditions to test this hypothesis. Fourth, there are many small‐volume hospitals in this study; their estimates for readmission and mortality are less reliable than for large‐volume hospitals, potentially limiting our ability to detect correlations in this group of hospitals.

This study lends credence to the hypothesis that 30‐day risk‐standardized mortality and readmission rates for individual conditions may reflect aspects of hospital‐wide quality or at least medicine‐wide quality, although the correlations are not large enough to conclude that hospital‐wide factors play a dominant role, and there are other possible explanations for the correlations. Further work is warranted to better understand the causes of the correlations, and to better specify the nature of hospital factors that contribute to correlations among outcomes.

The Centers for Medicare & Medicaid Services (CMS) publicly reports hospital‐specific, 30‐day risk‐standardized mortality and readmission rates for Medicare fee‐for‐service patients admitted with acute myocardial infarction (AMI), heart failure (HF), and pneumonia.1 These measures are intended to reflect hospital performance on quality of care provided to patients during and after hospitalization.2, 3

Quality‐of‐care measures for a given disease are often assumed to reflect the quality of care for that particular condition. However, studies have found limited association between condition‐specific process measures and either mortality or readmission rates for those conditions.46 Mortality and readmission rates may instead reflect broader hospital‐wide or specialty‐wide structure, culture, and practice. For example, studies have previously found that hospitals differ in mortality or readmission rates according to organizational structure,7 financial structure,8 culture,9, 10 information technology,11 patient volume,1214 academic status,12 and other institution‐wide factors.12 There is now a strong policy push towards developing hospital‐wide (all‐condition) measures, beginning with readmission.15

It is not clear how much of the quality of care for a given condition is attributable to hospital‐wide influences that affect all conditions rather than disease‐specific factors. If readmission or mortality performance for a particular condition reflects, in large part, broader institutional characteristics, then improvement efforts might better be focused on hospital‐wide activities, such as team training or implementing electronic medical records. On the other hand, if the disease‐specific measures reflect quality strictly for those conditions, then improvement efforts would be better focused on disease‐specific care, such as early identification of the relevant patient population or standardizing disease‐specific care. As hospitals work to improve performance across an increasingly wide variety of conditions, it is becoming more important for hospitals to prioritize and focus their activities effectively and efficiently.

One means of determining the relative contribution of hospital versus disease factors is to explore whether outcome rates are consistent among different conditions cared for in the same hospital. If mortality (or readmission) rates across different conditions are highly correlated, it would suggest that hospital‐wide factors may play a substantive role in outcomes. Some studies have found that mortality for a particular surgical condition is a useful proxy for mortality for other surgical conditions,16, 17 while other studies have found little correlation among mortality rates for various medical conditions.18, 19 It is also possible that correlation varies according to hospital characteristics; for example, smaller or nonteaching hospitals might be more homogenous in their care than larger, less homogeneous institutions. No studies have been performed using publicly reported estimates of risk‐standardized mortality or readmission rates. In this study we use the publicly reported measures of 30‐day mortality and 30‐day readmission for AMI, HF, and pneumonia to examine whether, and to what degree, mortality rates track together within US hospitals, and separately, to what degree readmission rates track together within US hospitals.

METHODS

RESULTS

The mortality cohort included 4559 hospitals, and the readmission cohort included 4468 hospitals. The majority of hospitals was small, nonteaching, and did not have advanced cardiac capabilities such as cardiac surgery or cardiac catheterization (Table 1).

For mortality measures, the smallest median number of cases per hospital was for AMI (48; interquartile range [IQR], 13,171), and the greatest number was for pneumonia (178; IQR, 87, 336). The same pattern held for readmission measures (AMI median 33; IQR; 9, 150; pneumonia median 191; IQR, 95, 352.5). With respect to mortality measures, AMI had the highest rate and HF the lowest rate; however, for readmission measures, HF had the highest rate and pneumonia the lowest rate (Table 2).

Every mortality measure was significantly correlated with every other mortality measure (range of correlation coefficients, 0.270.41, P < 0.0001 for all 3 correlations). For example, the correlation between risk‐standardized mortality rates (RSMR) for HF and pneumonia was 0.41. Similarly, every readmission measure was significantly correlated with every other readmission measure (range of correlation coefficients, 0.320.47; P < 0.0001 for all 3 correlations). Overall, the lowest correlation was between risk‐standardized mortality rates for AMI and pneumonia (r = 0.27), and the highest correlation was between risk‐standardized readmission rates (RSRR) for HF and pneumonia (r = 0.47) (Table 3).

Both the factor analysis for the mortality measures and the factor analysis for the readmission measures yielded only one factor with an eigenvalue >1. In each factor analysis, this single common factor kept more than half of the data based on the cumulative eigenvalue (55% for mortality measures and 60% for readmission measures). For the mortality measures, the pattern of RSMR for myocardial infarction (MI), heart failure (HF), and pneumonia (PN) in the factor was high (0.68 for MI, 0.78 for HF, and 0.76 for PN); the same was true of the RSRR in the readmission measures (0.72 for MI, 0.81 for HF, and 0.78 for PN).

For all condition pairs and both outcomes, a third or more of hospitals were in the same quartile of performance for both conditions of the pair (Table 4). Hospitals were more likely to be in the same quartile of performance if they were in the top or bottom quartile than if they were in the middle. Less than 10% of hospitals were in the top quartile for one condition in the mortality or readmission pair and in the bottom quartile for the other condition in the pair. Kappa scores for same quartile of performance between pairs of outcomes ranged from 0.16 to 0.27, and were highest for HF and pneumonia for both mortality and readmission rates.

In subgroup analyses, the highest mortality correlation was between HF and pneumonia in hospitals with more than 600 beds (r = 0.51, P = 0.0009), and the highest readmission correlation was between AMI and HF in hospitals with more than 600 beds (r = 0.67, P < 0.0001). Across both measures and all 3 condition pairs, correlations between conditions increased with increasing hospital bed size, presence of cardiac surgery capability, and increasing population of the hospital's Census Bureau statistical area. Furthermore, for most measures and condition pairs, correlations between conditions were highest in not‐for‐profit hospitals, hospitals belonging to the Council of Teaching Hospitals, and non‐safety net hospitals (Table 3).

For all condition pairs, the correlation between readmission rates was significantly higher than the correlation between mortality rates (P < 0.01). In subgroup analyses, readmission correlations were also significantly higher than mortality correlations for all pairs of conditions among moderate‐sized hospitals, among nonprofit hospitals, among teaching hospitals that did not belong to the Council of Teaching Hospitals, and among non‐safety net hospitals (Table 5).

DISCUSSION

In this study, we found that risk‐standardized mortality rates for 3 common medical conditions were moderately correlated within institutions, as were risk‐standardized readmission rates. Readmission rates were more strongly correlated than mortality rates, and all rates tracked closest together in large, urban, and/or teaching hospitals. Very few hospitals were in the top quartile of performance for one condition and in the bottom quartile for a different condition.

Our findings are consistent with the hypothesis that 30‐day risk‐standardized mortality and 30‐day risk‐standardized readmission rates, in part, capture broad aspects of hospital quality that transcend condition‐specific activities. In this study, readmission rates tracked better together than mortality rates for every pair of conditions, suggesting that there may be a greater contribution of hospital‐wide environment, structure, and processes to readmission rates than to mortality rates. This difference is plausible because services specific to readmission, such as discharge planning, care coordination, medication reconciliation, and discharge communication with patients and outpatient clinicians, are typically hospital‐wide processes.

Our study differs from earlier studies of medical conditions in that the correlations we found were higher.18, 19 There are several possible explanations for this difference. First, during the intervening 1525 years since those studies were performed, care for these conditions has evolved substantially, such that there are now more standardized protocols available for all 3 of these diseases. Hospitals that are sufficiently organized or acculturated to systematically implement care protocols may have the infrastructure or culture to do so for all conditions, increasing correlation of performance among conditions. In addition, there are now more technologies and systems available that span care for multiple conditions, such as electronic medical records and quality committees, than were available in previous generations. Second, one of these studies utilized less robust risk‐adjustment,18 and neither used the same methodology of risk standardization. Nonetheless, it is interesting to note that Rosenthal and colleagues identified the same increase in correlation with higher volumes than we did.19 Studies investigating mortality correlations among surgical procedures, on the other hand, have generally found higher correlations than we found in these medical conditions.16, 17

Accountable care organizations will be assessed using an all‐condition readmission measure,31 several states track all‐condition readmission rates,3234 and several countries measure all‐condition mortality.35 An all‐condition measure for quality assessment first requires that there be a hospital‐wide quality signal above and beyond disease‐specific care. This study suggests that a moderate signal exists for readmission and, to a slightly lesser extent, for mortality, across 3 common conditions. There are other considerations, however, in developing all‐condition measures. There must be adequate risk adjustment for the wide variety of conditions that are included, and there must be a means of accounting for the variation in types of conditions and procedures cared for by different hospitals. Our study does not address these challenges, which have been described to be substantial for mortality measures.35

We were surprised by the finding that risk‐standardized rates correlated more strongly within larger institutions than smaller ones, because one might assume that care within smaller hospitals might be more homogenous. It may be easier, however, to detect a quality signal in hospitals with higher volumes of patients for all 3 conditions, because estimates for these hospitals are more precise. Consequently, we have greater confidence in results for larger volumes, and suspect a similar quality signal may be present but more difficult to detect statistically in smaller hospitals. Overall correlations were higher when we restricted the sample to hospitals with at least 25 cases, as is used for public reporting. It is also possible that the finding is real given that large‐volume hospitals have been demonstrated to provide better care for these conditions and are more likely to adopt systems of care that affect multiple conditions, such as electronic medical records.14, 36

The kappa scores comparing quartile of national performance for pairs of conditions were only in the fair range. There are several possible explanations for this fact: 1) outcomes for these 3 conditions are not measuring the same constructs; 2) they are all measuring the same construct, but they are unreliable in doing so; and/or 3) hospitals have similar latent quality for all 3 conditions, but the national quality of performance differs by condition, yielding variable relative performance per hospital for each condition. Based solely on our findings, we cannot distinguish which, if any, of these explanations may be true.31

Our study has several limitations. First, all 3 conditions currently publicly reported by CMS are medical diagnoses, although AMI patients may be cared for in distinct cardiology units and often undergo procedures; therefore, we cannot determine the degree to which correlations reflect hospital‐wide quality versus medicine‐wide quality. An institution may have a weak medicine department but a strong surgical department or vice versa. Second, it is possible that the correlations among conditions for readmission and among conditions for mortality are attributable to patient characteristics that are not adequately adjusted for in the risk‐adjustment model, such as socioeconomic factors, or to hospital characteristics not related to quality, such as coding practices or inter‐hospital transfer rates. For this to be true, these unmeasured characteristics would have to be consistent across different conditions within each hospital and have a consistent influence on outcomes. Third, it is possible that public reporting may have prompted disease‐specific focus on these conditions. We do not have data from non‐publicly reported conditions to test this hypothesis. Fourth, there are many small‐volume hospitals in this study; their estimates for readmission and mortality are less reliable than for large‐volume hospitals, potentially limiting our ability to detect correlations in this group of hospitals.

This study lends credence to the hypothesis that 30‐day risk‐standardized mortality and readmission rates for individual conditions may reflect aspects of hospital‐wide quality or at least medicine‐wide quality, although the correlations are not large enough to conclude that hospital‐wide factors play a dominant role, and there are other possible explanations for the correlations. Further work is warranted to better understand the causes of the correlations, and to better specify the nature of hospital factors that contribute to correlations among outcomes.

Antimicrobial use provides the selective pressure that cause bacteria to develop antimicrobial resistance.1 Currently, clones of bacteria with very limited antimicrobial sensitivity are gradually spreading around the world.2 The intensive care unit (ICU) is a focus of resistant bacteria within the hospital3 as a result of high illness severity, widespread use of invasive monitoring or therapeutic devices, frequency of bacterial infection (found in approximately 51% of patients4), and consequent extensive use of broad‐spectrum antimicrobials (in 71% of patients).4

When prescribing antimicrobials, the ICU clinician often faces a dilemma. First, the traditional symptoms and signs of infection (such as characteristic patient history, fever, increased white cell count, etc) are common in ICU patients even in the absence of infection, making distinction of infectious and noninfectious causes of patient deterioration difficult. Second, delaying antimicrobial therapy, prescribing inadequate antimicrobials, or allowing bacterial infections to go untreated, increases patient mortality,57 resulting in guideline recommendations to start broad‐spectrum antimicrobials as soon as possible in the presence of suspected severe sepsis.8 While third, and in contrast, unnecessary antimicrobial therapy increases the risk of antimicrobial‐related complications, such as Clostridium difficile colitis (with a crude mortality of up to 20%9), and potentially endangers the greater population of ICU patients by increasing the prevalence of resistant organisms. Choosing between delaying necessary antimicrobial therapy and exposing the patient to unnecessary therapy requires that 2 contrasting risks be balancedthat of untreated infection versus late antimicrobial complications.

The main aim of this study was to assess how often administration of antimicrobials for suspected infection could be justified by the presence of infection. The primary outcome measure was accuracy of antimicrobial administration, defined as the proportion of antimicrobials started for suspected infection where infection was later proven to have been present. Secondary outcome measures examined: (1) whether clinician suspicion of infection correlated with the presence of defined infection; (2) the ID specialist's accuracy for empiric antimicrobial administration; (3) whether common clinical parameters were associated with clinician certainty regarding the presence of infection; and (4) use of antimicrobials in the presence or absence of infection. These data are important in order to identify possibilities for improving antimicrobial administration.

METHODS

Accuracy of Antimicrobial Start Decisions

Accuracy was calculated for both the clinicians and the study ID specialist, and expressed as a proportion. The denominator for ICU clinician accuracy was the total number of antibiotic start decisions for suspected infection, and the numerator was the number of decisions where infection was defined by the ID specialist. For the study ID specialist, the denominator was the number of occasions when antibiotic administration was considered justified in the Step 1 analysis, and the numerator was the number of these cases where infection was defined (Figure 1). The correlation between clinician certainty and the study‐defined presence of infection was examined. The accuracy of clinical antimicrobial decisions made during the first 48 hours of ICU admission was compared to decisions made after 48 hours.

Figure 1

In order to assess the robustness of the study findings, the ICU clinician accuracy was examined in a sensitivity analysis. Accuracy was calculated using a lower cutoff for the study ID specialist's definition of infectionpossible infection (score 2) or above, rather than probable infection or above.

RESULTS

Data were collected on 119 consecutive ICU patients over 4 months (Table 1). Antimicrobials were started for suspected infection in 80/119 (67%) patients, for prophylaxis in 55/119 (46%) patients, and for other reasons in 42/119 (35%) patients. More than one indication was present during the patient's ICU admission among 41/119 (34%) patients, while for 6/119 (5%) patients, no antimicrobials were prescribed at all. Among these patients, antimicrobials were administered on 250 occasions, including 125/250 (50%) occasions for suspected infection (empirical decisions), 62/250 (25%) occasions for procedural prophylaxis (prophylaxis‐driven), and on 63/250 (25%) occasions for other reasons (antimicrobial changes following receipt of culture results, or continuation of antimicrobials prescribed prior to ICU admission). Microbiological cultures were obtained from the study population on 2132 occasions, including 395 blood cultures in which significant organisms (not reflecting contamination) grew on 57/395 (14%) occasions.

Among the empiric antimicrobial start decisions, infection was defined by the study ID specialist on 67/125 (54%) occasions, representing the clinicians' diagnostic accuracy. These infections included 17 (25%) respiratory, 16 (24%) abdominal, 13 (19%) soft tissue, 11 (16%) blood stream, 6 (9%) urinary, and 4 (6%) other infections.

Three attending clinicians treated patients during the study period, and their accuracies were similar (21 infections defined/44 start decisions for suspected infection, 48%; 24/38, 63%; 22/43, 51%, for each attending; P = ns for all comparisons). Clinician accuracy was higher for empirical antimicrobial start decisions, made within 48 hours of ICU admission, compared to later decisions (35 defined infections/53 early antibiotic start decisions [66%] vs 32 defined infections/72 late antibiotic start decisions [44%]; P = 0.02).

In a sensitivity analysis, decreasing the cutoff for the study ID specialist's definition of infection from probable (and above) to possible (and above) lead to reclassification of 14/125 (11%) antimicrobial start decisions from no infection defined to infection defined. This increased physician accuracy from 67/125 (54%) to 78/125 (62%), and conversely decreased potential antimicrobial overuse from 58/125 (46%) to 47/125 (38%) decisions (P = ns).

When starting antimicrobials for suspected infection, the clinicians were asked to record their certainty in the presence of infection. Infections were defined on 6/19 (31%) occasions when the clinician certainty score was low (2) versus 61/106 (57%) when the clinician certainty score was high (3, P = 0.037; Figure 2). Correlation between the clinician certainty score and the presence of defined infection was good (r² = 0.78).

Figure 2

The study ID specialist agreed with the clinician's decision to start antimicrobial therapy on 87/125 (70%) occasions. Infection was subsequently defined on 66/87 (76%) occasions, representing the study ID specialist's diagnostic accuracy. The study ID specialist's accuracy was significantly higher than the clinician's (66/87 [76%] versus 67/125 [54%]; P = 0.001). Notably, there was only 1 case (3%) where empiric therapy was deemed unnecessary by the study ID specialist, and where infection was subsequently defined. In this case, the clinicians started antibiotic therapy for suspected ventilator‐associated pneumonia in a 66‐year‐old patient on the 28th day of an ICU admission for head and spinal cord trauma. The ID specialist's certainty for the presence of infection was 3probable. The patient ultimately survived and was discharged to a rehabilitation facility.

Comparing physiological data for antimicrobial start decisions with high clinician certainty (score 3) versus low certainty of infection (score 2), revealed that none of the physiological data, nor changes over time were significantly associated with clinician certainty. Further use of high doses of vasopressors (>0.1 mcg/kg/min, SOFA score 4) was present at 42/106 (40%) high certainty decisions versus 7/19 (37%) low certainty decisions (P = 0.819). This underscores the physicians' difficulty in distinguishing between infectious and inflammatory causes of deterioration (Table 2).

During the study period, 2541 days of antimicrobial therapy were given of which 1677 (66%), 413 (16%), and 451 (18%) were given, respectively, empirically (for suspected infection), for procedural prophylaxis, and as targeted therapy. Antimicrobial course length was 11.5 9.2 days in the presence of defined infection versus 10.7 9.1 days in the absence of defined infection (P = 0.655). Overall, 658/2541 (26%) days of therapy could potentially have been saved by reducing antimicrobial prescriptions for suspected infections which were not defined.

DISCUSSION

The use of empirical antimicrobials could be justified by the presence of defined infection on only 54% of occasions when they were administered, suggesting considerable potential overuse of these drugs. ICU‐clinician certainty for the presence of infection correlated well with the number of infections actually defined, however, infections were defined when certainty was low (Figure 2) and antimicrobials prescribed even when clinician certainty was minimal. Common clinical physiological and laboratory parameters did not seem to assist in the clinicians' decision‐making, as there were no significant differences in any of these values between empiric decisions with high or low certainty. The study ID specialist showed significantly better accuracy in antimicrobial decision‐making than the ICU clinicians. He agreed with antimicrobial administration on only 70% of occasions that clinicians started empiric therapy, and had a higher diagnostic accuracy at a cost of only 1 untreated infection.

Two main possibilities are suggested to explain the potential antimicrobial overuse. First, ICU physicians are loath to leave infections untreated and potentially cause immediate increases in mortality.8 This leads to uncertainty avoidance or risk aversive behavior that is demonstrated in our study by the inclusion of antimicrobial administration decisions made even when physicians' certainty regarding the presence of infection was low. Uncertainty avoidance has been shown to be significantly associated with antimicrobial prescribing practices,13 however, it discounts the risk of antimicrobial complications associated with unnecessary antimicrobial therapy. Second, the diagnosis of infection, and particularly nosocomial infection, in ICU patients is difficult. Symptoms cannot be elicited in obtunded ventilated ICU patients, the physical exam can be equivocal, bacterial growth in cultures (with the exception of blood cultures) often reflects colonization rather than infection, and the laboratory and imaging findings of inflammation and infection are very similar. Our data demonstrated some of these difficulties. Diagnostic accuracy was higher in infections suspected during the first 48 hours of ICU admission when compared to later, presumably as infection leading to ICU admission is associated with symptoms, signs, and an acute change in the patient's condition, factors that may be absent when a patient develops a nosocomial infection. Further, physiological parameters did not correlate with the certainty that ICU clinicians expressed in their decision, indicating the difficulty in interpreting these data. Finally, infection was defined in 30% of low certainty decisions, indicating that clinical impression alone is not a reliable tool for determining the presence of infection.

Three sets of interventions could be suggested to improve antimicrobial decision‐makingincreased use of the ID consult, improved laboratory tests for the diagnosis of infection, and a policy of de‐escalation. Use of antimicrobial stewardship (often through involvement of an ID physician) reduces antimicrobial usage and the occurrence of resistant bacteria without adverse patient outcomes.14 Indeed, our study ID consult showed more accurate antimicrobial prescribing than the ICU clinicians, although he may have been subject to the potential biases described below. All antimicrobial administration decisions taken during the study were, however, made in consultation with the clinical ID consult. The lower performance of the clinical ID consult (when compared to the study ID consult) may have resulted from difficulties in the real‐time interaction with the clinicians or from decisions taken during non‐office hours. During non‐office hours, the on‐call ICU resident presented cases to an on‐call ID specialist, neither of whom may have been familiar with all the complex case details and therefore may have preferred to err by commission than by omission.

More accurate laboratory tests, such as procalcitonin or real‐time bacterial polymerase chain reaction (PCR),15 could be beneficial as they might increase physician confidence in decision‐making. Procalcitonin has been used in a wide variety of settings1618 (including the ICU1921) to safely decrease antimicrobial starts and/or antimicrobial course length. Despite this, in a large multicenter study of procalcitonin use in ICU patients,19 compliance with the antibiotic start protocol was very low. Antibiotics were administered by the participating physicians in 73/93 (78%) cases where the procalcitonin tests indicated that antimicrobials were not required, representing protocol violations. In parallel to our study, this demonstrates the reluctance of physicians to abstain from prescribing antimicrobial therapy for suspected infection even when the likelihood of infection may be low.

A strategy of de‐escalation offers the possibility of starting broad‐spectrum antimicrobials early and, subsequently, narrowing or stopping therapy according to the clinical course and the microbiological results.22, 23 This strategy allows clinicians to start antimicrobials even when the suspicion of infection is low, but to stop them rapidly as the clinical picture clarifies. Unfortunately, the mean antimicrobial course length in this study was not influenced by the presence of infection, indicating that this strategy was not employed successfully.

The proportion of patients prescribed antimicrobials for suspected infection in our study is similar to that found in others (eg, 34% of patients in a large French survey24). The proportion of potentially unnecessary antimicrobials in other studies is also similar, ranging from 14% to 50%.2428 In the ICU, the majority of antimicrobial usage studies are microbiology‐based and examine whether bacteria cultured are resistant to the antimicrobials chosen. They have shown that inappropriate antimicrobial therapy occurs on 20%36% of occasions.6, 7, 29 The current study furthers knowledge on antimicrobials decision‐making in the ICU, by examining the actual requirement for antimicrobial therapy based on the presence of infection, ie, whether antimicrobials were needed at all.

The principal limitation of the study concerns the determination of the presence of infection. The study premise was that antimicrobials are overused, and this may have biased the study ID consult to underestimate appropriateness of antimicrobial therapy and to define fewer infections. Further, making theoretical decisions in the research office avoids the medical, ethical, and legal issues related to clinical practice, as there is no risk associated with error. This may have allowed the study ID specialist to be overly conservative in his definitions of infections. A wider team of decision‐makers to determine the presence of infection, including both ID and ICU specialists, would have lent more weight to their determinations, however, this was logistically impossible. To limit the potential bias, infections were defined as objectively as possible based on the CDC criteria.12 Further, the sensitivity analysis showed that while decreasing the study ID specialist's threshold for the definition of infection from probable and above to possible and above improved physician accuracy, over a third of antimicrobial start decisions remained unjustified by the presence of defined infection. The study was performed in only 1 center and may not reflect general ICU practice, although, as discussed above, the antibiotic decision‐making accuracy is in the same orders of magnitude as those found in other somewhat similar studies. Finally, even if unnecessary antimicrobial use was overestimated, the possibility for significant improvement in antimicrobial administration accuracy remains.

In conclusion, our data suggest that on up to 46% of occasions, empirical antimicrobials are prescribed in the absence of infection. We suggest that the potential antibiotic overuse results from difficulties in diagnosing ICU‐related infections, and from the high perceived risk of untreated infection as compared to the risks of potentially unnecessary antimicrobial therapy, representing a type of risk aversive behavior. As antimicrobial use is the primary factor promoting antibiotic resistance and may be a cause of other patient complications, efforts to improve antimicrobial‐related decision‐making should be mandatory.

Antimicrobial use provides the selective pressure that cause bacteria to develop antimicrobial resistance.1 Currently, clones of bacteria with very limited antimicrobial sensitivity are gradually spreading around the world.2 The intensive care unit (ICU) is a focus of resistant bacteria within the hospital3 as a result of high illness severity, widespread use of invasive monitoring or therapeutic devices, frequency of bacterial infection (found in approximately 51% of patients4), and consequent extensive use of broad‐spectrum antimicrobials (in 71% of patients).4

When prescribing antimicrobials, the ICU clinician often faces a dilemma. First, the traditional symptoms and signs of infection (such as characteristic patient history, fever, increased white cell count, etc) are common in ICU patients even in the absence of infection, making distinction of infectious and noninfectious causes of patient deterioration difficult. Second, delaying antimicrobial therapy, prescribing inadequate antimicrobials, or allowing bacterial infections to go untreated, increases patient mortality,57 resulting in guideline recommendations to start broad‐spectrum antimicrobials as soon as possible in the presence of suspected severe sepsis.8 While third, and in contrast, unnecessary antimicrobial therapy increases the risk of antimicrobial‐related complications, such as Clostridium difficile colitis (with a crude mortality of up to 20%9), and potentially endangers the greater population of ICU patients by increasing the prevalence of resistant organisms. Choosing between delaying necessary antimicrobial therapy and exposing the patient to unnecessary therapy requires that 2 contrasting risks be balancedthat of untreated infection versus late antimicrobial complications.

The main aim of this study was to assess how often administration of antimicrobials for suspected infection could be justified by the presence of infection. The primary outcome measure was accuracy of antimicrobial administration, defined as the proportion of antimicrobials started for suspected infection where infection was later proven to have been present. Secondary outcome measures examined: (1) whether clinician suspicion of infection correlated with the presence of defined infection; (2) the ID specialist's accuracy for empiric antimicrobial administration; (3) whether common clinical parameters were associated with clinician certainty regarding the presence of infection; and (4) use of antimicrobials in the presence or absence of infection. These data are important in order to identify possibilities for improving antimicrobial administration.

METHODS

Accuracy of Antimicrobial Start Decisions

Accuracy was calculated for both the clinicians and the study ID specialist, and expressed as a proportion. The denominator for ICU clinician accuracy was the total number of antibiotic start decisions for suspected infection, and the numerator was the number of decisions where infection was defined by the ID specialist. For the study ID specialist, the denominator was the number of occasions when antibiotic administration was considered justified in the Step 1 analysis, and the numerator was the number of these cases where infection was defined (Figure 1). The correlation between clinician certainty and the study‐defined presence of infection was examined. The accuracy of clinical antimicrobial decisions made during the first 48 hours of ICU admission was compared to decisions made after 48 hours.

Figure 1

In order to assess the robustness of the study findings, the ICU clinician accuracy was examined in a sensitivity analysis. Accuracy was calculated using a lower cutoff for the study ID specialist's definition of infectionpossible infection (score 2) or above, rather than probable infection or above.

RESULTS

Data were collected on 119 consecutive ICU patients over 4 months (Table 1). Antimicrobials were started for suspected infection in 80/119 (67%) patients, for prophylaxis in 55/119 (46%) patients, and for other reasons in 42/119 (35%) patients. More than one indication was present during the patient's ICU admission among 41/119 (34%) patients, while for 6/119 (5%) patients, no antimicrobials were prescribed at all. Among these patients, antimicrobials were administered on 250 occasions, including 125/250 (50%) occasions for suspected infection (empirical decisions), 62/250 (25%) occasions for procedural prophylaxis (prophylaxis‐driven), and on 63/250 (25%) occasions for other reasons (antimicrobial changes following receipt of culture results, or continuation of antimicrobials prescribed prior to ICU admission). Microbiological cultures were obtained from the study population on 2132 occasions, including 395 blood cultures in which significant organisms (not reflecting contamination) grew on 57/395 (14%) occasions.

Among the empiric antimicrobial start decisions, infection was defined by the study ID specialist on 67/125 (54%) occasions, representing the clinicians' diagnostic accuracy. These infections included 17 (25%) respiratory, 16 (24%) abdominal, 13 (19%) soft tissue, 11 (16%) blood stream, 6 (9%) urinary, and 4 (6%) other infections.

Three attending clinicians treated patients during the study period, and their accuracies were similar (21 infections defined/44 start decisions for suspected infection, 48%; 24/38, 63%; 22/43, 51%, for each attending; P = ns for all comparisons). Clinician accuracy was higher for empirical antimicrobial start decisions, made within 48 hours of ICU admission, compared to later decisions (35 defined infections/53 early antibiotic start decisions [66%] vs 32 defined infections/72 late antibiotic start decisions [44%]; P = 0.02).

In a sensitivity analysis, decreasing the cutoff for the study ID specialist's definition of infection from probable (and above) to possible (and above) lead to reclassification of 14/125 (11%) antimicrobial start decisions from no infection defined to infection defined. This increased physician accuracy from 67/125 (54%) to 78/125 (62%), and conversely decreased potential antimicrobial overuse from 58/125 (46%) to 47/125 (38%) decisions (P = ns).

When starting antimicrobials for suspected infection, the clinicians were asked to record their certainty in the presence of infection. Infections were defined on 6/19 (31%) occasions when the clinician certainty score was low (2) versus 61/106 (57%) when the clinician certainty score was high (3, P = 0.037; Figure 2). Correlation between the clinician certainty score and the presence of defined infection was good (r² = 0.78).

Figure 2

The study ID specialist agreed with the clinician's decision to start antimicrobial therapy on 87/125 (70%) occasions. Infection was subsequently defined on 66/87 (76%) occasions, representing the study ID specialist's diagnostic accuracy. The study ID specialist's accuracy was significantly higher than the clinician's (66/87 [76%] versus 67/125 [54%]; P = 0.001). Notably, there was only 1 case (3%) where empiric therapy was deemed unnecessary by the study ID specialist, and where infection was subsequently defined. In this case, the clinicians started antibiotic therapy for suspected ventilator‐associated pneumonia in a 66‐year‐old patient on the 28th day of an ICU admission for head and spinal cord trauma. The ID specialist's certainty for the presence of infection was 3probable. The patient ultimately survived and was discharged to a rehabilitation facility.

Comparing physiological data for antimicrobial start decisions with high clinician certainty (score 3) versus low certainty of infection (score 2), revealed that none of the physiological data, nor changes over time were significantly associated with clinician certainty. Further use of high doses of vasopressors (>0.1 mcg/kg/min, SOFA score 4) was present at 42/106 (40%) high certainty decisions versus 7/19 (37%) low certainty decisions (P = 0.819). This underscores the physicians' difficulty in distinguishing between infectious and inflammatory causes of deterioration (Table 2).

During the study period, 2541 days of antimicrobial therapy were given of which 1677 (66%), 413 (16%), and 451 (18%) were given, respectively, empirically (for suspected infection), for procedural prophylaxis, and as targeted therapy. Antimicrobial course length was 11.5 9.2 days in the presence of defined infection versus 10.7 9.1 days in the absence of defined infection (P = 0.655). Overall, 658/2541 (26%) days of therapy could potentially have been saved by reducing antimicrobial prescriptions for suspected infections which were not defined.

DISCUSSION

The use of empirical antimicrobials could be justified by the presence of defined infection on only 54% of occasions when they were administered, suggesting considerable potential overuse of these drugs. ICU‐clinician certainty for the presence of infection correlated well with the number of infections actually defined, however, infections were defined when certainty was low (Figure 2) and antimicrobials prescribed even when clinician certainty was minimal. Common clinical physiological and laboratory parameters did not seem to assist in the clinicians' decision‐making, as there were no significant differences in any of these values between empiric decisions with high or low certainty. The study ID specialist showed significantly better accuracy in antimicrobial decision‐making than the ICU clinicians. He agreed with antimicrobial administration on only 70% of occasions that clinicians started empiric therapy, and had a higher diagnostic accuracy at a cost of only 1 untreated infection.

Two main possibilities are suggested to explain the potential antimicrobial overuse. First, ICU physicians are loath to leave infections untreated and potentially cause immediate increases in mortality.8 This leads to uncertainty avoidance or risk aversive behavior that is demonstrated in our study by the inclusion of antimicrobial administration decisions made even when physicians' certainty regarding the presence of infection was low. Uncertainty avoidance has been shown to be significantly associated with antimicrobial prescribing practices,13 however, it discounts the risk of antimicrobial complications associated with unnecessary antimicrobial therapy. Second, the diagnosis of infection, and particularly nosocomial infection, in ICU patients is difficult. Symptoms cannot be elicited in obtunded ventilated ICU patients, the physical exam can be equivocal, bacterial growth in cultures (with the exception of blood cultures) often reflects colonization rather than infection, and the laboratory and imaging findings of inflammation and infection are very similar. Our data demonstrated some of these difficulties. Diagnostic accuracy was higher in infections suspected during the first 48 hours of ICU admission when compared to later, presumably as infection leading to ICU admission is associated with symptoms, signs, and an acute change in the patient's condition, factors that may be absent when a patient develops a nosocomial infection. Further, physiological parameters did not correlate with the certainty that ICU clinicians expressed in their decision, indicating the difficulty in interpreting these data. Finally, infection was defined in 30% of low certainty decisions, indicating that clinical impression alone is not a reliable tool for determining the presence of infection.

Three sets of interventions could be suggested to improve antimicrobial decision‐makingincreased use of the ID consult, improved laboratory tests for the diagnosis of infection, and a policy of de‐escalation. Use of antimicrobial stewardship (often through involvement of an ID physician) reduces antimicrobial usage and the occurrence of resistant bacteria without adverse patient outcomes.14 Indeed, our study ID consult showed more accurate antimicrobial prescribing than the ICU clinicians, although he may have been subject to the potential biases described below. All antimicrobial administration decisions taken during the study were, however, made in consultation with the clinical ID consult. The lower performance of the clinical ID consult (when compared to the study ID consult) may have resulted from difficulties in the real‐time interaction with the clinicians or from decisions taken during non‐office hours. During non‐office hours, the on‐call ICU resident presented cases to an on‐call ID specialist, neither of whom may have been familiar with all the complex case details and therefore may have preferred to err by commission than by omission.

More accurate laboratory tests, such as procalcitonin or real‐time bacterial polymerase chain reaction (PCR),15 could be beneficial as they might increase physician confidence in decision‐making. Procalcitonin has been used in a wide variety of settings1618 (including the ICU1921) to safely decrease antimicrobial starts and/or antimicrobial course length. Despite this, in a large multicenter study of procalcitonin use in ICU patients,19 compliance with the antibiotic start protocol was very low. Antibiotics were administered by the participating physicians in 73/93 (78%) cases where the procalcitonin tests indicated that antimicrobials were not required, representing protocol violations. In parallel to our study, this demonstrates the reluctance of physicians to abstain from prescribing antimicrobial therapy for suspected infection even when the likelihood of infection may be low.

A strategy of de‐escalation offers the possibility of starting broad‐spectrum antimicrobials early and, subsequently, narrowing or stopping therapy according to the clinical course and the microbiological results.22, 23 This strategy allows clinicians to start antimicrobials even when the suspicion of infection is low, but to stop them rapidly as the clinical picture clarifies. Unfortunately, the mean antimicrobial course length in this study was not influenced by the presence of infection, indicating that this strategy was not employed successfully.

The proportion of patients prescribed antimicrobials for suspected infection in our study is similar to that found in others (eg, 34% of patients in a large French survey24). The proportion of potentially unnecessary antimicrobials in other studies is also similar, ranging from 14% to 50%.2428 In the ICU, the majority of antimicrobial usage studies are microbiology‐based and examine whether bacteria cultured are resistant to the antimicrobials chosen. They have shown that inappropriate antimicrobial therapy occurs on 20%36% of occasions.6, 7, 29 The current study furthers knowledge on antimicrobials decision‐making in the ICU, by examining the actual requirement for antimicrobial therapy based on the presence of infection, ie, whether antimicrobials were needed at all.

The principal limitation of the study concerns the determination of the presence of infection. The study premise was that antimicrobials are overused, and this may have biased the study ID consult to underestimate appropriateness of antimicrobial therapy and to define fewer infections. Further, making theoretical decisions in the research office avoids the medical, ethical, and legal issues related to clinical practice, as there is no risk associated with error. This may have allowed the study ID specialist to be overly conservative in his definitions of infections. A wider team of decision‐makers to determine the presence of infection, including both ID and ICU specialists, would have lent more weight to their determinations, however, this was logistically impossible. To limit the potential bias, infections were defined as objectively as possible based on the CDC criteria.12 Further, the sensitivity analysis showed that while decreasing the study ID specialist's threshold for the definition of infection from probable and above to possible and above improved physician accuracy, over a third of antimicrobial start decisions remained unjustified by the presence of defined infection. The study was performed in only 1 center and may not reflect general ICU practice, although, as discussed above, the antibiotic decision‐making accuracy is in the same orders of magnitude as those found in other somewhat similar studies. Finally, even if unnecessary antimicrobial use was overestimated, the possibility for significant improvement in antimicrobial administration accuracy remains.

In conclusion, our data suggest that on up to 46% of occasions, empirical antimicrobials are prescribed in the absence of infection. We suggest that the potential antibiotic overuse results from difficulties in diagnosing ICU‐related infections, and from the high perceived risk of untreated infection as compared to the risks of potentially unnecessary antimicrobial therapy, representing a type of risk aversive behavior. As antimicrobial use is the primary factor promoting antibiotic resistance and may be a cause of other patient complications, efforts to improve antimicrobial‐related decision‐making should be mandatory.

Antimicrobial use provides the selective pressure that cause bacteria to develop antimicrobial resistance.1 Currently, clones of bacteria with very limited antimicrobial sensitivity are gradually spreading around the world.2 The intensive care unit (ICU) is a focus of resistant bacteria within the hospital3 as a result of high illness severity, widespread use of invasive monitoring or therapeutic devices, frequency of bacterial infection (found in approximately 51% of patients4), and consequent extensive use of broad‐spectrum antimicrobials (in 71% of patients).4

When prescribing antimicrobials, the ICU clinician often faces a dilemma. First, the traditional symptoms and signs of infection (such as characteristic patient history, fever, increased white cell count, etc) are common in ICU patients even in the absence of infection, making distinction of infectious and noninfectious causes of patient deterioration difficult. Second, delaying antimicrobial therapy, prescribing inadequate antimicrobials, or allowing bacterial infections to go untreated, increases patient mortality,57 resulting in guideline recommendations to start broad‐spectrum antimicrobials as soon as possible in the presence of suspected severe sepsis.8 While third, and in contrast, unnecessary antimicrobial therapy increases the risk of antimicrobial‐related complications, such as Clostridium difficile colitis (with a crude mortality of up to 20%9), and potentially endangers the greater population of ICU patients by increasing the prevalence of resistant organisms. Choosing between delaying necessary antimicrobial therapy and exposing the patient to unnecessary therapy requires that 2 contrasting risks be balancedthat of untreated infection versus late antimicrobial complications.

The main aim of this study was to assess how often administration of antimicrobials for suspected infection could be justified by the presence of infection. The primary outcome measure was accuracy of antimicrobial administration, defined as the proportion of antimicrobials started for suspected infection where infection was later proven to have been present. Secondary outcome measures examined: (1) whether clinician suspicion of infection correlated with the presence of defined infection; (2) the ID specialist's accuracy for empiric antimicrobial administration; (3) whether common clinical parameters were associated with clinician certainty regarding the presence of infection; and (4) use of antimicrobials in the presence or absence of infection. These data are important in order to identify possibilities for improving antimicrobial administration.

METHODS

Accuracy of Antimicrobial Start Decisions

Accuracy was calculated for both the clinicians and the study ID specialist, and expressed as a proportion. The denominator for ICU clinician accuracy was the total number of antibiotic start decisions for suspected infection, and the numerator was the number of decisions where infection was defined by the ID specialist. For the study ID specialist, the denominator was the number of occasions when antibiotic administration was considered justified in the Step 1 analysis, and the numerator was the number of these cases where infection was defined (Figure 1). The correlation between clinician certainty and the study‐defined presence of infection was examined. The accuracy of clinical antimicrobial decisions made during the first 48 hours of ICU admission was compared to decisions made after 48 hours.

Figure 1

In order to assess the robustness of the study findings, the ICU clinician accuracy was examined in a sensitivity analysis. Accuracy was calculated using a lower cutoff for the study ID specialist's definition of infectionpossible infection (score 2) or above, rather than probable infection or above.

RESULTS

Data were collected on 119 consecutive ICU patients over 4 months (Table 1). Antimicrobials were started for suspected infection in 80/119 (67%) patients, for prophylaxis in 55/119 (46%) patients, and for other reasons in 42/119 (35%) patients. More than one indication was present during the patient's ICU admission among 41/119 (34%) patients, while for 6/119 (5%) patients, no antimicrobials were prescribed at all. Among these patients, antimicrobials were administered on 250 occasions, including 125/250 (50%) occasions for suspected infection (empirical decisions), 62/250 (25%) occasions for procedural prophylaxis (prophylaxis‐driven), and on 63/250 (25%) occasions for other reasons (antimicrobial changes following receipt of culture results, or continuation of antimicrobials prescribed prior to ICU admission). Microbiological cultures were obtained from the study population on 2132 occasions, including 395 blood cultures in which significant organisms (not reflecting contamination) grew on 57/395 (14%) occasions.

Among the empiric antimicrobial start decisions, infection was defined by the study ID specialist on 67/125 (54%) occasions, representing the clinicians' diagnostic accuracy. These infections included 17 (25%) respiratory, 16 (24%) abdominal, 13 (19%) soft tissue, 11 (16%) blood stream, 6 (9%) urinary, and 4 (6%) other infections.

Three attending clinicians treated patients during the study period, and their accuracies were similar (21 infections defined/44 start decisions for suspected infection, 48%; 24/38, 63%; 22/43, 51%, for each attending; P = ns for all comparisons). Clinician accuracy was higher for empirical antimicrobial start decisions, made within 48 hours of ICU admission, compared to later decisions (35 defined infections/53 early antibiotic start decisions [66%] vs 32 defined infections/72 late antibiotic start decisions [44%]; P = 0.02).

In a sensitivity analysis, decreasing the cutoff for the study ID specialist's definition of infection from probable (and above) to possible (and above) lead to reclassification of 14/125 (11%) antimicrobial start decisions from no infection defined to infection defined. This increased physician accuracy from 67/125 (54%) to 78/125 (62%), and conversely decreased potential antimicrobial overuse from 58/125 (46%) to 47/125 (38%) decisions (P = ns).

When starting antimicrobials for suspected infection, the clinicians were asked to record their certainty in the presence of infection. Infections were defined on 6/19 (31%) occasions when the clinician certainty score was low (2) versus 61/106 (57%) when the clinician certainty score was high (3, P = 0.037; Figure 2). Correlation between the clinician certainty score and the presence of defined infection was good (r² = 0.78).

Figure 2

The study ID specialist agreed with the clinician's decision to start antimicrobial therapy on 87/125 (70%) occasions. Infection was subsequently defined on 66/87 (76%) occasions, representing the study ID specialist's diagnostic accuracy. The study ID specialist's accuracy was significantly higher than the clinician's (66/87 [76%] versus 67/125 [54%]; P = 0.001). Notably, there was only 1 case (3%) where empiric therapy was deemed unnecessary by the study ID specialist, and where infection was subsequently defined. In this case, the clinicians started antibiotic therapy for suspected ventilator‐associated pneumonia in a 66‐year‐old patient on the 28th day of an ICU admission for head and spinal cord trauma. The ID specialist's certainty for the presence of infection was 3probable. The patient ultimately survived and was discharged to a rehabilitation facility.

Comparing physiological data for antimicrobial start decisions with high clinician certainty (score 3) versus low certainty of infection (score 2), revealed that none of the physiological data, nor changes over time were significantly associated with clinician certainty. Further use of high doses of vasopressors (>0.1 mcg/kg/min, SOFA score 4) was present at 42/106 (40%) high certainty decisions versus 7/19 (37%) low certainty decisions (P = 0.819). This underscores the physicians' difficulty in distinguishing between infectious and inflammatory causes of deterioration (Table 2).

During the study period, 2541 days of antimicrobial therapy were given of which 1677 (66%), 413 (16%), and 451 (18%) were given, respectively, empirically (for suspected infection), for procedural prophylaxis, and as targeted therapy. Antimicrobial course length was 11.5 9.2 days in the presence of defined infection versus 10.7 9.1 days in the absence of defined infection (P = 0.655). Overall, 658/2541 (26%) days of therapy could potentially have been saved by reducing antimicrobial prescriptions for suspected infections which were not defined.

DISCUSSION

The use of empirical antimicrobials could be justified by the presence of defined infection on only 54% of occasions when they were administered, suggesting considerable potential overuse of these drugs. ICU‐clinician certainty for the presence of infection correlated well with the number of infections actually defined, however, infections were defined when certainty was low (Figure 2) and antimicrobials prescribed even when clinician certainty was minimal. Common clinical physiological and laboratory parameters did not seem to assist in the clinicians' decision‐making, as there were no significant differences in any of these values between empiric decisions with high or low certainty. The study ID specialist showed significantly better accuracy in antimicrobial decision‐making than the ICU clinicians. He agreed with antimicrobial administration on only 70% of occasions that clinicians started empiric therapy, and had a higher diagnostic accuracy at a cost of only 1 untreated infection.

Two main possibilities are suggested to explain the potential antimicrobial overuse. First, ICU physicians are loath to leave infections untreated and potentially cause immediate increases in mortality.8 This leads to uncertainty avoidance or risk aversive behavior that is demonstrated in our study by the inclusion of antimicrobial administration decisions made even when physicians' certainty regarding the presence of infection was low. Uncertainty avoidance has been shown to be significantly associated with antimicrobial prescribing practices,13 however, it discounts the risk of antimicrobial complications associated with unnecessary antimicrobial therapy. Second, the diagnosis of infection, and particularly nosocomial infection, in ICU patients is difficult. Symptoms cannot be elicited in obtunded ventilated ICU patients, the physical exam can be equivocal, bacterial growth in cultures (with the exception of blood cultures) often reflects colonization rather than infection, and the laboratory and imaging findings of inflammation and infection are very similar. Our data demonstrated some of these difficulties. Diagnostic accuracy was higher in infections suspected during the first 48 hours of ICU admission when compared to later, presumably as infection leading to ICU admission is associated with symptoms, signs, and an acute change in the patient's condition, factors that may be absent when a patient develops a nosocomial infection. Further, physiological parameters did not correlate with the certainty that ICU clinicians expressed in their decision, indicating the difficulty in interpreting these data. Finally, infection was defined in 30% of low certainty decisions, indicating that clinical impression alone is not a reliable tool for determining the presence of infection.

Three sets of interventions could be suggested to improve antimicrobial decision‐makingincreased use of the ID consult, improved laboratory tests for the diagnosis of infection, and a policy of de‐escalation. Use of antimicrobial stewardship (often through involvement of an ID physician) reduces antimicrobial usage and the occurrence of resistant bacteria without adverse patient outcomes.14 Indeed, our study ID consult showed more accurate antimicrobial prescribing than the ICU clinicians, although he may have been subject to the potential biases described below. All antimicrobial administration decisions taken during the study were, however, made in consultation with the clinical ID consult. The lower performance of the clinical ID consult (when compared to the study ID consult) may have resulted from difficulties in the real‐time interaction with the clinicians or from decisions taken during non‐office hours. During non‐office hours, the on‐call ICU resident presented cases to an on‐call ID specialist, neither of whom may have been familiar with all the complex case details and therefore may have preferred to err by commission than by omission.

More accurate laboratory tests, such as procalcitonin or real‐time bacterial polymerase chain reaction (PCR),15 could be beneficial as they might increase physician confidence in decision‐making. Procalcitonin has been used in a wide variety of settings1618 (including the ICU1921) to safely decrease antimicrobial starts and/or antimicrobial course length. Despite this, in a large multicenter study of procalcitonin use in ICU patients,19 compliance with the antibiotic start protocol was very low. Antibiotics were administered by the participating physicians in 73/93 (78%) cases where the procalcitonin tests indicated that antimicrobials were not required, representing protocol violations. In parallel to our study, this demonstrates the reluctance of physicians to abstain from prescribing antimicrobial therapy for suspected infection even when the likelihood of infection may be low.

A strategy of de‐escalation offers the possibility of starting broad‐spectrum antimicrobials early and, subsequently, narrowing or stopping therapy according to the clinical course and the microbiological results.22, 23 This strategy allows clinicians to start antimicrobials even when the suspicion of infection is low, but to stop them rapidly as the clinical picture clarifies. Unfortunately, the mean antimicrobial course length in this study was not influenced by the presence of infection, indicating that this strategy was not employed successfully.

The proportion of patients prescribed antimicrobials for suspected infection in our study is similar to that found in others (eg, 34% of patients in a large French survey24). The proportion of potentially unnecessary antimicrobials in other studies is also similar, ranging from 14% to 50%.2428 In the ICU, the majority of antimicrobial usage studies are microbiology‐based and examine whether bacteria cultured are resistant to the antimicrobials chosen. They have shown that inappropriate antimicrobial therapy occurs on 20%36% of occasions.6, 7, 29 The current study furthers knowledge on antimicrobials decision‐making in the ICU, by examining the actual requirement for antimicrobial therapy based on the presence of infection, ie, whether antimicrobials were needed at all.

The principal limitation of the study concerns the determination of the presence of infection. The study premise was that antimicrobials are overused, and this may have biased the study ID consult to underestimate appropriateness of antimicrobial therapy and to define fewer infections. Further, making theoretical decisions in the research office avoids the medical, ethical, and legal issues related to clinical practice, as there is no risk associated with error. This may have allowed the study ID specialist to be overly conservative in his definitions of infections. A wider team of decision‐makers to determine the presence of infection, including both ID and ICU specialists, would have lent more weight to their determinations, however, this was logistically impossible. To limit the potential bias, infections were defined as objectively as possible based on the CDC criteria.12 Further, the sensitivity analysis showed that while decreasing the study ID specialist's threshold for the definition of infection from probable and above to possible and above improved physician accuracy, over a third of antimicrobial start decisions remained unjustified by the presence of defined infection. The study was performed in only 1 center and may not reflect general ICU practice, although, as discussed above, the antibiotic decision‐making accuracy is in the same orders of magnitude as those found in other somewhat similar studies. Finally, even if unnecessary antimicrobial use was overestimated, the possibility for significant improvement in antimicrobial administration accuracy remains.

In conclusion, our data suggest that on up to 46% of occasions, empirical antimicrobials are prescribed in the absence of infection. We suggest that the potential antibiotic overuse results from difficulties in diagnosing ICU‐related infections, and from the high perceived risk of untreated infection as compared to the risks of potentially unnecessary antimicrobial therapy, representing a type of risk aversive behavior. As antimicrobial use is the primary factor promoting antibiotic resistance and may be a cause of other patient complications, efforts to improve antimicrobial‐related decision‐making should be mandatory.