Zhenqiu Lin, PhD

Algorithm Development

To create a planned readmission algorithm, we first defined planned. We determined that readmissions for obstetrical delivery, maintenance chemotherapy, major organ transplant, and rehabilitation should always be considered planned in the sense that they are desired and/or inevitable, even if not specifically planned on a certain date. Apart from these specific types of readmissions, we defined planned readmissions as nonacute readmissions for scheduled procedures, because the vast majority of planned admissions are related to procedures. We also defined readmissions for acute illness or for complications of care as unplanned for the purposes of a quality measure. Even if such readmissions included a potentially planned procedure, because complications of care represent an important dimension of quality that should not be excluded from outcome measurement, these admissions should not be removed from the measure outcome. This definition of planned readmissions does not imply that all unplanned readmissions are unexpected or avoidable. However, it has proven very difficult to reliably define avoidable readmissions, even by expert review of charts, and we did not attempt to do so here.[6, 7]

In the second stage, we operationalized this definition into an algorithm. We used the Agency for Healthcare Research and Quality's Clinical Classification Software (CCS) codes to group thousands of individual procedure and diagnosis International Classification of Disease, Ninth Revision, Clinical Modification (ICD‐9‐CM) codes into clinically coherent, mutually exclusive procedure CCS categories and mutually exclusive diagnosis CCS categories, respectively. Clinicians on the investigative team reviewed the procedure categories to identify those that are commonly planned and that would require inpatient admission. We also reviewed the diagnosis categories to identify acute diagnoses unlikely to accompany elective procedures. We then created a flow diagram through which every readmission could be run to determine whether it was planned or unplanned based on our categorizations of procedures and diagnoses (Figure 1, and Supporting Information, Appendix A, in the online version of this article). This version of the algorithm (v1.0) was submitted to the National Quality Forum (NQF) as part of the hospital‐wide readmission measure. The measure (NQR #1789) received endorsement in April 2012.

Figure 1

In the third stage of development, we posted the algorithm for 2 public comment periods and recruited 27 outside experts to review and refine the algorithm following a standardized, structured process (see Supporting Information, Appendix B, in the online version of this article). Because the measures publicly report and hold hospitals accountable for unplanned readmission rates, we felt it most important that the algorithm include as few planned readmissions in the reported, unplanned outcome as possible (ie, have high negative predictive value). Therefore, in equivocal situations in which experts felt procedure categories were equally often planned or unplanned, we added those procedures to the potentially planned list. We also solicited feedback from hospitals on algorithm performance during a confidential test run of the hospital‐wide readmission measure in the fall of 2012. Based on all of this feedback, we made a number of changes to the algorithm, which was then identified as v2.1. Version 2.1 of the algorithm was submitted to the NQF as part of the endorsement process for the acute myocardial infarction and heart failure readmission measures and was endorsed by the NQF in January 2013. The algorithm (v2.1) is now applied, adapted if necessary, to all publicly reported readmission measures.[8]

Algorithm Validation: Study Cohort

We recruited 2 hospital systems to participate in a chart validation study of the accuracy of the planned readmission algorithm (v2.1). Within these 2 health systems, we selected 7 hospitals with varying bed size, teaching status, and safety‐net status. Each included 1 large academic teaching hospital that serves as a regional referral center. For each hospital's index admissions, we applied the inclusion and exclusion criteria from the hospital‐wide readmission measure. Index admissions were included for patients age 65 years or older; enrolled in Medicare fee‐for‐service (FFS); discharged from a nonfederal, short‐stay, acute‐care hospital or critical access hospital; without an in‐hospital death; not transferred to another acute‐care facility; and enrolled in Part A Medicare for 1 year prior to discharge. We excluded index admissions for patients without at least 30 days postdischarge enrollment in FFS Medicare, discharged against medical advice, admitted for medical treatment of cancer or primary psychiatric disease, admitted to a Prospective Payment System‐exempt cancer hospital, or who died during the index hospitalization. In addition, for this study, we included only index admissions that were followed by a readmission to a hospital within the participating health system between July 1, 2011 and June 30, 2012. Institutional review board approval was obtained from each of the participating health systems, which granted waivers of signed informed consent and Health Insurance Portability and Accountability Act waivers.

Algorithm Validation: Sample Size Calculation

We determined a priori that the minimum acceptable positive predictive value, or proportion of all readmissions the algorithm labels planned that are truly planned, would be 60%, and the minimum acceptable negative predictive value, or proportion of all readmissions the algorithm labels as unplanned that are truly unplanned, would be 80%. We calculated the sample size required to be confident of these values 10% and determined we would need a total of 291 planned charts and 162 unplanned charts. We inflated these numbers by 20% to account for missing or unobtainable charts for a total of 550 charts. To achieve this sample size, we included all eligible readmissions from all participating hospitals that were categorized as planned. At the 5 smaller hospitals, we randomly selected an equal number of unplanned readmissions occurring at any hospital in its healthcare system. At the 2 largest hospitals, we randomly selected 50 unplanned readmissions occurring at any hospital in its healthcare system.

Algorithm Validation: Data Abstraction

We developed an abstraction tool, tested and refined it using sample charts, and built the final the tool into a secure, password‐protected Microsoft Access 2007 (Microsoft Corp., Redmond, WA) database (see Supporting Information, Appendix C, in the online version of this article). Experienced chart abstractors with RN or MD degrees from each hospital site participated in a 1‐hour training session to become familiar with reviewing medical charts, defining planned/unplanned readmissions, and the data abstraction process. For each readmission, we asked abstractors to review as needed: emergency department triage and physician notes, admission history and physical, operative report, discharge summary, and/or discharge summary from a prior admission. The abstractors verified the accuracy of the administrative billing data, including procedures and principal diagnosis. In addition, they abstracted the source of admission and dates of all major procedures. Then the abstractors provided their opinion and supporting rationale as to whether a readmission was planned or unplanned. They were not asked to determine whether the readmission was preventable. To determine the inter‐rater reliability of data abstraction, an independent abstractor at each health system recoded a random sample of 10% of the charts.

Statistical Analysis

To ensure that we had obtained a representative sample of charts, we identified the 10 most commonly planned procedures among cases identified as planned by the algorithm in the validation cohort and then compared this with planned cases nationally. To confirm the reliability of the abstraction process, we used the kappa statistic to determine the inter‐rater reliability of the determination of planned or unplanned status. Additionally, the full study team, including 5 practicing clinicians, reviewed the details of every chart abstraction in which the algorithm was found to have misclassified the readmission as planned or unplanned. In 11 cases we determined that the abstractor had misunderstood the definition of planned readmission (ie, not all direct admissions are necessarily planned) and we reclassified the chart review assignment accordingly.

We calculated sensitivity, specificity, positive predictive value, and negative predictive value of the algorithm for the validation cohort as a whole, weighted to account for the prevalence of planned readmissions as defined by the algorithm in the national data (7.8%). Weighting is necessary because we did not obtain a pure random sample, but rather selected a stratified sample that oversampled algorithm‐identified planned readmissions.[9] We also calculated these rates separately for large hospitals (>600 beds) and for small hospitals (600 beds).

Finally, we examined performance of the algorithm for individual procedures and diagnoses to determine whether any procedures or diagnoses should be added or removed from the algorithm. First, we reviewed the diagnoses, procedures, and brief narratives provided by the abstractors for all cases in which the algorithm misclassified the readmission as either planned or unplanned. Second, we calculated the positive predictive value for each procedure that had been flagged as planned by the algorithm, and reviewed all readmissions (correctly and incorrectly classified) in which procedures with low positive predictive value took place. We also calculated the frequency with which the procedure was the only qualifying procedure resulting in an accurate or inaccurate classification. Third, to identify changes that should be made to the lists of acute and nonacute diagnoses, we reviewed the principal diagnosis for all readmissions misclassified by the algorithm as either planned or unplanned, and examined the specific ICD‐9‐CM codes within each CCS group that were most commonly associated with misclassifications.

After determining the changes that should be made to the algorithm based on these analyses, we recalculated the sensitivity, specificity, positive predictive value, and negative predictive value of the proposed revised algorithm (v3.0). All analyses used SAS version 9.3 (SAS Institute, Cary, NC).

Study Cohort

Characteristics of participating hospitals are shown in Table 1. Hospitals represented in this sample ranged in size, teaching status, and safety net status, although all were nonprofit. We selected 663 readmissions for review, 363 planned and 300 unplanned. Overall we were able to select 80% of hospitals planned cases for review; the remainder occurred at hospitals outside the participating hospital system. Abstractors were able to locate and review 634 (96%) of the eligible charts (range, 86%100% per hospital). The kappa statistic for inter‐rater reliability was 0.83.

The study sample included 57/67 (85%) of the procedure or condition categories on the potentially planned list. The most common procedure CCS categories among planned readmissions (v2.1) in the validation cohort were very similar to those in the national dataset (see Supporting Information, Appendix D, in the online version of this article). Of the top 20 most commonly planned procedure CCS categories in the validation set, all but 2, therapeutic radiology for cancer treatment (CCS 211) and peripheral vascular bypass (CCS 55), were among the top 20 most commonly planned procedure CCS categories in the national data.

Test Characteristics of Algorithm

The weighted test characteristics of the current algorithm (v2.1) are shown in Table 2. Overall, the algorithm correctly identified 266 readmissions as unplanned and 181 readmissions as planned, and misidentified 170 readmissions as planned and 15 as unplanned. Once weighted to account for the stratified sampling design, the overall prevalence of true planned readmissions was 8.9% of readmissions. The weighted sensitivity was 45.1% overall and was higher in large teaching centers than in smaller community hospitals. The weighted specificity was 95.9%. The positive predictive value was 51.6%, and the negative predictive value was 94.7%.

Accuracy of Individual Diagnoses and Procedures

The positive predictive value of the algorithm for individual procedure categories varied widely, from 0% to 100% among procedures with at least 10 cases (Table 3). The procedure for which the algorithm was least accurate was CCS 211, therapeutic radiology for cancer treatment (0% positive predictive value). By contrast, maintenance chemotherapy (90%) and other therapeutic procedures, hemic and lymphatic system (100%) were most accurate. Common procedures with less than 50% positive predictive value (ie, that the algorithm commonly misclassified as planned) were diagnostic cardiac catheterization (25%); debridement of wound, infection, or burn (25%); amputation of lower extremity (29%); insertion, revision, replacement, removal of cardiac pacemaker or cardioverter/defibrillator (33%); and other hernia repair (43%). Of these, diagnostic cardiac catheterization and cardiac devices are the first and second most common procedures nationally, respectively.

The readmissions with least abstractor agreement were those involving CCS 157 (amputation of lower extremity) and CCS 169 (debridement of wound, infection or burn). Readmissions for these procedures were nearly always performed as a consequence of acute worsening of chronic conditions such as osteomyelitis or ulceration. Abstractors were divided over whether these readmissions were appropriate to call planned.

Changes to the Algorithm

We determined that the accuracy of the algorithm would be improved by removing 2 procedure categories from the planned procedure list (therapeutic radiation [CCS 211] and cancer chemotherapy [CCS 224]), adding 1 diagnosis category to the acute diagnosis list (hypertension with complications [CCS 99]), and splitting 2 diagnosis condition categories into acute and nonacute ICD‐9‐CM codes (pancreatic disorders [CCS 149] and biliary tract disease [CCS 152]). Detailed rationales for each modification to the planned readmission algorithm are described in Table 4. We felt further examination of diagnostic cardiac catheterization and cardiac devices was warranted given their high frequency, despite low positive predictive value. We also elected not to alter the categorization of amputation or debridement because it was not easy to determine whether these admissions were planned or unplanned even with chart review. We plan further analyses of these procedure categories.

The revised algorithm (v3.0) had a weighted sensitivity of 49.8%, weighted specificity of 96.5%, positive predictive value of 58.7%, and negative predictive value of 94.5% (Table 2). In aggregate, these changes would increase the reported unplanned readmission rate from 16.0% to 16.1% in the hospital‐wide readmission measure, using 2011 to 2012 data, and would decrease the fraction of all readmissions considered planned from 7.8% to 7.2%.

Algorithm Development

To create a planned readmission algorithm, we first defined planned. We determined that readmissions for obstetrical delivery, maintenance chemotherapy, major organ transplant, and rehabilitation should always be considered planned in the sense that they are desired and/or inevitable, even if not specifically planned on a certain date. Apart from these specific types of readmissions, we defined planned readmissions as nonacute readmissions for scheduled procedures, because the vast majority of planned admissions are related to procedures. We also defined readmissions for acute illness or for complications of care as unplanned for the purposes of a quality measure. Even if such readmissions included a potentially planned procedure, because complications of care represent an important dimension of quality that should not be excluded from outcome measurement, these admissions should not be removed from the measure outcome. This definition of planned readmissions does not imply that all unplanned readmissions are unexpected or avoidable. However, it has proven very difficult to reliably define avoidable readmissions, even by expert review of charts, and we did not attempt to do so here.[6, 7]

In the second stage, we operationalized this definition into an algorithm. We used the Agency for Healthcare Research and Quality's Clinical Classification Software (CCS) codes to group thousands of individual procedure and diagnosis International Classification of Disease, Ninth Revision, Clinical Modification (ICD‐9‐CM) codes into clinically coherent, mutually exclusive procedure CCS categories and mutually exclusive diagnosis CCS categories, respectively. Clinicians on the investigative team reviewed the procedure categories to identify those that are commonly planned and that would require inpatient admission. We also reviewed the diagnosis categories to identify acute diagnoses unlikely to accompany elective procedures. We then created a flow diagram through which every readmission could be run to determine whether it was planned or unplanned based on our categorizations of procedures and diagnoses (Figure 1, and Supporting Information, Appendix A, in the online version of this article). This version of the algorithm (v1.0) was submitted to the National Quality Forum (NQF) as part of the hospital‐wide readmission measure. The measure (NQR #1789) received endorsement in April 2012.

Figure 1

In the third stage of development, we posted the algorithm for 2 public comment periods and recruited 27 outside experts to review and refine the algorithm following a standardized, structured process (see Supporting Information, Appendix B, in the online version of this article). Because the measures publicly report and hold hospitals accountable for unplanned readmission rates, we felt it most important that the algorithm include as few planned readmissions in the reported, unplanned outcome as possible (ie, have high negative predictive value). Therefore, in equivocal situations in which experts felt procedure categories were equally often planned or unplanned, we added those procedures to the potentially planned list. We also solicited feedback from hospitals on algorithm performance during a confidential test run of the hospital‐wide readmission measure in the fall of 2012. Based on all of this feedback, we made a number of changes to the algorithm, which was then identified as v2.1. Version 2.1 of the algorithm was submitted to the NQF as part of the endorsement process for the acute myocardial infarction and heart failure readmission measures and was endorsed by the NQF in January 2013. The algorithm (v2.1) is now applied, adapted if necessary, to all publicly reported readmission measures.[8]

Algorithm Validation: Study Cohort

We recruited 2 hospital systems to participate in a chart validation study of the accuracy of the planned readmission algorithm (v2.1). Within these 2 health systems, we selected 7 hospitals with varying bed size, teaching status, and safety‐net status. Each included 1 large academic teaching hospital that serves as a regional referral center. For each hospital's index admissions, we applied the inclusion and exclusion criteria from the hospital‐wide readmission measure. Index admissions were included for patients age 65 years or older; enrolled in Medicare fee‐for‐service (FFS); discharged from a nonfederal, short‐stay, acute‐care hospital or critical access hospital; without an in‐hospital death; not transferred to another acute‐care facility; and enrolled in Part A Medicare for 1 year prior to discharge. We excluded index admissions for patients without at least 30 days postdischarge enrollment in FFS Medicare, discharged against medical advice, admitted for medical treatment of cancer or primary psychiatric disease, admitted to a Prospective Payment System‐exempt cancer hospital, or who died during the index hospitalization. In addition, for this study, we included only index admissions that were followed by a readmission to a hospital within the participating health system between July 1, 2011 and June 30, 2012. Institutional review board approval was obtained from each of the participating health systems, which granted waivers of signed informed consent and Health Insurance Portability and Accountability Act waivers.

Algorithm Validation: Sample Size Calculation

We determined a priori that the minimum acceptable positive predictive value, or proportion of all readmissions the algorithm labels planned that are truly planned, would be 60%, and the minimum acceptable negative predictive value, or proportion of all readmissions the algorithm labels as unplanned that are truly unplanned, would be 80%. We calculated the sample size required to be confident of these values 10% and determined we would need a total of 291 planned charts and 162 unplanned charts. We inflated these numbers by 20% to account for missing or unobtainable charts for a total of 550 charts. To achieve this sample size, we included all eligible readmissions from all participating hospitals that were categorized as planned. At the 5 smaller hospitals, we randomly selected an equal number of unplanned readmissions occurring at any hospital in its healthcare system. At the 2 largest hospitals, we randomly selected 50 unplanned readmissions occurring at any hospital in its healthcare system.

Algorithm Validation: Data Abstraction

We developed an abstraction tool, tested and refined it using sample charts, and built the final the tool into a secure, password‐protected Microsoft Access 2007 (Microsoft Corp., Redmond, WA) database (see Supporting Information, Appendix C, in the online version of this article). Experienced chart abstractors with RN or MD degrees from each hospital site participated in a 1‐hour training session to become familiar with reviewing medical charts, defining planned/unplanned readmissions, and the data abstraction process. For each readmission, we asked abstractors to review as needed: emergency department triage and physician notes, admission history and physical, operative report, discharge summary, and/or discharge summary from a prior admission. The abstractors verified the accuracy of the administrative billing data, including procedures and principal diagnosis. In addition, they abstracted the source of admission and dates of all major procedures. Then the abstractors provided their opinion and supporting rationale as to whether a readmission was planned or unplanned. They were not asked to determine whether the readmission was preventable. To determine the inter‐rater reliability of data abstraction, an independent abstractor at each health system recoded a random sample of 10% of the charts.

Statistical Analysis

To ensure that we had obtained a representative sample of charts, we identified the 10 most commonly planned procedures among cases identified as planned by the algorithm in the validation cohort and then compared this with planned cases nationally. To confirm the reliability of the abstraction process, we used the kappa statistic to determine the inter‐rater reliability of the determination of planned or unplanned status. Additionally, the full study team, including 5 practicing clinicians, reviewed the details of every chart abstraction in which the algorithm was found to have misclassified the readmission as planned or unplanned. In 11 cases we determined that the abstractor had misunderstood the definition of planned readmission (ie, not all direct admissions are necessarily planned) and we reclassified the chart review assignment accordingly.

We calculated sensitivity, specificity, positive predictive value, and negative predictive value of the algorithm for the validation cohort as a whole, weighted to account for the prevalence of planned readmissions as defined by the algorithm in the national data (7.8%). Weighting is necessary because we did not obtain a pure random sample, but rather selected a stratified sample that oversampled algorithm‐identified planned readmissions.[9] We also calculated these rates separately for large hospitals (>600 beds) and for small hospitals (600 beds).

Finally, we examined performance of the algorithm for individual procedures and diagnoses to determine whether any procedures or diagnoses should be added or removed from the algorithm. First, we reviewed the diagnoses, procedures, and brief narratives provided by the abstractors for all cases in which the algorithm misclassified the readmission as either planned or unplanned. Second, we calculated the positive predictive value for each procedure that had been flagged as planned by the algorithm, and reviewed all readmissions (correctly and incorrectly classified) in which procedures with low positive predictive value took place. We also calculated the frequency with which the procedure was the only qualifying procedure resulting in an accurate or inaccurate classification. Third, to identify changes that should be made to the lists of acute and nonacute diagnoses, we reviewed the principal diagnosis for all readmissions misclassified by the algorithm as either planned or unplanned, and examined the specific ICD‐9‐CM codes within each CCS group that were most commonly associated with misclassifications.

After determining the changes that should be made to the algorithm based on these analyses, we recalculated the sensitivity, specificity, positive predictive value, and negative predictive value of the proposed revised algorithm (v3.0). All analyses used SAS version 9.3 (SAS Institute, Cary, NC).

Study Cohort

Characteristics of participating hospitals are shown in Table 1. Hospitals represented in this sample ranged in size, teaching status, and safety net status, although all were nonprofit. We selected 663 readmissions for review, 363 planned and 300 unplanned. Overall we were able to select 80% of hospitals planned cases for review; the remainder occurred at hospitals outside the participating hospital system. Abstractors were able to locate and review 634 (96%) of the eligible charts (range, 86%100% per hospital). The kappa statistic for inter‐rater reliability was 0.83.

The study sample included 57/67 (85%) of the procedure or condition categories on the potentially planned list. The most common procedure CCS categories among planned readmissions (v2.1) in the validation cohort were very similar to those in the national dataset (see Supporting Information, Appendix D, in the online version of this article). Of the top 20 most commonly planned procedure CCS categories in the validation set, all but 2, therapeutic radiology for cancer treatment (CCS 211) and peripheral vascular bypass (CCS 55), were among the top 20 most commonly planned procedure CCS categories in the national data.

Test Characteristics of Algorithm

The weighted test characteristics of the current algorithm (v2.1) are shown in Table 2. Overall, the algorithm correctly identified 266 readmissions as unplanned and 181 readmissions as planned, and misidentified 170 readmissions as planned and 15 as unplanned. Once weighted to account for the stratified sampling design, the overall prevalence of true planned readmissions was 8.9% of readmissions. The weighted sensitivity was 45.1% overall and was higher in large teaching centers than in smaller community hospitals. The weighted specificity was 95.9%. The positive predictive value was 51.6%, and the negative predictive value was 94.7%.

Accuracy of Individual Diagnoses and Procedures

The positive predictive value of the algorithm for individual procedure categories varied widely, from 0% to 100% among procedures with at least 10 cases (Table 3). The procedure for which the algorithm was least accurate was CCS 211, therapeutic radiology for cancer treatment (0% positive predictive value). By contrast, maintenance chemotherapy (90%) and other therapeutic procedures, hemic and lymphatic system (100%) were most accurate. Common procedures with less than 50% positive predictive value (ie, that the algorithm commonly misclassified as planned) were diagnostic cardiac catheterization (25%); debridement of wound, infection, or burn (25%); amputation of lower extremity (29%); insertion, revision, replacement, removal of cardiac pacemaker or cardioverter/defibrillator (33%); and other hernia repair (43%). Of these, diagnostic cardiac catheterization and cardiac devices are the first and second most common procedures nationally, respectively.

The readmissions with least abstractor agreement were those involving CCS 157 (amputation of lower extremity) and CCS 169 (debridement of wound, infection or burn). Readmissions for these procedures were nearly always performed as a consequence of acute worsening of chronic conditions such as osteomyelitis or ulceration. Abstractors were divided over whether these readmissions were appropriate to call planned.

Changes to the Algorithm

We determined that the accuracy of the algorithm would be improved by removing 2 procedure categories from the planned procedure list (therapeutic radiation [CCS 211] and cancer chemotherapy [CCS 224]), adding 1 diagnosis category to the acute diagnosis list (hypertension with complications [CCS 99]), and splitting 2 diagnosis condition categories into acute and nonacute ICD‐9‐CM codes (pancreatic disorders [CCS 149] and biliary tract disease [CCS 152]). Detailed rationales for each modification to the planned readmission algorithm are described in Table 4. We felt further examination of diagnostic cardiac catheterization and cardiac devices was warranted given their high frequency, despite low positive predictive value. We also elected not to alter the categorization of amputation or debridement because it was not easy to determine whether these admissions were planned or unplanned even with chart review. We plan further analyses of these procedure categories.

The revised algorithm (v3.0) had a weighted sensitivity of 49.8%, weighted specificity of 96.5%, positive predictive value of 58.7%, and negative predictive value of 94.5% (Table 2). In aggregate, these changes would increase the reported unplanned readmission rate from 16.0% to 16.1% in the hospital‐wide readmission measure, using 2011 to 2012 data, and would decrease the fraction of all readmissions considered planned from 7.8% to 7.2%.

Algorithm Development

To create a planned readmission algorithm, we first defined planned. We determined that readmissions for obstetrical delivery, maintenance chemotherapy, major organ transplant, and rehabilitation should always be considered planned in the sense that they are desired and/or inevitable, even if not specifically planned on a certain date. Apart from these specific types of readmissions, we defined planned readmissions as nonacute readmissions for scheduled procedures, because the vast majority of planned admissions are related to procedures. We also defined readmissions for acute illness or for complications of care as unplanned for the purposes of a quality measure. Even if such readmissions included a potentially planned procedure, because complications of care represent an important dimension of quality that should not be excluded from outcome measurement, these admissions should not be removed from the measure outcome. This definition of planned readmissions does not imply that all unplanned readmissions are unexpected or avoidable. However, it has proven very difficult to reliably define avoidable readmissions, even by expert review of charts, and we did not attempt to do so here.[6, 7]

In the second stage, we operationalized this definition into an algorithm. We used the Agency for Healthcare Research and Quality's Clinical Classification Software (CCS) codes to group thousands of individual procedure and diagnosis International Classification of Disease, Ninth Revision, Clinical Modification (ICD‐9‐CM) codes into clinically coherent, mutually exclusive procedure CCS categories and mutually exclusive diagnosis CCS categories, respectively. Clinicians on the investigative team reviewed the procedure categories to identify those that are commonly planned and that would require inpatient admission. We also reviewed the diagnosis categories to identify acute diagnoses unlikely to accompany elective procedures. We then created a flow diagram through which every readmission could be run to determine whether it was planned or unplanned based on our categorizations of procedures and diagnoses (Figure 1, and Supporting Information, Appendix A, in the online version of this article). This version of the algorithm (v1.0) was submitted to the National Quality Forum (NQF) as part of the hospital‐wide readmission measure. The measure (NQR #1789) received endorsement in April 2012.

Figure 1

In the third stage of development, we posted the algorithm for 2 public comment periods and recruited 27 outside experts to review and refine the algorithm following a standardized, structured process (see Supporting Information, Appendix B, in the online version of this article). Because the measures publicly report and hold hospitals accountable for unplanned readmission rates, we felt it most important that the algorithm include as few planned readmissions in the reported, unplanned outcome as possible (ie, have high negative predictive value). Therefore, in equivocal situations in which experts felt procedure categories were equally often planned or unplanned, we added those procedures to the potentially planned list. We also solicited feedback from hospitals on algorithm performance during a confidential test run of the hospital‐wide readmission measure in the fall of 2012. Based on all of this feedback, we made a number of changes to the algorithm, which was then identified as v2.1. Version 2.1 of the algorithm was submitted to the NQF as part of the endorsement process for the acute myocardial infarction and heart failure readmission measures and was endorsed by the NQF in January 2013. The algorithm (v2.1) is now applied, adapted if necessary, to all publicly reported readmission measures.[8]

Algorithm Validation: Study Cohort

We recruited 2 hospital systems to participate in a chart validation study of the accuracy of the planned readmission algorithm (v2.1). Within these 2 health systems, we selected 7 hospitals with varying bed size, teaching status, and safety‐net status. Each included 1 large academic teaching hospital that serves as a regional referral center. For each hospital's index admissions, we applied the inclusion and exclusion criteria from the hospital‐wide readmission measure. Index admissions were included for patients age 65 years or older; enrolled in Medicare fee‐for‐service (FFS); discharged from a nonfederal, short‐stay, acute‐care hospital or critical access hospital; without an in‐hospital death; not transferred to another acute‐care facility; and enrolled in Part A Medicare for 1 year prior to discharge. We excluded index admissions for patients without at least 30 days postdischarge enrollment in FFS Medicare, discharged against medical advice, admitted for medical treatment of cancer or primary psychiatric disease, admitted to a Prospective Payment System‐exempt cancer hospital, or who died during the index hospitalization. In addition, for this study, we included only index admissions that were followed by a readmission to a hospital within the participating health system between July 1, 2011 and June 30, 2012. Institutional review board approval was obtained from each of the participating health systems, which granted waivers of signed informed consent and Health Insurance Portability and Accountability Act waivers.

Algorithm Validation: Sample Size Calculation

We determined a priori that the minimum acceptable positive predictive value, or proportion of all readmissions the algorithm labels planned that are truly planned, would be 60%, and the minimum acceptable negative predictive value, or proportion of all readmissions the algorithm labels as unplanned that are truly unplanned, would be 80%. We calculated the sample size required to be confident of these values 10% and determined we would need a total of 291 planned charts and 162 unplanned charts. We inflated these numbers by 20% to account for missing or unobtainable charts for a total of 550 charts. To achieve this sample size, we included all eligible readmissions from all participating hospitals that were categorized as planned. At the 5 smaller hospitals, we randomly selected an equal number of unplanned readmissions occurring at any hospital in its healthcare system. At the 2 largest hospitals, we randomly selected 50 unplanned readmissions occurring at any hospital in its healthcare system.

Algorithm Validation: Data Abstraction

We developed an abstraction tool, tested and refined it using sample charts, and built the final the tool into a secure, password‐protected Microsoft Access 2007 (Microsoft Corp., Redmond, WA) database (see Supporting Information, Appendix C, in the online version of this article). Experienced chart abstractors with RN or MD degrees from each hospital site participated in a 1‐hour training session to become familiar with reviewing medical charts, defining planned/unplanned readmissions, and the data abstraction process. For each readmission, we asked abstractors to review as needed: emergency department triage and physician notes, admission history and physical, operative report, discharge summary, and/or discharge summary from a prior admission. The abstractors verified the accuracy of the administrative billing data, including procedures and principal diagnosis. In addition, they abstracted the source of admission and dates of all major procedures. Then the abstractors provided their opinion and supporting rationale as to whether a readmission was planned or unplanned. They were not asked to determine whether the readmission was preventable. To determine the inter‐rater reliability of data abstraction, an independent abstractor at each health system recoded a random sample of 10% of the charts.

Statistical Analysis

To ensure that we had obtained a representative sample of charts, we identified the 10 most commonly planned procedures among cases identified as planned by the algorithm in the validation cohort and then compared this with planned cases nationally. To confirm the reliability of the abstraction process, we used the kappa statistic to determine the inter‐rater reliability of the determination of planned or unplanned status. Additionally, the full study team, including 5 practicing clinicians, reviewed the details of every chart abstraction in which the algorithm was found to have misclassified the readmission as planned or unplanned. In 11 cases we determined that the abstractor had misunderstood the definition of planned readmission (ie, not all direct admissions are necessarily planned) and we reclassified the chart review assignment accordingly.

We calculated sensitivity, specificity, positive predictive value, and negative predictive value of the algorithm for the validation cohort as a whole, weighted to account for the prevalence of planned readmissions as defined by the algorithm in the national data (7.8%). Weighting is necessary because we did not obtain a pure random sample, but rather selected a stratified sample that oversampled algorithm‐identified planned readmissions.[9] We also calculated these rates separately for large hospitals (>600 beds) and for small hospitals (600 beds).

Finally, we examined performance of the algorithm for individual procedures and diagnoses to determine whether any procedures or diagnoses should be added or removed from the algorithm. First, we reviewed the diagnoses, procedures, and brief narratives provided by the abstractors for all cases in which the algorithm misclassified the readmission as either planned or unplanned. Second, we calculated the positive predictive value for each procedure that had been flagged as planned by the algorithm, and reviewed all readmissions (correctly and incorrectly classified) in which procedures with low positive predictive value took place. We also calculated the frequency with which the procedure was the only qualifying procedure resulting in an accurate or inaccurate classification. Third, to identify changes that should be made to the lists of acute and nonacute diagnoses, we reviewed the principal diagnosis for all readmissions misclassified by the algorithm as either planned or unplanned, and examined the specific ICD‐9‐CM codes within each CCS group that were most commonly associated with misclassifications.

After determining the changes that should be made to the algorithm based on these analyses, we recalculated the sensitivity, specificity, positive predictive value, and negative predictive value of the proposed revised algorithm (v3.0). All analyses used SAS version 9.3 (SAS Institute, Cary, NC).

Study Cohort

Characteristics of participating hospitals are shown in Table 1. Hospitals represented in this sample ranged in size, teaching status, and safety net status, although all were nonprofit. We selected 663 readmissions for review, 363 planned and 300 unplanned. Overall we were able to select 80% of hospitals planned cases for review; the remainder occurred at hospitals outside the participating hospital system. Abstractors were able to locate and review 634 (96%) of the eligible charts (range, 86%100% per hospital). The kappa statistic for inter‐rater reliability was 0.83.

The study sample included 57/67 (85%) of the procedure or condition categories on the potentially planned list. The most common procedure CCS categories among planned readmissions (v2.1) in the validation cohort were very similar to those in the national dataset (see Supporting Information, Appendix D, in the online version of this article). Of the top 20 most commonly planned procedure CCS categories in the validation set, all but 2, therapeutic radiology for cancer treatment (CCS 211) and peripheral vascular bypass (CCS 55), were among the top 20 most commonly planned procedure CCS categories in the national data.

Test Characteristics of Algorithm

The weighted test characteristics of the current algorithm (v2.1) are shown in Table 2. Overall, the algorithm correctly identified 266 readmissions as unplanned and 181 readmissions as planned, and misidentified 170 readmissions as planned and 15 as unplanned. Once weighted to account for the stratified sampling design, the overall prevalence of true planned readmissions was 8.9% of readmissions. The weighted sensitivity was 45.1% overall and was higher in large teaching centers than in smaller community hospitals. The weighted specificity was 95.9%. The positive predictive value was 51.6%, and the negative predictive value was 94.7%.

Accuracy of Individual Diagnoses and Procedures

The positive predictive value of the algorithm for individual procedure categories varied widely, from 0% to 100% among procedures with at least 10 cases (Table 3). The procedure for which the algorithm was least accurate was CCS 211, therapeutic radiology for cancer treatment (0% positive predictive value). By contrast, maintenance chemotherapy (90%) and other therapeutic procedures, hemic and lymphatic system (100%) were most accurate. Common procedures with less than 50% positive predictive value (ie, that the algorithm commonly misclassified as planned) were diagnostic cardiac catheterization (25%); debridement of wound, infection, or burn (25%); amputation of lower extremity (29%); insertion, revision, replacement, removal of cardiac pacemaker or cardioverter/defibrillator (33%); and other hernia repair (43%). Of these, diagnostic cardiac catheterization and cardiac devices are the first and second most common procedures nationally, respectively.

The readmissions with least abstractor agreement were those involving CCS 157 (amputation of lower extremity) and CCS 169 (debridement of wound, infection or burn). Readmissions for these procedures were nearly always performed as a consequence of acute worsening of chronic conditions such as osteomyelitis or ulceration. Abstractors were divided over whether these readmissions were appropriate to call planned.

Changes to the Algorithm

We determined that the accuracy of the algorithm would be improved by removing 2 procedure categories from the planned procedure list (therapeutic radiation [CCS 211] and cancer chemotherapy [CCS 224]), adding 1 diagnosis category to the acute diagnosis list (hypertension with complications [CCS 99]), and splitting 2 diagnosis condition categories into acute and nonacute ICD‐9‐CM codes (pancreatic disorders [CCS 149] and biliary tract disease [CCS 152]). Detailed rationales for each modification to the planned readmission algorithm are described in Table 4. We felt further examination of diagnostic cardiac catheterization and cardiac devices was warranted given their high frequency, despite low positive predictive value. We also elected not to alter the categorization of amputation or debridement because it was not easy to determine whether these admissions were planned or unplanned even with chart review. We plan further analyses of these procedure categories.

The revised algorithm (v3.0) had a weighted sensitivity of 49.8%, weighted specificity of 96.5%, positive predictive value of 58.7%, and negative predictive value of 94.5% (Table 2). In aggregate, these changes would increase the reported unplanned readmission rate from 16.0% to 16.1% in the hospital‐wide readmission measure, using 2011 to 2012 data, and would decrease the fraction of all readmissions considered planned from 7.8% to 7.2%.

Data Sources

We used 20052006 claims data from Medicare inpatient, outpatient, and carrier (physician) Standard Analytic Files to develop and validate the administrative model. The Medicare Enrollment Database was used to determine Medicare fee‐for‐service enrollment and mortality statuses. A medical record model, used for additional validation of the administrative model, was developed using information abstracted from the charts of 75,616 pneumonia cases from 19982001 as part of the National Pneumonia Project, a CMS quality improvement initiative.18

Study Cohort

We identified hospitalizations of patients 65 years of age and older with a principal diagnosis of pneumonia (International Classification of Diseases, 9th Revision, Clinical Modification codes 480.XX, 481, 482.XX, 483.X, 485, 486, 487.0) as potential index pneumonia admissions. Because our focus was readmission for patients discharged from acute care settings, we excluded admissions in which patients died or were transferred to another acute care facility. Additionally, we restricted analysis to patients who had been enrolled in fee‐for‐service Medicare Parts A and B, for at least 12 months prior to their pneumonia hospitalization, so that we could use diagnostic codes from all inpatient and outpatient encounters during that period to enhance identification of comorbidities.

Outcome

The outcome was 30‐day readmission, defined as occurrence of at least one hospitalization for any cause within 30 days of discharge after an index admission. Readmissions were identified from hospital claims data, and were attributed to the hospital that had discharged the patient. A 30‐day time frame was selected because it is a clinically meaningful period during which hospitals can be expected to collaborate with other organizations and providers to implement measures to reduce the risk of rehospitalization.

Candidate and Final Model Variables

Candidate variables for the administrative claims model were selected by a clinician team from 189 diagnostic groups included in the Hierarchical Condition Category (HCC) clinical classification system.19 The HCC clinical classification system was developed for CMS in preparation for all‐encounter risk adjustment for Medicare Advantage (managed care). Under the HCC algorithm, the 15,000+ ICD‐9‐CM diagnosis codes are assigned to one of 189 clinically‐coherent condition categories (CCs). We used the April 2008 version of the ICD‐9‐CM to CC assignment map, which is maintained by CMS and posted at http://www.qualitynet.org. A total of 154 CCs were considered to be potentially relevant to readmission outcome and were included for further consideration. Some CCs were further combined into clinically coherent groupings of CCs. Our set of candidate variables ultimately included 97 CC‐based variables, two demographic variables (age and sex), and two procedure codes potentially relevant to readmission risk (history of percutaneous coronary intervention [PCI] and history of coronary artery bypass graft [CABG]).

The final risk‐adjustment model included 39 variables selected by the team of clinicians and analysts, primarily based on their clinical relevance but also with knowledge of the strength of their statistical association with readmission outcome (Table 1). For each patient, the presence or absence of these conditions was assessed from multiple sources, including secondary diagnoses during the index admission, principal and secondary diagnoses from hospital admissions in the 12 months prior to the index admission, and diagnoses from hospital outpatient and physician encounters 12 months before the index admission. A small number of CCs were considered to represent potential complications of care (eg, bleeding). Because we did not want to adjust for complications of care occurring during the index admission, a patient was not considered to have one of these conditions unless it was also present in at least one encounter prior to the index admission.

Model Derivation

For the development of the administrative claims model, we randomly sampled half of 2006 hospitalizations that met inclusion criteria. To assess model performance at the patient level, we calculated the area under the receiver operating curve (AUC), and calculated observed readmission rates in the lowest and highest deciles on the basis of predicted readmission probabilities. We also compared performance with a null model, a model that adjusted for age and sex, and a model that included all candidate variables.20

Risk‐Standardized Readmission Rates

Using hierarchical logistic regression, we modeled the log‐odds of readmission within 30 days of discharge from an index pneumonia admission as a function of patient demographic and clinical characteristics, and a random hospital‐specific intercept. This strategy accounts for within‐hospital correlation, or clustering, of observed outcomes, and models the assumption that underlying differences in quality among hospitals being evaluated lead to systematic differences in outcomes. We then calculated hospital‐specific readmission rates as the ratio of predicted‐to‐expected readmissions (similar to observed/expected ratio), multiplied by the national unadjusted ratea form of indirect standardization. Predicted number of readmissions in each hospital is estimated given the same patient mix and its estimated hospital‐specific intercept. Expected number of readmissions in each hospital is estimated using its patient mix and the average hospital‐specific intercept. To assess hospital performance in any given year, we re‐estimate model coefficients using that year's data.

Model Validation: Administrative Claims

We compared the model performance in the development sample with its performance in the sample from the 2006 data that was not selected for the development set, and separately among pneumonia admissions in 2005. The model was recalibrated in each validation set.

Model Validation: Medical Record Abstraction

We developed a separate medical record‐based model of readmission risk using information from charts that had previously been abstracted as part of CMS's National Pneumonia Project. To select variables for this model, the clinician team: 1) reviewed the list of variables that were included in a medical record model that was previously developed for validating the National Quality Forum‐approved pneumonia mortality measure; 2) reviewed a list of other potential candidate variables available in the National Pneumonia Project dataset; and 3) reviewed variables that emerged as potentially important predictors of readmission, based on a systematic review of the literature that was conducted as part of measure development. This selection process resulted in a final medical record model that included 35 variables.

We linked patients in the National Pneumonia Project cohort to their Medicare claims data, including claims from one year before the index hospitalization, so that we could calculate risk‐standardized readmission rates in this cohort separately using medical record and claims‐based models. This analysis was conducted at the state level, for the 50 states plus the District of Columbia and Puerto Rico, because medical record data were unavailable in sufficient numbers to permit hospital‐level comparisons. To examine the relationship between risk‐standardized rates obtained from medical record and administrative data models, we estimated a linear regression model describing the association between the two rates, weighting each state by number of index hospitalizations, and calculated the correlation coefficient and the intercept and slope of this equation. A slope close to 1 and an intercept close to 0 would provide evidence that risk‐standardized state readmission rates from the medical record and claims models were similar. We also calculated the difference between state risk‐standardized readmission rates from the two models.

Analyses were conducted with the use of SAS version 9.1.3 (SAS Institute Inc, Cary, NC). Models were fitted separately for the National Pneumonia Project and 2006 cohort. We estimated the hierarchical models using the GLIMMIX procedure in SAS. The Human Investigation Committee at the Yale School of Medicine approved an exemption for the authors to use CMS claims and enrollment data for research analyses and publication.

Model Derivation and Performance

After exclusions were applied, the 2006 sample included 453,251 pneumonia hospitalizations (Figure 1). The development sample consisted of 226,545 hospitalizations at 4675 hospitals, with an overall unadjusted 30‐day readmission rate of 17.4%. In 11,694 index cases (5.2%), the patient died within 30 days without being readmitted. Median readmission rate was 16.3%, 25th and 75th percentile rates were 11.1% and 21.3%, and at the 10th and 90th percentile, hospital readmission rates ranged from 4.6% to 26.7% (Figure 2).

Figure 1

Figure 2

The claims model included 39 variables (age, sex, and 37 clinical variables) (Table 1). The mean age of the cohort was 80.0 years, with 55.5% women and 11.1% nonwhite patients. Mean observed readmission rate in the development sample ranged from 9% in the lowest decile of predicted pneumonia readmission rates to 32% in the highest predicted decile, a range of 23%. The AUC was 0.63. For comparison, a model with only age and sex had an AUC of 0.51, and a model with all candidate variables had an AUC equal to 0.63 (Table 2).

Hospital Risk‐Standardized Readmission Rates

Risk‐standardized readmission rates varied across hospitals (Figure 3). Median risk‐standardized readmission rate was 17.3%, and the 25th and 75th percentiles were 16.9% and 17.9%, respectively. The 5th percentile was 16.0% and the 95th percentile was 19.1%. Odds of readmission for a hospital one standard deviation above average was 1.4 times that of a hospital one standard deviation below average.

Figure 3

Administrative Model Validation

In the remaining 50% of pneumonia index hospitalizations from 2006, and the entire 2005 cohort, regression coefficients and standard errors of model variables were similar to those in the development data set. Model performance using 2005 data was consistent with model performance using the 2006 development and validation half‐samples (Table 2).

Medical Record Validation

After exclusions, the medical record sample taken from the National Pneumonia Project included 47,429 cases, with an unadjusted 30‐day readmission rate of 17.0%. The final medical record risk‐adjustment model included a total of 35 variables, whose prevalence and association with readmission risk varied modestly (Table 3). Performance of the medical record and administrative models was similar (areas under the ROC curve 0.59 and 0.63, respectively) (Table 4). Additionally, in the administrative model, predicted readmission rates ranged from 8% in the lowest predicted decile to 30% in the highest predicted decile, while in the medical record model, the corresponding rates varied from 10% to 26%.

The correlation coefficient of the estimated state‐specific standardized readmission rates from the administrative and medical record models was 0.96, and the proportion of the variance explained by the model was 0.92 (Figure 4).

Figure 4

Data Sources

We used 20052006 claims data from Medicare inpatient, outpatient, and carrier (physician) Standard Analytic Files to develop and validate the administrative model. The Medicare Enrollment Database was used to determine Medicare fee‐for‐service enrollment and mortality statuses. A medical record model, used for additional validation of the administrative model, was developed using information abstracted from the charts of 75,616 pneumonia cases from 19982001 as part of the National Pneumonia Project, a CMS quality improvement initiative.18

Study Cohort

We identified hospitalizations of patients 65 years of age and older with a principal diagnosis of pneumonia (International Classification of Diseases, 9th Revision, Clinical Modification codes 480.XX, 481, 482.XX, 483.X, 485, 486, 487.0) as potential index pneumonia admissions. Because our focus was readmission for patients discharged from acute care settings, we excluded admissions in which patients died or were transferred to another acute care facility. Additionally, we restricted analysis to patients who had been enrolled in fee‐for‐service Medicare Parts A and B, for at least 12 months prior to their pneumonia hospitalization, so that we could use diagnostic codes from all inpatient and outpatient encounters during that period to enhance identification of comorbidities.

Outcome

The outcome was 30‐day readmission, defined as occurrence of at least one hospitalization for any cause within 30 days of discharge after an index admission. Readmissions were identified from hospital claims data, and were attributed to the hospital that had discharged the patient. A 30‐day time frame was selected because it is a clinically meaningful period during which hospitals can be expected to collaborate with other organizations and providers to implement measures to reduce the risk of rehospitalization.

Candidate and Final Model Variables

Candidate variables for the administrative claims model were selected by a clinician team from 189 diagnostic groups included in the Hierarchical Condition Category (HCC) clinical classification system.19 The HCC clinical classification system was developed for CMS in preparation for all‐encounter risk adjustment for Medicare Advantage (managed care). Under the HCC algorithm, the 15,000+ ICD‐9‐CM diagnosis codes are assigned to one of 189 clinically‐coherent condition categories (CCs). We used the April 2008 version of the ICD‐9‐CM to CC assignment map, which is maintained by CMS and posted at http://www.qualitynet.org. A total of 154 CCs were considered to be potentially relevant to readmission outcome and were included for further consideration. Some CCs were further combined into clinically coherent groupings of CCs. Our set of candidate variables ultimately included 97 CC‐based variables, two demographic variables (age and sex), and two procedure codes potentially relevant to readmission risk (history of percutaneous coronary intervention [PCI] and history of coronary artery bypass graft [CABG]).

The final risk‐adjustment model included 39 variables selected by the team of clinicians and analysts, primarily based on their clinical relevance but also with knowledge of the strength of their statistical association with readmission outcome (Table 1). For each patient, the presence or absence of these conditions was assessed from multiple sources, including secondary diagnoses during the index admission, principal and secondary diagnoses from hospital admissions in the 12 months prior to the index admission, and diagnoses from hospital outpatient and physician encounters 12 months before the index admission. A small number of CCs were considered to represent potential complications of care (eg, bleeding). Because we did not want to adjust for complications of care occurring during the index admission, a patient was not considered to have one of these conditions unless it was also present in at least one encounter prior to the index admission.

Model Derivation

For the development of the administrative claims model, we randomly sampled half of 2006 hospitalizations that met inclusion criteria. To assess model performance at the patient level, we calculated the area under the receiver operating curve (AUC), and calculated observed readmission rates in the lowest and highest deciles on the basis of predicted readmission probabilities. We also compared performance with a null model, a model that adjusted for age and sex, and a model that included all candidate variables.20

Risk‐Standardized Readmission Rates

Using hierarchical logistic regression, we modeled the log‐odds of readmission within 30 days of discharge from an index pneumonia admission as a function of patient demographic and clinical characteristics, and a random hospital‐specific intercept. This strategy accounts for within‐hospital correlation, or clustering, of observed outcomes, and models the assumption that underlying differences in quality among hospitals being evaluated lead to systematic differences in outcomes. We then calculated hospital‐specific readmission rates as the ratio of predicted‐to‐expected readmissions (similar to observed/expected ratio), multiplied by the national unadjusted ratea form of indirect standardization. Predicted number of readmissions in each hospital is estimated given the same patient mix and its estimated hospital‐specific intercept. Expected number of readmissions in each hospital is estimated using its patient mix and the average hospital‐specific intercept. To assess hospital performance in any given year, we re‐estimate model coefficients using that year's data.

Model Validation: Administrative Claims

We compared the model performance in the development sample with its performance in the sample from the 2006 data that was not selected for the development set, and separately among pneumonia admissions in 2005. The model was recalibrated in each validation set.

Model Validation: Medical Record Abstraction

We developed a separate medical record‐based model of readmission risk using information from charts that had previously been abstracted as part of CMS's National Pneumonia Project. To select variables for this model, the clinician team: 1) reviewed the list of variables that were included in a medical record model that was previously developed for validating the National Quality Forum‐approved pneumonia mortality measure; 2) reviewed a list of other potential candidate variables available in the National Pneumonia Project dataset; and 3) reviewed variables that emerged as potentially important predictors of readmission, based on a systematic review of the literature that was conducted as part of measure development. This selection process resulted in a final medical record model that included 35 variables.

We linked patients in the National Pneumonia Project cohort to their Medicare claims data, including claims from one year before the index hospitalization, so that we could calculate risk‐standardized readmission rates in this cohort separately using medical record and claims‐based models. This analysis was conducted at the state level, for the 50 states plus the District of Columbia and Puerto Rico, because medical record data were unavailable in sufficient numbers to permit hospital‐level comparisons. To examine the relationship between risk‐standardized rates obtained from medical record and administrative data models, we estimated a linear regression model describing the association between the two rates, weighting each state by number of index hospitalizations, and calculated the correlation coefficient and the intercept and slope of this equation. A slope close to 1 and an intercept close to 0 would provide evidence that risk‐standardized state readmission rates from the medical record and claims models were similar. We also calculated the difference between state risk‐standardized readmission rates from the two models.

Analyses were conducted with the use of SAS version 9.1.3 (SAS Institute Inc, Cary, NC). Models were fitted separately for the National Pneumonia Project and 2006 cohort. We estimated the hierarchical models using the GLIMMIX procedure in SAS. The Human Investigation Committee at the Yale School of Medicine approved an exemption for the authors to use CMS claims and enrollment data for research analyses and publication.

Model Derivation and Performance

After exclusions were applied, the 2006 sample included 453,251 pneumonia hospitalizations (Figure 1). The development sample consisted of 226,545 hospitalizations at 4675 hospitals, with an overall unadjusted 30‐day readmission rate of 17.4%. In 11,694 index cases (5.2%), the patient died within 30 days without being readmitted. Median readmission rate was 16.3%, 25th and 75th percentile rates were 11.1% and 21.3%, and at the 10th and 90th percentile, hospital readmission rates ranged from 4.6% to 26.7% (Figure 2).

Figure 1

Figure 2

The claims model included 39 variables (age, sex, and 37 clinical variables) (Table 1). The mean age of the cohort was 80.0 years, with 55.5% women and 11.1% nonwhite patients. Mean observed readmission rate in the development sample ranged from 9% in the lowest decile of predicted pneumonia readmission rates to 32% in the highest predicted decile, a range of 23%. The AUC was 0.63. For comparison, a model with only age and sex had an AUC of 0.51, and a model with all candidate variables had an AUC equal to 0.63 (Table 2).

Hospital Risk‐Standardized Readmission Rates

Risk‐standardized readmission rates varied across hospitals (Figure 3). Median risk‐standardized readmission rate was 17.3%, and the 25th and 75th percentiles were 16.9% and 17.9%, respectively. The 5th percentile was 16.0% and the 95th percentile was 19.1%. Odds of readmission for a hospital one standard deviation above average was 1.4 times that of a hospital one standard deviation below average.

Figure 3

Administrative Model Validation

In the remaining 50% of pneumonia index hospitalizations from 2006, and the entire 2005 cohort, regression coefficients and standard errors of model variables were similar to those in the development data set. Model performance using 2005 data was consistent with model performance using the 2006 development and validation half‐samples (Table 2).

Medical Record Validation

After exclusions, the medical record sample taken from the National Pneumonia Project included 47,429 cases, with an unadjusted 30‐day readmission rate of 17.0%. The final medical record risk‐adjustment model included a total of 35 variables, whose prevalence and association with readmission risk varied modestly (Table 3). Performance of the medical record and administrative models was similar (areas under the ROC curve 0.59 and 0.63, respectively) (Table 4). Additionally, in the administrative model, predicted readmission rates ranged from 8% in the lowest predicted decile to 30% in the highest predicted decile, while in the medical record model, the corresponding rates varied from 10% to 26%.

The correlation coefficient of the estimated state‐specific standardized readmission rates from the administrative and medical record models was 0.96, and the proportion of the variance explained by the model was 0.92 (Figure 4).

Figure 4

Data Sources

We used 20052006 claims data from Medicare inpatient, outpatient, and carrier (physician) Standard Analytic Files to develop and validate the administrative model. The Medicare Enrollment Database was used to determine Medicare fee‐for‐service enrollment and mortality statuses. A medical record model, used for additional validation of the administrative model, was developed using information abstracted from the charts of 75,616 pneumonia cases from 19982001 as part of the National Pneumonia Project, a CMS quality improvement initiative.18

Study Cohort

We identified hospitalizations of patients 65 years of age and older with a principal diagnosis of pneumonia (International Classification of Diseases, 9th Revision, Clinical Modification codes 480.XX, 481, 482.XX, 483.X, 485, 486, 487.0) as potential index pneumonia admissions. Because our focus was readmission for patients discharged from acute care settings, we excluded admissions in which patients died or were transferred to another acute care facility. Additionally, we restricted analysis to patients who had been enrolled in fee‐for‐service Medicare Parts A and B, for at least 12 months prior to their pneumonia hospitalization, so that we could use diagnostic codes from all inpatient and outpatient encounters during that period to enhance identification of comorbidities.

Outcome

The outcome was 30‐day readmission, defined as occurrence of at least one hospitalization for any cause within 30 days of discharge after an index admission. Readmissions were identified from hospital claims data, and were attributed to the hospital that had discharged the patient. A 30‐day time frame was selected because it is a clinically meaningful period during which hospitals can be expected to collaborate with other organizations and providers to implement measures to reduce the risk of rehospitalization.

Candidate and Final Model Variables

Candidate variables for the administrative claims model were selected by a clinician team from 189 diagnostic groups included in the Hierarchical Condition Category (HCC) clinical classification system.19 The HCC clinical classification system was developed for CMS in preparation for all‐encounter risk adjustment for Medicare Advantage (managed care). Under the HCC algorithm, the 15,000+ ICD‐9‐CM diagnosis codes are assigned to one of 189 clinically‐coherent condition categories (CCs). We used the April 2008 version of the ICD‐9‐CM to CC assignment map, which is maintained by CMS and posted at http://www.qualitynet.org. A total of 154 CCs were considered to be potentially relevant to readmission outcome and were included for further consideration. Some CCs were further combined into clinically coherent groupings of CCs. Our set of candidate variables ultimately included 97 CC‐based variables, two demographic variables (age and sex), and two procedure codes potentially relevant to readmission risk (history of percutaneous coronary intervention [PCI] and history of coronary artery bypass graft [CABG]).

The final risk‐adjustment model included 39 variables selected by the team of clinicians and analysts, primarily based on their clinical relevance but also with knowledge of the strength of their statistical association with readmission outcome (Table 1). For each patient, the presence or absence of these conditions was assessed from multiple sources, including secondary diagnoses during the index admission, principal and secondary diagnoses from hospital admissions in the 12 months prior to the index admission, and diagnoses from hospital outpatient and physician encounters 12 months before the index admission. A small number of CCs were considered to represent potential complications of care (eg, bleeding). Because we did not want to adjust for complications of care occurring during the index admission, a patient was not considered to have one of these conditions unless it was also present in at least one encounter prior to the index admission.

Model Derivation

For the development of the administrative claims model, we randomly sampled half of 2006 hospitalizations that met inclusion criteria. To assess model performance at the patient level, we calculated the area under the receiver operating curve (AUC), and calculated observed readmission rates in the lowest and highest deciles on the basis of predicted readmission probabilities. We also compared performance with a null model, a model that adjusted for age and sex, and a model that included all candidate variables.20

Risk‐Standardized Readmission Rates

Using hierarchical logistic regression, we modeled the log‐odds of readmission within 30 days of discharge from an index pneumonia admission as a function of patient demographic and clinical characteristics, and a random hospital‐specific intercept. This strategy accounts for within‐hospital correlation, or clustering, of observed outcomes, and models the assumption that underlying differences in quality among hospitals being evaluated lead to systematic differences in outcomes. We then calculated hospital‐specific readmission rates as the ratio of predicted‐to‐expected readmissions (similar to observed/expected ratio), multiplied by the national unadjusted ratea form of indirect standardization. Predicted number of readmissions in each hospital is estimated given the same patient mix and its estimated hospital‐specific intercept. Expected number of readmissions in each hospital is estimated using its patient mix and the average hospital‐specific intercept. To assess hospital performance in any given year, we re‐estimate model coefficients using that year's data.

Model Validation: Administrative Claims

We compared the model performance in the development sample with its performance in the sample from the 2006 data that was not selected for the development set, and separately among pneumonia admissions in 2005. The model was recalibrated in each validation set.

Model Validation: Medical Record Abstraction

We developed a separate medical record‐based model of readmission risk using information from charts that had previously been abstracted as part of CMS's National Pneumonia Project. To select variables for this model, the clinician team: 1) reviewed the list of variables that were included in a medical record model that was previously developed for validating the National Quality Forum‐approved pneumonia mortality measure; 2) reviewed a list of other potential candidate variables available in the National Pneumonia Project dataset; and 3) reviewed variables that emerged as potentially important predictors of readmission, based on a systematic review of the literature that was conducted as part of measure development. This selection process resulted in a final medical record model that included 35 variables.

We linked patients in the National Pneumonia Project cohort to their Medicare claims data, including claims from one year before the index hospitalization, so that we could calculate risk‐standardized readmission rates in this cohort separately using medical record and claims‐based models. This analysis was conducted at the state level, for the 50 states plus the District of Columbia and Puerto Rico, because medical record data were unavailable in sufficient numbers to permit hospital‐level comparisons. To examine the relationship between risk‐standardized rates obtained from medical record and administrative data models, we estimated a linear regression model describing the association between the two rates, weighting each state by number of index hospitalizations, and calculated the correlation coefficient and the intercept and slope of this equation. A slope close to 1 and an intercept close to 0 would provide evidence that risk‐standardized state readmission rates from the medical record and claims models were similar. We also calculated the difference between state risk‐standardized readmission rates from the two models.

Analyses were conducted with the use of SAS version 9.1.3 (SAS Institute Inc, Cary, NC). Models were fitted separately for the National Pneumonia Project and 2006 cohort. We estimated the hierarchical models using the GLIMMIX procedure in SAS. The Human Investigation Committee at the Yale School of Medicine approved an exemption for the authors to use CMS claims and enrollment data for research analyses and publication.

Model Derivation and Performance

After exclusions were applied, the 2006 sample included 453,251 pneumonia hospitalizations (Figure 1). The development sample consisted of 226,545 hospitalizations at 4675 hospitals, with an overall unadjusted 30‐day readmission rate of 17.4%. In 11,694 index cases (5.2%), the patient died within 30 days without being readmitted. Median readmission rate was 16.3%, 25th and 75th percentile rates were 11.1% and 21.3%, and at the 10th and 90th percentile, hospital readmission rates ranged from 4.6% to 26.7% (Figure 2).

Figure 1

Figure 2

The claims model included 39 variables (age, sex, and 37 clinical variables) (Table 1). The mean age of the cohort was 80.0 years, with 55.5% women and 11.1% nonwhite patients. Mean observed readmission rate in the development sample ranged from 9% in the lowest decile of predicted pneumonia readmission rates to 32% in the highest predicted decile, a range of 23%. The AUC was 0.63. For comparison, a model with only age and sex had an AUC of 0.51, and a model with all candidate variables had an AUC equal to 0.63 (Table 2).

Hospital Risk‐Standardized Readmission Rates

Risk‐standardized readmission rates varied across hospitals (Figure 3). Median risk‐standardized readmission rate was 17.3%, and the 25th and 75th percentiles were 16.9% and 17.9%, respectively. The 5th percentile was 16.0% and the 95th percentile was 19.1%. Odds of readmission for a hospital one standard deviation above average was 1.4 times that of a hospital one standard deviation below average.

Figure 3

Administrative Model Validation

In the remaining 50% of pneumonia index hospitalizations from 2006, and the entire 2005 cohort, regression coefficients and standard errors of model variables were similar to those in the development data set. Model performance using 2005 data was consistent with model performance using the 2006 development and validation half‐samples (Table 2).

Medical Record Validation

After exclusions, the medical record sample taken from the National Pneumonia Project included 47,429 cases, with an unadjusted 30‐day readmission rate of 17.0%. The final medical record risk‐adjustment model included a total of 35 variables, whose prevalence and association with readmission risk varied modestly (Table 3). Performance of the medical record and administrative models was similar (areas under the ROC curve 0.59 and 0.63, respectively) (Table 4). Additionally, in the administrative model, predicted readmission rates ranged from 8% in the lowest predicted decile to 30% in the highest predicted decile, while in the medical record model, the corresponding rates varied from 10% to 26%.

The correlation coefficient of the estimated state‐specific standardized readmission rates from the administrative and medical record models was 0.96, and the proportion of the variance explained by the model was 0.92 (Figure 4).

Figure 4

Design, Setting, Subjects

We conducted a cross‐sectional study at the hospital level of the outcomes of care of fee‐for‐service patients hospitalized for pneumonia between July 2006 and June 2009. Patients are eligible to be included in the measures if they are 65 years or older, have a principal diagnosis of pneumonia (International Classification of Diseases, Ninth Revision, Clinical Modification codes 480.X, 481, 482.XX, 483.X, 485, 486, and 487.0), and are cared for at a nonfederal acute care hospital in the US and its organized territories, including Puerto Rico, Guam, the US Virgin Islands, and the Northern Mariana Islands.

The mortality measure excludes patients enrolled in the Medicare hospice program in the year prior to, or on the day of admission, those in whom pneumonia is listed as a secondary diagnosis (to eliminate cases resulting from complications of hospitalization), those discharged against medical advice, and patients who are discharged alive but whose length of stay in the hospital is less than 1 day (because of concerns about the accuracy of the pneumonia diagnosis). Patients are also excluded if their administrative records for the period of analysis (1 year prior to hospitalization and 30 days following discharge) were not available or were incomplete, because these are needed to assess comorbid illness and outcomes. The readmission measure is similar, but does not exclude patients on the basis of hospice program enrollment (because these patients have been admitted and readmissions for hospice patients are likely unplanned events that can be measured and reduced), nor on the basis of hospital length of stay (because patients discharged within 24 hours may be at a heightened risk of readmission).11, 12

Information about patient comorbidities is derived from diagnoses recorded in the year prior to the index hospitalization as found in Medicare inpatient, outpatient, and carrier (physician) standard analytic files. Comorbidities are identified using the Condition Categories of the Hierarchical Condition Category grouper, which sorts the more than 15,000 possible diagnostic codes into 189 clinically‐coherent conditions and which was originally developed to support risk‐adjusted payments within Medicare managed care.13

Outcomes

The patient outcomes assessed include death from any cause within 30 days of admission and readmission for any cause within 30 days of discharge. All‐cause, rather than disease‐specific, readmission was chosen because hospital readmission as a consequence of suboptimal inpatient care or discharge coordination may manifest in many different diagnoses, and no validated method is available to distinguish related from unrelated readmissions. The measures use the Medicare Enrollment Database to determine mortality status, and acute care hospital inpatient claims are used to identify readmission events. For patients with multiple hospitalizations during the study period, the mortality measure randomly selects one hospitalization to use for determination of mortality. Admissions that are counted as readmissions (i.e., those that occurred within 30 days of discharge following hospitalization for pneumonia) are not also treated as index hospitalizations. In the case of patients who are transferred to or from another acute care facility, responsibility for deaths is assigned to the hospital that initially admitted the patient, while responsibility for readmissions is assigned to the hospital that ultimately discharges the patient to a nonacute setting (e.g., home, skilled nursing facilities).

Risk‐Standardization Methods

Hierarchical logistic regression is used to model the log‐odds of mortality or readmission within 30 days of admission or discharge from an index pneumonia admission as a function of patient demographic and clinical characteristics and a random hospital‐specific intercept. This strategy accounts for within‐hospital correlation of the observed outcomes, and reflects the assumption that underlying differences in quality among the hospitals being evaluated lead to systematic differences in outcomes. In contrast to nonhierarchical models which ignore hospital effects, this method attempts to measure the influence of the hospital on patient outcome after adjusting for patient characteristics. Comorbidities from the index admission that could represent potential complications of care are not included in the model unless they are also documented in the 12 months prior to admission. Hospital‐specific mortality and readmission rates are calculated as the ratio of predicted‐to‐expected events (similar to the observed/expected ratio), multiplied by the national unadjusted rate, a form of indirect standardization.

The model for mortality has a c‐statistic of 0.72 whereas a model based on medical record review that was developed for validation purposes had a c‐statistic of 0.77. The model for readmission has a c‐statistic of 0.63 whereas a model based on medical review had a c‐statistic of 0.59. The mortality and readmission models produce similar state‐level mortality and readmission rate estimates as the models derived from medical record review, and can therefore serve as reasonable surrogates. These methods, including their development and validation, have been described fully elsewhere,14, 15 and have been evaluated and subsequently endorsed by the National Quality Forum.16

Identification of Geographic Regions

To characterize patterns of performance geographically we identified the 306 hospital referral regions for each hospital in our analysis using definitions provided by the Dartmouth Atlas of Health Care project. Unlike a hospital‐level analysis, the hospital referral regions represent regional markets for tertiary care and are widely used to summarize variation in medical care inputs, utilization patterns, and health outcomes and provide a more detailed look at variation in outcomes than results at the state level.17

Analyses

Summary statistics were constructed using frequencies and proportions for categorical data, and means, medians and interquartile ranges for continuous variables. To characterize 30‐day risk‐standardized mortality and readmission rates at the hospital‐referral region level, we calculated means and percentiles by weighting each hospital's value by the inverse of the variance of the hospital's estimated rate. Hospitals with larger sample sizes, and therefore more precise estimates, lend more weight to the average. Hierarchical models were estimated using the SAS GLIMMIX procedure. Bayesian shrinkage was used to estimate rates in order to take into account the greater uncertainty in the true rates of hospitals with small caseloads. Using this technique, estimated rates at low volume institutions are shrunken toward the population mean, while hospitals with large caseloads have a relatively smaller amount of shrinkage and the estimate is closer to the hospital's observed rate.18

To determine whether a hospital's performance is significantly different than the national rate we measured whether the 95% interval estimate for the risk‐standardized rate overlapped with the national crude mortality or readmission rate. This information is used to categorize hospitals on Hospital Compare as better than the US national rate, worse than the US national rate, or no different than the US national rate. Hospitals with fewer than 25 cases in the 3‐year period, are excluded from this categorization on Hospital Compare.

Analyses were conducted with the use of SAS 9.1.3 (SAS Institute Inc, Cary, NC). We created the hospital referral region maps using ArcGIS version 9.3 (ESRI, Redlands, CA). The Human Investigation Committee at the Yale School of Medicine approved an exemption for the authors to use CMS claims and enrollment data for research analyses and publication.

Hospital‐Specific Risk‐Standardized 30‐Day Mortality and Readmission Rates

Of the 1,118,583 patients included in the mortality analysis 129,444 (11.6%) died within 30 days of hospital admission. The median (Q1, Q3) hospital 30‐day risk‐standardized mortality rate was 11.1% (10.0%, 12.3%), and ranged from 6.7% to 20.9% (Table 1, Figure 1). Hospitals at the 10th percentile had 30‐day risk‐standardized mortality rates of 9.0% while for those at the 90th percentile of performance the rate was 13.5%. The odds of all‐cause mortality for a patient treated at a hospital that was one standard deviation above the national average was 1.68 times higher than that of a patient treated at a hospital that was one standard deviation below the national average.

Figure 1

For the 3‐year period 2006 to 2009, 222 (4.7%) hospitals were categorized as having a mortality rate that was better than the national average, 3968 (83.7%) were no different than the national average, 221 (4.6%) were worse and 332 (7.0%) did not meet the minimum case threshold.

Among the 1,161,817 patients included in the readmission analysis 212,638 (18.3%) were readmitted within 30 days of hospital discharge. The median (Q1,Q3) 30‐day risk‐standardized readmission rate was 18.2% (17.2%, 19.2%) and ranged from 13.6% to 26.7% (Table 1, Figure 2). Hospitals at the 10th percentile had 30‐day risk‐standardized readmission rates of 16.4% while for those at the 90th percentile of performance the rate was 20.4%. The odds of all‐cause readmission for a patient treated at a hospital that was one standard deviation above the national average was 1.40 times higher than the odds of all‐cause readmission if treated at a hospital that was one standard deviation below the national average.

Figure 2

For the 3‐year period 2006 to 2009, 64 (1.3%) hospitals were categorized as having a readmission rate that was better than the national average, 4203 (88.2%) were no different than the national average, 163 (3.4%) were worse and 333 (7.0%) had less than 25 cases and were therefore not categorized.

While risk‐standardized readmission rates were substantially higher than risk‐standardized mortality rates, mortality rates varied more. For example, the top 10% of hospitals had a relative mortality rate that was 33% lower than those in the bottom 10%, as compared with just a 20% relative difference for readmission rates. The coefficient of variation, a normalized measure of dispersion that unlike the standard deviation is independent of the population mean, was 10.7 for risk‐standardized mortality rates and 4.9 for readmission rates.

Regional Risk‐Standardized 30‐Day Mortality and Readmission Rates

Figures 3 and 4 show the distribution of 30‐day risk‐standardized mortality and readmission rates among hospital referral regions by quintile. Highest mortality regions were found across the entire country, including parts of Northern New England, the Mid and South Atlantic, East and the West South Central, East and West North Central, and the Mountain and Pacific regions of the West. The lowest mortality rates were observed in Southern New England, parts of the Mid and South Atlantic, East and West South Central, and parts of the Mountain and Pacific regions of the West (Figure 3).

Figure 3

Figure 4

Readmission rates were higher in the eastern portions of the US (including the Northeast, Mid and South Atlantic, East South Central) as well as the East North Central, and small parts of the West North Central portions of the Midwest and in Central California. The lowest readmission rates were observed in the West (Mountain and Pacific regions), parts of the Midwest (East and West North Central) and small pockets within the South and Northeast (Figure 4).

Design, Setting, Subjects

We conducted a cross‐sectional study at the hospital level of the outcomes of care of fee‐for‐service patients hospitalized for pneumonia between July 2006 and June 2009. Patients are eligible to be included in the measures if they are 65 years or older, have a principal diagnosis of pneumonia (International Classification of Diseases, Ninth Revision, Clinical Modification codes 480.X, 481, 482.XX, 483.X, 485, 486, and 487.0), and are cared for at a nonfederal acute care hospital in the US and its organized territories, including Puerto Rico, Guam, the US Virgin Islands, and the Northern Mariana Islands.

The mortality measure excludes patients enrolled in the Medicare hospice program in the year prior to, or on the day of admission, those in whom pneumonia is listed as a secondary diagnosis (to eliminate cases resulting from complications of hospitalization), those discharged against medical advice, and patients who are discharged alive but whose length of stay in the hospital is less than 1 day (because of concerns about the accuracy of the pneumonia diagnosis). Patients are also excluded if their administrative records for the period of analysis (1 year prior to hospitalization and 30 days following discharge) were not available or were incomplete, because these are needed to assess comorbid illness and outcomes. The readmission measure is similar, but does not exclude patients on the basis of hospice program enrollment (because these patients have been admitted and readmissions for hospice patients are likely unplanned events that can be measured and reduced), nor on the basis of hospital length of stay (because patients discharged within 24 hours may be at a heightened risk of readmission).11, 12

Information about patient comorbidities is derived from diagnoses recorded in the year prior to the index hospitalization as found in Medicare inpatient, outpatient, and carrier (physician) standard analytic files. Comorbidities are identified using the Condition Categories of the Hierarchical Condition Category grouper, which sorts the more than 15,000 possible diagnostic codes into 189 clinically‐coherent conditions and which was originally developed to support risk‐adjusted payments within Medicare managed care.13

Outcomes

The patient outcomes assessed include death from any cause within 30 days of admission and readmission for any cause within 30 days of discharge. All‐cause, rather than disease‐specific, readmission was chosen because hospital readmission as a consequence of suboptimal inpatient care or discharge coordination may manifest in many different diagnoses, and no validated method is available to distinguish related from unrelated readmissions. The measures use the Medicare Enrollment Database to determine mortality status, and acute care hospital inpatient claims are used to identify readmission events. For patients with multiple hospitalizations during the study period, the mortality measure randomly selects one hospitalization to use for determination of mortality. Admissions that are counted as readmissions (i.e., those that occurred within 30 days of discharge following hospitalization for pneumonia) are not also treated as index hospitalizations. In the case of patients who are transferred to or from another acute care facility, responsibility for deaths is assigned to the hospital that initially admitted the patient, while responsibility for readmissions is assigned to the hospital that ultimately discharges the patient to a nonacute setting (e.g., home, skilled nursing facilities).

Risk‐Standardization Methods

Hierarchical logistic regression is used to model the log‐odds of mortality or readmission within 30 days of admission or discharge from an index pneumonia admission as a function of patient demographic and clinical characteristics and a random hospital‐specific intercept. This strategy accounts for within‐hospital correlation of the observed outcomes, and reflects the assumption that underlying differences in quality among the hospitals being evaluated lead to systematic differences in outcomes. In contrast to nonhierarchical models which ignore hospital effects, this method attempts to measure the influence of the hospital on patient outcome after adjusting for patient characteristics. Comorbidities from the index admission that could represent potential complications of care are not included in the model unless they are also documented in the 12 months prior to admission. Hospital‐specific mortality and readmission rates are calculated as the ratio of predicted‐to‐expected events (similar to the observed/expected ratio), multiplied by the national unadjusted rate, a form of indirect standardization.

The model for mortality has a c‐statistic of 0.72 whereas a model based on medical record review that was developed for validation purposes had a c‐statistic of 0.77. The model for readmission has a c‐statistic of 0.63 whereas a model based on medical review had a c‐statistic of 0.59. The mortality and readmission models produce similar state‐level mortality and readmission rate estimates as the models derived from medical record review, and can therefore serve as reasonable surrogates. These methods, including their development and validation, have been described fully elsewhere,14, 15 and have been evaluated and subsequently endorsed by the National Quality Forum.16

Identification of Geographic Regions

To characterize patterns of performance geographically we identified the 306 hospital referral regions for each hospital in our analysis using definitions provided by the Dartmouth Atlas of Health Care project. Unlike a hospital‐level analysis, the hospital referral regions represent regional markets for tertiary care and are widely used to summarize variation in medical care inputs, utilization patterns, and health outcomes and provide a more detailed look at variation in outcomes than results at the state level.17

Analyses

Summary statistics were constructed using frequencies and proportions for categorical data, and means, medians and interquartile ranges for continuous variables. To characterize 30‐day risk‐standardized mortality and readmission rates at the hospital‐referral region level, we calculated means and percentiles by weighting each hospital's value by the inverse of the variance of the hospital's estimated rate. Hospitals with larger sample sizes, and therefore more precise estimates, lend more weight to the average. Hierarchical models were estimated using the SAS GLIMMIX procedure. Bayesian shrinkage was used to estimate rates in order to take into account the greater uncertainty in the true rates of hospitals with small caseloads. Using this technique, estimated rates at low volume institutions are shrunken toward the population mean, while hospitals with large caseloads have a relatively smaller amount of shrinkage and the estimate is closer to the hospital's observed rate.18

To determine whether a hospital's performance is significantly different than the national rate we measured whether the 95% interval estimate for the risk‐standardized rate overlapped with the national crude mortality or readmission rate. This information is used to categorize hospitals on Hospital Compare as better than the US national rate, worse than the US national rate, or no different than the US national rate. Hospitals with fewer than 25 cases in the 3‐year period, are excluded from this categorization on Hospital Compare.

Analyses were conducted with the use of SAS 9.1.3 (SAS Institute Inc, Cary, NC). We created the hospital referral region maps using ArcGIS version 9.3 (ESRI, Redlands, CA). The Human Investigation Committee at the Yale School of Medicine approved an exemption for the authors to use CMS claims and enrollment data for research analyses and publication.

Hospital‐Specific Risk‐Standardized 30‐Day Mortality and Readmission Rates

Of the 1,118,583 patients included in the mortality analysis 129,444 (11.6%) died within 30 days of hospital admission. The median (Q1, Q3) hospital 30‐day risk‐standardized mortality rate was 11.1% (10.0%, 12.3%), and ranged from 6.7% to 20.9% (Table 1, Figure 1). Hospitals at the 10th percentile had 30‐day risk‐standardized mortality rates of 9.0% while for those at the 90th percentile of performance the rate was 13.5%. The odds of all‐cause mortality for a patient treated at a hospital that was one standard deviation above the national average was 1.68 times higher than that of a patient treated at a hospital that was one standard deviation below the national average.

Figure 1

For the 3‐year period 2006 to 2009, 222 (4.7%) hospitals were categorized as having a mortality rate that was better than the national average, 3968 (83.7%) were no different than the national average, 221 (4.6%) were worse and 332 (7.0%) did not meet the minimum case threshold.

Among the 1,161,817 patients included in the readmission analysis 212,638 (18.3%) were readmitted within 30 days of hospital discharge. The median (Q1,Q3) 30‐day risk‐standardized readmission rate was 18.2% (17.2%, 19.2%) and ranged from 13.6% to 26.7% (Table 1, Figure 2). Hospitals at the 10th percentile had 30‐day risk‐standardized readmission rates of 16.4% while for those at the 90th percentile of performance the rate was 20.4%. The odds of all‐cause readmission for a patient treated at a hospital that was one standard deviation above the national average was 1.40 times higher than the odds of all‐cause readmission if treated at a hospital that was one standard deviation below the national average.

Figure 2

For the 3‐year period 2006 to 2009, 64 (1.3%) hospitals were categorized as having a readmission rate that was better than the national average, 4203 (88.2%) were no different than the national average, 163 (3.4%) were worse and 333 (7.0%) had less than 25 cases and were therefore not categorized.

While risk‐standardized readmission rates were substantially higher than risk‐standardized mortality rates, mortality rates varied more. For example, the top 10% of hospitals had a relative mortality rate that was 33% lower than those in the bottom 10%, as compared with just a 20% relative difference for readmission rates. The coefficient of variation, a normalized measure of dispersion that unlike the standard deviation is independent of the population mean, was 10.7 for risk‐standardized mortality rates and 4.9 for readmission rates.

Regional Risk‐Standardized 30‐Day Mortality and Readmission Rates

Figures 3 and 4 show the distribution of 30‐day risk‐standardized mortality and readmission rates among hospital referral regions by quintile. Highest mortality regions were found across the entire country, including parts of Northern New England, the Mid and South Atlantic, East and the West South Central, East and West North Central, and the Mountain and Pacific regions of the West. The lowest mortality rates were observed in Southern New England, parts of the Mid and South Atlantic, East and West South Central, and parts of the Mountain and Pacific regions of the West (Figure 3).