Heart failure is a frequent cause of hospital admission in the United States, with an estimated cost of $31 billion dollars per year.[1] Discharging a patient with heart failure requires a multidisciplinary approach that includes anticipating a discharge date, scheduling follow‐up, reconciling medications, assessing home‐care or placement needs, and delivering patient education.[2, 3] Comprehensive transitional care interventions reduce readmissions and mortality.[2] Individually tailored and structured discharge plans decrease length of stay and readmissions.[3] The Centers for Medicare and Medicaid Services recently proposed that discharge planning begin within 24 hours of inpatient admissions,[4] despite inadequate data surrounding the optimal time to begin discharge planning.[3] In addition to enabling transitional care, identifying patients vulnerable to extended hospitalization aids in risk stratification, as prolonged length of stay is associated with increased risk of readmission and mortality.[5, 6]

Physicians are not able to accurately prognosticate whether patients will experience short‐term outcomes such as readmissions or mortality.[7, 8] Likewise, physicians do not predict length of stay accurately for heterogeneous patient populations,[9, 10, 11] even on the morning prior to anticipated discharge.[12] Prediction accuracy for patients admitted with heart failure, however, has not been adequately studied. The objectives of this study were to measure the accuracy of inpatient physicians' early predictions of length of stay for patients admitted with heart failure and to determine whether level of experience improved accuracy.

METHODS

In this prospective, observational study, we measured physicians' predictions of length of stay for patients admitted to a heart failure teaching service at an academic tertiary care hospital. Three resident/emntern teams rotate admitting responsibilities every 3 days, supervised by 1 attending cardiologist. Patients admitted overnight may be admitted independently by the on‐call resident without intern collaboration.

All physicians staffing our center's heart failure teaching service between August 1, 2013 and November 19, 2013 were recruited, and consecutively admitted adult patients were included. Patients were excluded if they did not have any cardiac diagnosis or if still admitted at study completion in February 2014. Deceased patients' time of death was counted as discharge.

Interns, residents, and attending cardiologists were interviewed independently within 24 hours of admission and asked to predict length of stay. Interns and residents were interviewed prior to rounds, and attendings thereafter. Electronic medical records were reviewed to determine date and time of admission and discharge, demographics, clinical variables, and discharge diagnoses.

The primary outcome was accuracy of predictions of length of stay stratified by level of experience. Based on prior pilot data, at 80% power and significance level () of 0.05, we estimated that predictions were needed on 100 patients to detect a 2‐day difference between actual and predicted length of stay.

Student t tests were used to compare the difference between predicted and actual length of stay for each level of training. Analysis of variance (ANOVA) was used to compare accuracy of prediction by training level. Generalized estimating equation (GEE) modeling was applied to compare predictions among interns, residents, and attending cardiologists, accounting for clustering by individual physician. GEE models were adjusted for study week in a sensitivity analysis to determine if predictions improved over time.

Analysis was performed using SAS 9.3 (SAS Institute Inc., Cary, NC) and R 2.14 (The R Foundation for Statistical Computing, Vienna, Austria). Institutional review board approval was granted, and physicians provided informed consent. All authors had access to primary data devoid of protected health information.

RESULTS

In total, 22 interns (<6 months experience), 25 residents (13 years experience), and 8 attending cardiologists (mean 19 9.7 years experience) were studied. Predictions were performed on 171 consecutively admitted patients. Five patients had noncardiac diagnoses and 1 patient remained admitted, leaving 165 patients for analysis. Predictions were made by all 3 physician levels on 98 patients. There were 67 patients with incomplete predictions as a result of 63 intern, 13 attending, and 4 resident predictions that were unobtainable. Absent intern data predominantly resulted from night shift admissions. Remaining missing data were due to time‐sensitive physician tasks that interfered with physician interviews.

Patient characteristics are described in Table 1. Physicians provided 415 predictions on 165 patients, 157 (95%) of whom survived to hospital discharge. Mean and median lengths of stay were 10.9 and 8 days (interquartile range [IQR], 4 to 13). Mean intern (N = 102), resident (N = 161), and attending (N = 152) predictions were 5.4 days (95% confidence interval [CI]: 4.6 to 6.2), 6.6 days (95% CI: 5.8 to 7.4) and 7.2 days (95% CI: 6.4 to 7.9), respectively. Median intern, resident, and attending predictions were 5 days (IQR, 3 to 7), 5 days (IQR, 3 to 7), and 6 days (IQR, 4 to 10). Mean differences between predicted and actual length of stay for interns, residents and attendings were 9 days (95% CI: 8.2 to 3.6), 4.3 days (95% C: 6.0 to 2.7), and 3.5 days (95% CI: 5.1 to 2.0). The mean difference between predicted and actual length of stay was statistically significant for all groups (P < 0.0001). Median intern, resident, and attending differences between predicted and actual were 2 days (IQR, 7 to 0), 2 days (IQR, 7 to 0), and 1 day (IQR, 5 to 1), respectively. Predictions correlated poorly with actual length of stay (R² = 0.11).

Ninety‐eight patients (59%) received predictions from physicians at all 3 experience levels. Mean and median lengths of stay were 11.3 days and 7.5 days (IQR, 4 to 13). Concordant with the entire cohort, median intern, resident, and attending predictions for these patients were 5 days (IQR, 3 to 7), 5 days (IQR, 3 to 7), and 6 days (IQR, 4 to 10), respectively. Differences between predicted and actual length of stay were statistically significant for all groups: the mean difference for interns, residents, and attendings was 5.8 days (95% CI: 8.2 to 3.4, P < 0.0001), 4.6 days (95% CI: 7.1 to 2.0, P = 0.0001), and 4.3 days (95% CI: 6.5 to 2.1, P = 0.0003), respectively (Figure 1).

Figure 1

There are differences among providers with improved prediction as level of experience increased, but this is not statistically significant as determined by ANOVA (p=0.64) or by GEE modeling to account for clustering of predictions by physician (P = 0.61). Analysis that adjusted for study week yielded similar results. Thus, experience did not improve accuracy.

DISCUSSION

We prospectively measured accuracy of physicians' length of stay predictions of heart failure patients and compared accuracy by experience level. All physicians underestimated length of stay, with average differences between 3.5 and 6 days. Most notably, level of experience did not improve accuracy. Although we anticipated that experience would improve prediction, our findings are not compatible with this hypothesis. Future studies of factors affecting length of stay predictions would help to better understand our findings.

Our results are consistent with small, single‐center studies of different patient and physician cohorts. Hulter Asberg found that internists at a hospital were unable to predict whether a patient would remain admitted 10 days or more, with poor interobserver reliability.[9] Mak et al. demonstrated that emergency physicians underestimated length of stay by an average of 2 days when predicting length of stay on a broad spectrum of patients in an emergency department.[10] Physician predictions of length of stay have been found to be inaccurate in a center's oncologic intensive care unit population.[11] Sullivan et al. found that academic general medicine physicians predicted discharge with 27% sensitivity the morning prior to next‐day discharge, which improved significantly to 67% by the afternoon, concluding that physicians can provide meaningful discharge predictions the afternoon prior to next‐day discharge.[12] By focusing on patients with heart failure, a major driver of hospitalization and readmission, and comparing providers by level of experience, we augment this existing body of work.

In addition to identifying patients at risk for readmission and mortality,[5, 6] accurate discharge prediction may improve safety of weekend discharges and patient satisfaction. Heart failure patients discharged on weekends receive less complete discharge instructions,[13] suffer higher mortality, and are readmitted more frequently than those discharged on weekdays.[14] Early and accurate predictions may enhance interventions targeting patients with anticipated weekend discharges. Furthermore, inadequate communication regarding anticipated discharge timing is a source of patient dissatisfaction,[15] and accurate prediction of discharge, if shared with patients, may improve patient satisfaction.

Limitations of our study include that it was a single‐center study at a large academic tertiary care hospital with predictions assessed on a teaching service. Severity of illness of this cohort may be a barrier to generalizability, and physicians may predict prognosis of healthier patients more accurately. We recorded predictions at the time of admission, and did not assess whether accuracy improved closer to discharge. We did not collect predictions from non‐physician team members. Sample size and absent data regarding the causes of prolonged hospitalization prohibited an analyses of variables associated with prediction inaccuracy.

CONCLUSIONS

Physicians do not accurately forecast heart failure patients' length of stay at the time of admission, and level of experience does not improve accuracy. Future studies are warranted to determine whether predictions closer to discharge, by an interdisciplinary team, or with assistance of risk‐prediction models are more accurate than physician predictions at admission, and whether early identification of patients at risk for prolonged hospitalization improves outcomes. Ultimately, early and accurate length of stay forecasts may improve risk stratification, patient satisfaction, and discharge planning, and reduce adverse outcomes related to at‐risk discharges.

Heart failure is a frequent cause of hospital admission in the United States, with an estimated cost of $31 billion dollars per year.[1] Discharging a patient with heart failure requires a multidisciplinary approach that includes anticipating a discharge date, scheduling follow‐up, reconciling medications, assessing home‐care or placement needs, and delivering patient education.[2, 3] Comprehensive transitional care interventions reduce readmissions and mortality.[2] Individually tailored and structured discharge plans decrease length of stay and readmissions.[3] The Centers for Medicare and Medicaid Services recently proposed that discharge planning begin within 24 hours of inpatient admissions,[4] despite inadequate data surrounding the optimal time to begin discharge planning.[3] In addition to enabling transitional care, identifying patients vulnerable to extended hospitalization aids in risk stratification, as prolonged length of stay is associated with increased risk of readmission and mortality.[5, 6]

Physicians are not able to accurately prognosticate whether patients will experience short‐term outcomes such as readmissions or mortality.[7, 8] Likewise, physicians do not predict length of stay accurately for heterogeneous patient populations,[9, 10, 11] even on the morning prior to anticipated discharge.[12] Prediction accuracy for patients admitted with heart failure, however, has not been adequately studied. The objectives of this study were to measure the accuracy of inpatient physicians' early predictions of length of stay for patients admitted with heart failure and to determine whether level of experience improved accuracy.

METHODS

In this prospective, observational study, we measured physicians' predictions of length of stay for patients admitted to a heart failure teaching service at an academic tertiary care hospital. Three resident/emntern teams rotate admitting responsibilities every 3 days, supervised by 1 attending cardiologist. Patients admitted overnight may be admitted independently by the on‐call resident without intern collaboration.

All physicians staffing our center's heart failure teaching service between August 1, 2013 and November 19, 2013 were recruited, and consecutively admitted adult patients were included. Patients were excluded if they did not have any cardiac diagnosis or if still admitted at study completion in February 2014. Deceased patients' time of death was counted as discharge.

Interns, residents, and attending cardiologists were interviewed independently within 24 hours of admission and asked to predict length of stay. Interns and residents were interviewed prior to rounds, and attendings thereafter. Electronic medical records were reviewed to determine date and time of admission and discharge, demographics, clinical variables, and discharge diagnoses.

The primary outcome was accuracy of predictions of length of stay stratified by level of experience. Based on prior pilot data, at 80% power and significance level () of 0.05, we estimated that predictions were needed on 100 patients to detect a 2‐day difference between actual and predicted length of stay.

Student t tests were used to compare the difference between predicted and actual length of stay for each level of training. Analysis of variance (ANOVA) was used to compare accuracy of prediction by training level. Generalized estimating equation (GEE) modeling was applied to compare predictions among interns, residents, and attending cardiologists, accounting for clustering by individual physician. GEE models were adjusted for study week in a sensitivity analysis to determine if predictions improved over time.

Analysis was performed using SAS 9.3 (SAS Institute Inc., Cary, NC) and R 2.14 (The R Foundation for Statistical Computing, Vienna, Austria). Institutional review board approval was granted, and physicians provided informed consent. All authors had access to primary data devoid of protected health information.

RESULTS

In total, 22 interns (<6 months experience), 25 residents (13 years experience), and 8 attending cardiologists (mean 19 9.7 years experience) were studied. Predictions were performed on 171 consecutively admitted patients. Five patients had noncardiac diagnoses and 1 patient remained admitted, leaving 165 patients for analysis. Predictions were made by all 3 physician levels on 98 patients. There were 67 patients with incomplete predictions as a result of 63 intern, 13 attending, and 4 resident predictions that were unobtainable. Absent intern data predominantly resulted from night shift admissions. Remaining missing data were due to time‐sensitive physician tasks that interfered with physician interviews.

Patient characteristics are described in Table 1. Physicians provided 415 predictions on 165 patients, 157 (95%) of whom survived to hospital discharge. Mean and median lengths of stay were 10.9 and 8 days (interquartile range [IQR], 4 to 13). Mean intern (N = 102), resident (N = 161), and attending (N = 152) predictions were 5.4 days (95% confidence interval [CI]: 4.6 to 6.2), 6.6 days (95% CI: 5.8 to 7.4) and 7.2 days (95% CI: 6.4 to 7.9), respectively. Median intern, resident, and attending predictions were 5 days (IQR, 3 to 7), 5 days (IQR, 3 to 7), and 6 days (IQR, 4 to 10). Mean differences between predicted and actual length of stay for interns, residents and attendings were 9 days (95% CI: 8.2 to 3.6), 4.3 days (95% C: 6.0 to 2.7), and 3.5 days (95% CI: 5.1 to 2.0). The mean difference between predicted and actual length of stay was statistically significant for all groups (P < 0.0001). Median intern, resident, and attending differences between predicted and actual were 2 days (IQR, 7 to 0), 2 days (IQR, 7 to 0), and 1 day (IQR, 5 to 1), respectively. Predictions correlated poorly with actual length of stay (R² = 0.11).

Ninety‐eight patients (59%) received predictions from physicians at all 3 experience levels. Mean and median lengths of stay were 11.3 days and 7.5 days (IQR, 4 to 13). Concordant with the entire cohort, median intern, resident, and attending predictions for these patients were 5 days (IQR, 3 to 7), 5 days (IQR, 3 to 7), and 6 days (IQR, 4 to 10), respectively. Differences between predicted and actual length of stay were statistically significant for all groups: the mean difference for interns, residents, and attendings was 5.8 days (95% CI: 8.2 to 3.4, P < 0.0001), 4.6 days (95% CI: 7.1 to 2.0, P = 0.0001), and 4.3 days (95% CI: 6.5 to 2.1, P = 0.0003), respectively (Figure 1).

Figure 1

There are differences among providers with improved prediction as level of experience increased, but this is not statistically significant as determined by ANOVA (p=0.64) or by GEE modeling to account for clustering of predictions by physician (P = 0.61). Analysis that adjusted for study week yielded similar results. Thus, experience did not improve accuracy.

DISCUSSION

We prospectively measured accuracy of physicians' length of stay predictions of heart failure patients and compared accuracy by experience level. All physicians underestimated length of stay, with average differences between 3.5 and 6 days. Most notably, level of experience did not improve accuracy. Although we anticipated that experience would improve prediction, our findings are not compatible with this hypothesis. Future studies of factors affecting length of stay predictions would help to better understand our findings.

Our results are consistent with small, single‐center studies of different patient and physician cohorts. Hulter Asberg found that internists at a hospital were unable to predict whether a patient would remain admitted 10 days or more, with poor interobserver reliability.[9] Mak et al. demonstrated that emergency physicians underestimated length of stay by an average of 2 days when predicting length of stay on a broad spectrum of patients in an emergency department.[10] Physician predictions of length of stay have been found to be inaccurate in a center's oncologic intensive care unit population.[11] Sullivan et al. found that academic general medicine physicians predicted discharge with 27% sensitivity the morning prior to next‐day discharge, which improved significantly to 67% by the afternoon, concluding that physicians can provide meaningful discharge predictions the afternoon prior to next‐day discharge.[12] By focusing on patients with heart failure, a major driver of hospitalization and readmission, and comparing providers by level of experience, we augment this existing body of work.

In addition to identifying patients at risk for readmission and mortality,[5, 6] accurate discharge prediction may improve safety of weekend discharges and patient satisfaction. Heart failure patients discharged on weekends receive less complete discharge instructions,[13] suffer higher mortality, and are readmitted more frequently than those discharged on weekdays.[14] Early and accurate predictions may enhance interventions targeting patients with anticipated weekend discharges. Furthermore, inadequate communication regarding anticipated discharge timing is a source of patient dissatisfaction,[15] and accurate prediction of discharge, if shared with patients, may improve patient satisfaction.

Limitations of our study include that it was a single‐center study at a large academic tertiary care hospital with predictions assessed on a teaching service. Severity of illness of this cohort may be a barrier to generalizability, and physicians may predict prognosis of healthier patients more accurately. We recorded predictions at the time of admission, and did not assess whether accuracy improved closer to discharge. We did not collect predictions from non‐physician team members. Sample size and absent data regarding the causes of prolonged hospitalization prohibited an analyses of variables associated with prediction inaccuracy.

CONCLUSIONS

Physicians do not accurately forecast heart failure patients' length of stay at the time of admission, and level of experience does not improve accuracy. Future studies are warranted to determine whether predictions closer to discharge, by an interdisciplinary team, or with assistance of risk‐prediction models are more accurate than physician predictions at admission, and whether early identification of patients at risk for prolonged hospitalization improves outcomes. Ultimately, early and accurate length of stay forecasts may improve risk stratification, patient satisfaction, and discharge planning, and reduce adverse outcomes related to at‐risk discharges.

Heart failure is a frequent cause of hospital admission in the United States, with an estimated cost of $31 billion dollars per year.[1] Discharging a patient with heart failure requires a multidisciplinary approach that includes anticipating a discharge date, scheduling follow‐up, reconciling medications, assessing home‐care or placement needs, and delivering patient education.[2, 3] Comprehensive transitional care interventions reduce readmissions and mortality.[2] Individually tailored and structured discharge plans decrease length of stay and readmissions.[3] The Centers for Medicare and Medicaid Services recently proposed that discharge planning begin within 24 hours of inpatient admissions,[4] despite inadequate data surrounding the optimal time to begin discharge planning.[3] In addition to enabling transitional care, identifying patients vulnerable to extended hospitalization aids in risk stratification, as prolonged length of stay is associated with increased risk of readmission and mortality.[5, 6]

Physicians are not able to accurately prognosticate whether patients will experience short‐term outcomes such as readmissions or mortality.[7, 8] Likewise, physicians do not predict length of stay accurately for heterogeneous patient populations,[9, 10, 11] even on the morning prior to anticipated discharge.[12] Prediction accuracy for patients admitted with heart failure, however, has not been adequately studied. The objectives of this study were to measure the accuracy of inpatient physicians' early predictions of length of stay for patients admitted with heart failure and to determine whether level of experience improved accuracy.

METHODS

In this prospective, observational study, we measured physicians' predictions of length of stay for patients admitted to a heart failure teaching service at an academic tertiary care hospital. Three resident/emntern teams rotate admitting responsibilities every 3 days, supervised by 1 attending cardiologist. Patients admitted overnight may be admitted independently by the on‐call resident without intern collaboration.

All physicians staffing our center's heart failure teaching service between August 1, 2013 and November 19, 2013 were recruited, and consecutively admitted adult patients were included. Patients were excluded if they did not have any cardiac diagnosis or if still admitted at study completion in February 2014. Deceased patients' time of death was counted as discharge.

Interns, residents, and attending cardiologists were interviewed independently within 24 hours of admission and asked to predict length of stay. Interns and residents were interviewed prior to rounds, and attendings thereafter. Electronic medical records were reviewed to determine date and time of admission and discharge, demographics, clinical variables, and discharge diagnoses.

The primary outcome was accuracy of predictions of length of stay stratified by level of experience. Based on prior pilot data, at 80% power and significance level () of 0.05, we estimated that predictions were needed on 100 patients to detect a 2‐day difference between actual and predicted length of stay.

Student t tests were used to compare the difference between predicted and actual length of stay for each level of training. Analysis of variance (ANOVA) was used to compare accuracy of prediction by training level. Generalized estimating equation (GEE) modeling was applied to compare predictions among interns, residents, and attending cardiologists, accounting for clustering by individual physician. GEE models were adjusted for study week in a sensitivity analysis to determine if predictions improved over time.

Analysis was performed using SAS 9.3 (SAS Institute Inc., Cary, NC) and R 2.14 (The R Foundation for Statistical Computing, Vienna, Austria). Institutional review board approval was granted, and physicians provided informed consent. All authors had access to primary data devoid of protected health information.

RESULTS

In total, 22 interns (<6 months experience), 25 residents (13 years experience), and 8 attending cardiologists (mean 19 9.7 years experience) were studied. Predictions were performed on 171 consecutively admitted patients. Five patients had noncardiac diagnoses and 1 patient remained admitted, leaving 165 patients for analysis. Predictions were made by all 3 physician levels on 98 patients. There were 67 patients with incomplete predictions as a result of 63 intern, 13 attending, and 4 resident predictions that were unobtainable. Absent intern data predominantly resulted from night shift admissions. Remaining missing data were due to time‐sensitive physician tasks that interfered with physician interviews.

Patient characteristics are described in Table 1. Physicians provided 415 predictions on 165 patients, 157 (95%) of whom survived to hospital discharge. Mean and median lengths of stay were 10.9 and 8 days (interquartile range [IQR], 4 to 13). Mean intern (N = 102), resident (N = 161), and attending (N = 152) predictions were 5.4 days (95% confidence interval [CI]: 4.6 to 6.2), 6.6 days (95% CI: 5.8 to 7.4) and 7.2 days (95% CI: 6.4 to 7.9), respectively. Median intern, resident, and attending predictions were 5 days (IQR, 3 to 7), 5 days (IQR, 3 to 7), and 6 days (IQR, 4 to 10). Mean differences between predicted and actual length of stay for interns, residents and attendings were 9 days (95% CI: 8.2 to 3.6), 4.3 days (95% C: 6.0 to 2.7), and 3.5 days (95% CI: 5.1 to 2.0). The mean difference between predicted and actual length of stay was statistically significant for all groups (P < 0.0001). Median intern, resident, and attending differences between predicted and actual were 2 days (IQR, 7 to 0), 2 days (IQR, 7 to 0), and 1 day (IQR, 5 to 1), respectively. Predictions correlated poorly with actual length of stay (R² = 0.11).

Ninety‐eight patients (59%) received predictions from physicians at all 3 experience levels. Mean and median lengths of stay were 11.3 days and 7.5 days (IQR, 4 to 13). Concordant with the entire cohort, median intern, resident, and attending predictions for these patients were 5 days (IQR, 3 to 7), 5 days (IQR, 3 to 7), and 6 days (IQR, 4 to 10), respectively. Differences between predicted and actual length of stay were statistically significant for all groups: the mean difference for interns, residents, and attendings was 5.8 days (95% CI: 8.2 to 3.4, P < 0.0001), 4.6 days (95% CI: 7.1 to 2.0, P = 0.0001), and 4.3 days (95% CI: 6.5 to 2.1, P = 0.0003), respectively (Figure 1).

Figure 1

There are differences among providers with improved prediction as level of experience increased, but this is not statistically significant as determined by ANOVA (p=0.64) or by GEE modeling to account for clustering of predictions by physician (P = 0.61). Analysis that adjusted for study week yielded similar results. Thus, experience did not improve accuracy.

DISCUSSION

We prospectively measured accuracy of physicians' length of stay predictions of heart failure patients and compared accuracy by experience level. All physicians underestimated length of stay, with average differences between 3.5 and 6 days. Most notably, level of experience did not improve accuracy. Although we anticipated that experience would improve prediction, our findings are not compatible with this hypothesis. Future studies of factors affecting length of stay predictions would help to better understand our findings.

Our results are consistent with small, single‐center studies of different patient and physician cohorts. Hulter Asberg found that internists at a hospital were unable to predict whether a patient would remain admitted 10 days or more, with poor interobserver reliability.[9] Mak et al. demonstrated that emergency physicians underestimated length of stay by an average of 2 days when predicting length of stay on a broad spectrum of patients in an emergency department.[10] Physician predictions of length of stay have been found to be inaccurate in a center's oncologic intensive care unit population.[11] Sullivan et al. found that academic general medicine physicians predicted discharge with 27% sensitivity the morning prior to next‐day discharge, which improved significantly to 67% by the afternoon, concluding that physicians can provide meaningful discharge predictions the afternoon prior to next‐day discharge.[12] By focusing on patients with heart failure, a major driver of hospitalization and readmission, and comparing providers by level of experience, we augment this existing body of work.

In addition to identifying patients at risk for readmission and mortality,[5, 6] accurate discharge prediction may improve safety of weekend discharges and patient satisfaction. Heart failure patients discharged on weekends receive less complete discharge instructions,[13] suffer higher mortality, and are readmitted more frequently than those discharged on weekdays.[14] Early and accurate predictions may enhance interventions targeting patients with anticipated weekend discharges. Furthermore, inadequate communication regarding anticipated discharge timing is a source of patient dissatisfaction,[15] and accurate prediction of discharge, if shared with patients, may improve patient satisfaction.

Limitations of our study include that it was a single‐center study at a large academic tertiary care hospital with predictions assessed on a teaching service. Severity of illness of this cohort may be a barrier to generalizability, and physicians may predict prognosis of healthier patients more accurately. We recorded predictions at the time of admission, and did not assess whether accuracy improved closer to discharge. We did not collect predictions from non‐physician team members. Sample size and absent data regarding the causes of prolonged hospitalization prohibited an analyses of variables associated with prediction inaccuracy.

CONCLUSIONS

Physicians do not accurately forecast heart failure patients' length of stay at the time of admission, and level of experience does not improve accuracy. Future studies are warranted to determine whether predictions closer to discharge, by an interdisciplinary team, or with assistance of risk‐prediction models are more accurate than physician predictions at admission, and whether early identification of patients at risk for prolonged hospitalization improves outcomes. Ultimately, early and accurate length of stay forecasts may improve risk stratification, patient satisfaction, and discharge planning, and reduce adverse outcomes related to at‐risk discharges.

According to the American College of Radiology Appropriateness Criteria, renal ultrasound (RUS) is the most appropriate imaging examination for evaluating patients with acute kidney injury (AKI), with a rating score of 9, representing the strongest level of recommendation.[1, 2] However, recent studies suggest that RUS may be performed in patients with certain risk factors for ureteral obstruction,[1] which would lead to important reductions in the use of medical imaging. Licurse developed a risk stratification framework to help clinicians identify patients in whom RUS was most likely to be beneficial.[2] The model was built based on clinical predictors that included race, recent exposure to inpatient nephrotoxic medications, history of hydronephrosis, recurrent urinary tract infections, benign prostatic hyperplasia, abdominal or pelvic cancer, neurogenic bladder, single functional kidney, previous pelvic surgery, congestive heart failure, and prerenal AKI. It was found, using a cross‐sectional study design that included derivation and validation samples, that a low‐risk population could be identified based on demographic and clinical risk factors; in this population, the prevalence of hydronephrosis, as well as the rate of hydronephrosis requiring an intervention, was only <1%.

However, due to several study limitations, including that it was performed at a single center,[3] the stratification prediction rule has yet to be adopted broadly. Although at least 1 other study has similarly found that RUS may not be efficacious in patients with no suggestive history and with other more likely causes for renal failure,[1] to the best of our knowledge, no large, external, prospective trial to validate the selective use of RUS in patients with AKI has been reported. Therefore, the aim of this study was to evaluate the accuracy and usefulness of the Licurse renal ultrasonography risk stratification model for hospitalized patients with AKI.

METHODS

RESULTS

During the 23‐month study period, a total of 961 RUS studies were completed for inpatients with AKI; 778 unique studies met our inclusion criteria (Figure 1).

Figure 1

Based on the manual review of objective clinical data from the random sample of 80 medical records for 480 clinical attributes, overall, there was 90.2% (433/480) concordance rate between the structured data entry and that captured in free text in the clinical notes. There were some variations in the concordance rates for each clinical attribute, ranging from 78.8% (63/80) for exposure to nephrotoxic drugs to 95% for history of congestive heart failure.

On univariate analysis, patients with past medical history of hydronephrosis had a 5‐fold higher likelihood of developing a recurrence of hydronephrosis (45.9% [50/109] vs 8.4% [56/669], P < 0.001). Similarly, they also had a 9.5‐fold higher likelihood of requiring urologic interventions related to the hydronephrosis (12.8% [14/109] vs 1.4% [9/669], P < 0.001). Having diagnoses predisposing the patient for urinary obstruction (benign prostate hyperplasia, abdominal/pelvic cancer, neurogenic bladder, single functional kidney, and history of pelvic surgery) was correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with a diagnosis predisposing the patient for urinary obstructions, 22.1% (59/267) had hydronephrosis on imaging, whereas 9.2% (47/511) of patients without such a diagnosis had hydronephrosis (P < 0.001).

Conversely, having a recent exposure to nephrotoxic medications was negatively correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with recent exposure to nephrotoxic medications, 7.1% (20/280) had hydronephrosis on imaging, whereas the prevalence of hydronephrosis was 17.3% (86/498) in patients without such an exposure (P < 0.001) (Table 1).

Adjusted for other covariates, the multiple variable model showed that a diagnosis predisposing patients for obstruction (odds ratio [OR]: 2.0, P = 0.004), history of hydronephrosis (OR: 7.4, P < 0.001), absence of a history of congestive heart failure (OR: 2.7, P = 0.009), and lack of exposure to nephrotoxic medications (OR: 1.9, P = 0.022) were statistically significant predictors for hydronephrosis (Table 2).

After applying the Licurse renal ultrasonography risk stratification model, 176 (22.6%), 190 (24.4%), and 412 (53.0%) patients were classified as low risk, medium risk, and high risk for hydronephrosis, respectively. The incidence rates for hydronephrosis in the pretest probability risk groups were 4.0%, 6.8%, and 20.9% for low‐, medium‐, and high‐risk patients, respectively (P < 0.0001). The rates for urologic interventions were 1.1%, 0.5%, and 4.9% in the risk groups from low to high (P < 0.0001) (Figure 2).

Figure 2

Overall, the Licurse model, using a cutoff between low‐risk and medium/high‐risk patients, had sensitivity of 91.3% (95% confidence interval [CI]: 73.2%‐97.6%) for HNRI and 93.4% (95% CI: 87.0%‐96.8%) for presence of HN. Specificity was low for both HNRI (23.0% [95% CI: 20.2%‐26.2%]) and HN (25.1% [95% CI: 22.0%‐28.6%]). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI, defined as the rate of imaging performed in the low‐risk group, was 22.6%. In the low‐risk group, the number needed to screen to find 1 case of HN was 25, and to find 1 case of HNRI it was 88. The negative predictive value for hydronephrosis was 96.0% (95% CI: 92.0%‐98.1%) and 98.9% for HNRI (95% CI: 96.0%‐99.7%) (Table 3).

Supporting Table 1, in the online version of this article, shows a sensitivity analysis using different cutoff thresholds in the Licurse model for classifying low‐risk patients. A lower threshold cutoff (ie, a cutoff of <1) significantly increases the sensitivity (98.1% [95% CI: 93.4%‐99.5%] for HN; 100% [95% CI: 85.7%‐100%]) for HNRI, but at the cost of a lower specificity (7.6% [95% CI: 5.8%‐9.8%] for HN and 7.0% [95% CI: 5.4%‐9.1%] for HNRI). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI would be 6.0%, the number needed to screen to find 1 case of HN would be 26, and 1 case of HNRI would be infinity.

DISCUSSION

In this prospective observational study, we found that the Licurse risk stratification model, using a cutoff between low‐ risk and medium/high‐risk patients, had 91.3% (95% CI: 73.2%‐97.6%) sensitivity for predicting patients who would require urologic intervention and 93.4% (95% CI: 87.0%‐96.8%) sensitivity for identifying patients with hydronephrosis. These findings were comparable to those found in the original validation cohort of the model, which showed sensitivity rates of 96.3% and 91.8%, respectively.[2] The negative predictive value for hydronephrosis and HNRI were sufficiently high, at 96.0% (95% CI: 92.0‐98.1) and 98.9% (95% CI: 96.0‐99.7), respectively.

Our results suggest that the Licurse model may be sufficient to rule out HN in the inpatient setting at our institution. The slight differences between the findings of our and the original studies may be due to differences in data extraction methodologies. In the original study, all data were retrospectively abstracted from medical records (discharge summaries and clinical notes) by 4 trained reviewers. However, such methodology is dependent on the quality of unstructured EHR data, which as noted in previous research, can be highly variable. Hogan and Wagner found that the correctness of EHR data can range from 44% to 100% and completeness from 1.1% to 100%, depending on the clinical concepts being studied.[4] Similarly, Thiru et al. found that the sensitivity of different types of EHR data ranged from 0.25 to 1.0.[5] Medical chart review can be labor intensive and time consuming. The lack of standardized methods for structured data capture has been a major limitation in decreasing research costs and speeding the rate of new medical discoveries through the secondary use of EHR data. By modifying our institutional clinical decision support (CDS) system to enable the necessary granular clinical data collection, we were able to obviate the need for resource intensive retrospective chart reviews. To our knowledge, this is the second example of a CDS tool specifically designed for capture of discrete data to validate a decision rule.[6] A similar process may also be useful to accelerate generation of new decision rules. With secondary use of EHR data becoming an increasingly important topic,[7] CDS may serve as an alternative method in the context of data reuse for clinical research. Based on a randomly selected chart review, it was noted that clinicians, overall, do try to communicate to the interpreting radiologists the clinical picture as accurately as they can, and rarely do providers drop their orders due to data entry.

Despite our data confirming Licurse's initial findings, it is important to note that as with any clinical prediction rules, there is a trade‐off between cost savings and potential missed diagnoses. Even the most accepted clinical decision rules, such as the Well's criteria for pulmonary embolism and deep vein thrombosis, has their inherent acceptable rates of false negative. What is considered to be acceptable may differ among providers and patients. Thus, a shared decision‐making model, in which the patient and provider actively engage in sharing of information regarding risks and benefits of both performing and bypassing the diagnostic testing, is preferred. For providers/patients who are more risk‐adverse, one could consider using a more sensitive cutoff (for example, using the <1 threshold), essentially increasing the sensitivity from 91.3% to 100% for HNRI and from 93.4% to 98.1% for HN.

Although one would not want to miss a hydronephrosis in a patient, a too aggressive imaging strategy is not without economic and downstream risks. At an estimated cost of $200 per renal ultrasonography,[2] a 22.6% reduction would result in an annual savings of nearly $20,000 at our institution. The financial costs of forgoing ultrasound studies at the risk of missing 1 case of HN or 1 case of HNRI would be $5000 and $17,600, respectively.

Data‐driven decision rules are becoming more commonly used in the current environment of increased emphasis on evidence‐based medicine.[8, 9, 10, 11, 12, 13] When applied appropriately, such prediction models can result in more efficient use of medical imaging while increasing value of care.[14, 15] However, prior to implementation in clinical practice, these models need to be externally validated across multiple institutions and in various practice settings. This is the largest study of which we are aware to validate the utility of a prediction model for AKI in the inpatient setting. Although we did find slightly smaller differences in hydronephrosis in inpatients across the low, moderate, and high pretest probability groups, this may be explained by the differences in methodology.

Our study has several limitations. First, it was performed at a single academic medical center, a similar setting as that of the original work. Thus, the generalizability of our findings in other settings is unclear. Second, it is possible that our ordering providers did not thoroughly and accurately enter data into the structured CPOE form. However, we randomly selected a sample for chart review and found 90% concordance between data captured and those in the EHR. Due to selection of our cohort that included only patients with AKI who underwent RUS, it is possible that some patients who were not imaged or imaged with other cross‐sectional modalities were excluded, resulting in differential test ordering bias. Finally, we did not include the potential benefits of RUS in affecting nonsurgical interventions of hydronephrosis (eg, Foley catheter insertion).

CONCLUSION

We found that the Licurse renal ultrasonography risk stratification model was sufficiently accurate in classifying patients at risk for ureteral obstruction among hospitalized patients with AKI.

According to the American College of Radiology Appropriateness Criteria, renal ultrasound (RUS) is the most appropriate imaging examination for evaluating patients with acute kidney injury (AKI), with a rating score of 9, representing the strongest level of recommendation.[1, 2] However, recent studies suggest that RUS may be performed in patients with certain risk factors for ureteral obstruction,[1] which would lead to important reductions in the use of medical imaging. Licurse developed a risk stratification framework to help clinicians identify patients in whom RUS was most likely to be beneficial.[2] The model was built based on clinical predictors that included race, recent exposure to inpatient nephrotoxic medications, history of hydronephrosis, recurrent urinary tract infections, benign prostatic hyperplasia, abdominal or pelvic cancer, neurogenic bladder, single functional kidney, previous pelvic surgery, congestive heart failure, and prerenal AKI. It was found, using a cross‐sectional study design that included derivation and validation samples, that a low‐risk population could be identified based on demographic and clinical risk factors; in this population, the prevalence of hydronephrosis, as well as the rate of hydronephrosis requiring an intervention, was only <1%.

However, due to several study limitations, including that it was performed at a single center,[3] the stratification prediction rule has yet to be adopted broadly. Although at least 1 other study has similarly found that RUS may not be efficacious in patients with no suggestive history and with other more likely causes for renal failure,[1] to the best of our knowledge, no large, external, prospective trial to validate the selective use of RUS in patients with AKI has been reported. Therefore, the aim of this study was to evaluate the accuracy and usefulness of the Licurse renal ultrasonography risk stratification model for hospitalized patients with AKI.

METHODS

RESULTS

During the 23‐month study period, a total of 961 RUS studies were completed for inpatients with AKI; 778 unique studies met our inclusion criteria (Figure 1).

Figure 1

Based on the manual review of objective clinical data from the random sample of 80 medical records for 480 clinical attributes, overall, there was 90.2% (433/480) concordance rate between the structured data entry and that captured in free text in the clinical notes. There were some variations in the concordance rates for each clinical attribute, ranging from 78.8% (63/80) for exposure to nephrotoxic drugs to 95% for history of congestive heart failure.

On univariate analysis, patients with past medical history of hydronephrosis had a 5‐fold higher likelihood of developing a recurrence of hydronephrosis (45.9% [50/109] vs 8.4% [56/669], P < 0.001). Similarly, they also had a 9.5‐fold higher likelihood of requiring urologic interventions related to the hydronephrosis (12.8% [14/109] vs 1.4% [9/669], P < 0.001). Having diagnoses predisposing the patient for urinary obstruction (benign prostate hyperplasia, abdominal/pelvic cancer, neurogenic bladder, single functional kidney, and history of pelvic surgery) was correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with a diagnosis predisposing the patient for urinary obstructions, 22.1% (59/267) had hydronephrosis on imaging, whereas 9.2% (47/511) of patients without such a diagnosis had hydronephrosis (P < 0.001).

Conversely, having a recent exposure to nephrotoxic medications was negatively correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with recent exposure to nephrotoxic medications, 7.1% (20/280) had hydronephrosis on imaging, whereas the prevalence of hydronephrosis was 17.3% (86/498) in patients without such an exposure (P < 0.001) (Table 1).

Adjusted for other covariates, the multiple variable model showed that a diagnosis predisposing patients for obstruction (odds ratio [OR]: 2.0, P = 0.004), history of hydronephrosis (OR: 7.4, P < 0.001), absence of a history of congestive heart failure (OR: 2.7, P = 0.009), and lack of exposure to nephrotoxic medications (OR: 1.9, P = 0.022) were statistically significant predictors for hydronephrosis (Table 2).

After applying the Licurse renal ultrasonography risk stratification model, 176 (22.6%), 190 (24.4%), and 412 (53.0%) patients were classified as low risk, medium risk, and high risk for hydronephrosis, respectively. The incidence rates for hydronephrosis in the pretest probability risk groups were 4.0%, 6.8%, and 20.9% for low‐, medium‐, and high‐risk patients, respectively (P < 0.0001). The rates for urologic interventions were 1.1%, 0.5%, and 4.9% in the risk groups from low to high (P < 0.0001) (Figure 2).

Figure 2

Overall, the Licurse model, using a cutoff between low‐risk and medium/high‐risk patients, had sensitivity of 91.3% (95% confidence interval [CI]: 73.2%‐97.6%) for HNRI and 93.4% (95% CI: 87.0%‐96.8%) for presence of HN. Specificity was low for both HNRI (23.0% [95% CI: 20.2%‐26.2%]) and HN (25.1% [95% CI: 22.0%‐28.6%]). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI, defined as the rate of imaging performed in the low‐risk group, was 22.6%. In the low‐risk group, the number needed to screen to find 1 case of HN was 25, and to find 1 case of HNRI it was 88. The negative predictive value for hydronephrosis was 96.0% (95% CI: 92.0%‐98.1%) and 98.9% for HNRI (95% CI: 96.0%‐99.7%) (Table 3).

Supporting Table 1, in the online version of this article, shows a sensitivity analysis using different cutoff thresholds in the Licurse model for classifying low‐risk patients. A lower threshold cutoff (ie, a cutoff of <1) significantly increases the sensitivity (98.1% [95% CI: 93.4%‐99.5%] for HN; 100% [95% CI: 85.7%‐100%]) for HNRI, but at the cost of a lower specificity (7.6% [95% CI: 5.8%‐9.8%] for HN and 7.0% [95% CI: 5.4%‐9.1%] for HNRI). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI would be 6.0%, the number needed to screen to find 1 case of HN would be 26, and 1 case of HNRI would be infinity.

DISCUSSION

In this prospective observational study, we found that the Licurse risk stratification model, using a cutoff between low‐ risk and medium/high‐risk patients, had 91.3% (95% CI: 73.2%‐97.6%) sensitivity for predicting patients who would require urologic intervention and 93.4% (95% CI: 87.0%‐96.8%) sensitivity for identifying patients with hydronephrosis. These findings were comparable to those found in the original validation cohort of the model, which showed sensitivity rates of 96.3% and 91.8%, respectively.[2] The negative predictive value for hydronephrosis and HNRI were sufficiently high, at 96.0% (95% CI: 92.0‐98.1) and 98.9% (95% CI: 96.0‐99.7), respectively.

Our results suggest that the Licurse model may be sufficient to rule out HN in the inpatient setting at our institution. The slight differences between the findings of our and the original studies may be due to differences in data extraction methodologies. In the original study, all data were retrospectively abstracted from medical records (discharge summaries and clinical notes) by 4 trained reviewers. However, such methodology is dependent on the quality of unstructured EHR data, which as noted in previous research, can be highly variable. Hogan and Wagner found that the correctness of EHR data can range from 44% to 100% and completeness from 1.1% to 100%, depending on the clinical concepts being studied.[4] Similarly, Thiru et al. found that the sensitivity of different types of EHR data ranged from 0.25 to 1.0.[5] Medical chart review can be labor intensive and time consuming. The lack of standardized methods for structured data capture has been a major limitation in decreasing research costs and speeding the rate of new medical discoveries through the secondary use of EHR data. By modifying our institutional clinical decision support (CDS) system to enable the necessary granular clinical data collection, we were able to obviate the need for resource intensive retrospective chart reviews. To our knowledge, this is the second example of a CDS tool specifically designed for capture of discrete data to validate a decision rule.[6] A similar process may also be useful to accelerate generation of new decision rules. With secondary use of EHR data becoming an increasingly important topic,[7] CDS may serve as an alternative method in the context of data reuse for clinical research. Based on a randomly selected chart review, it was noted that clinicians, overall, do try to communicate to the interpreting radiologists the clinical picture as accurately as they can, and rarely do providers drop their orders due to data entry.

Despite our data confirming Licurse's initial findings, it is important to note that as with any clinical prediction rules, there is a trade‐off between cost savings and potential missed diagnoses. Even the most accepted clinical decision rules, such as the Well's criteria for pulmonary embolism and deep vein thrombosis, has their inherent acceptable rates of false negative. What is considered to be acceptable may differ among providers and patients. Thus, a shared decision‐making model, in which the patient and provider actively engage in sharing of information regarding risks and benefits of both performing and bypassing the diagnostic testing, is preferred. For providers/patients who are more risk‐adverse, one could consider using a more sensitive cutoff (for example, using the <1 threshold), essentially increasing the sensitivity from 91.3% to 100% for HNRI and from 93.4% to 98.1% for HN.

Although one would not want to miss a hydronephrosis in a patient, a too aggressive imaging strategy is not without economic and downstream risks. At an estimated cost of $200 per renal ultrasonography,[2] a 22.6% reduction would result in an annual savings of nearly $20,000 at our institution. The financial costs of forgoing ultrasound studies at the risk of missing 1 case of HN or 1 case of HNRI would be $5000 and $17,600, respectively.

Data‐driven decision rules are becoming more commonly used in the current environment of increased emphasis on evidence‐based medicine.[8, 9, 10, 11, 12, 13] When applied appropriately, such prediction models can result in more efficient use of medical imaging while increasing value of care.[14, 15] However, prior to implementation in clinical practice, these models need to be externally validated across multiple institutions and in various practice settings. This is the largest study of which we are aware to validate the utility of a prediction model for AKI in the inpatient setting. Although we did find slightly smaller differences in hydronephrosis in inpatients across the low, moderate, and high pretest probability groups, this may be explained by the differences in methodology.

Our study has several limitations. First, it was performed at a single academic medical center, a similar setting as that of the original work. Thus, the generalizability of our findings in other settings is unclear. Second, it is possible that our ordering providers did not thoroughly and accurately enter data into the structured CPOE form. However, we randomly selected a sample for chart review and found 90% concordance between data captured and those in the EHR. Due to selection of our cohort that included only patients with AKI who underwent RUS, it is possible that some patients who were not imaged or imaged with other cross‐sectional modalities were excluded, resulting in differential test ordering bias. Finally, we did not include the potential benefits of RUS in affecting nonsurgical interventions of hydronephrosis (eg, Foley catheter insertion).

CONCLUSION

We found that the Licurse renal ultrasonography risk stratification model was sufficiently accurate in classifying patients at risk for ureteral obstruction among hospitalized patients with AKI.

According to the American College of Radiology Appropriateness Criteria, renal ultrasound (RUS) is the most appropriate imaging examination for evaluating patients with acute kidney injury (AKI), with a rating score of 9, representing the strongest level of recommendation.[1, 2] However, recent studies suggest that RUS may be performed in patients with certain risk factors for ureteral obstruction,[1] which would lead to important reductions in the use of medical imaging. Licurse developed a risk stratification framework to help clinicians identify patients in whom RUS was most likely to be beneficial.[2] The model was built based on clinical predictors that included race, recent exposure to inpatient nephrotoxic medications, history of hydronephrosis, recurrent urinary tract infections, benign prostatic hyperplasia, abdominal or pelvic cancer, neurogenic bladder, single functional kidney, previous pelvic surgery, congestive heart failure, and prerenal AKI. It was found, using a cross‐sectional study design that included derivation and validation samples, that a low‐risk population could be identified based on demographic and clinical risk factors; in this population, the prevalence of hydronephrosis, as well as the rate of hydronephrosis requiring an intervention, was only <1%.

However, due to several study limitations, including that it was performed at a single center,[3] the stratification prediction rule has yet to be adopted broadly. Although at least 1 other study has similarly found that RUS may not be efficacious in patients with no suggestive history and with other more likely causes for renal failure,[1] to the best of our knowledge, no large, external, prospective trial to validate the selective use of RUS in patients with AKI has been reported. Therefore, the aim of this study was to evaluate the accuracy and usefulness of the Licurse renal ultrasonography risk stratification model for hospitalized patients with AKI.

METHODS

RESULTS

During the 23‐month study period, a total of 961 RUS studies were completed for inpatients with AKI; 778 unique studies met our inclusion criteria (Figure 1).

Figure 1

Based on the manual review of objective clinical data from the random sample of 80 medical records for 480 clinical attributes, overall, there was 90.2% (433/480) concordance rate between the structured data entry and that captured in free text in the clinical notes. There were some variations in the concordance rates for each clinical attribute, ranging from 78.8% (63/80) for exposure to nephrotoxic drugs to 95% for history of congestive heart failure.

On univariate analysis, patients with past medical history of hydronephrosis had a 5‐fold higher likelihood of developing a recurrence of hydronephrosis (45.9% [50/109] vs 8.4% [56/669], P < 0.001). Similarly, they also had a 9.5‐fold higher likelihood of requiring urologic interventions related to the hydronephrosis (12.8% [14/109] vs 1.4% [9/669], P < 0.001). Having diagnoses predisposing the patient for urinary obstruction (benign prostate hyperplasia, abdominal/pelvic cancer, neurogenic bladder, single functional kidney, and history of pelvic surgery) was correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with a diagnosis predisposing the patient for urinary obstructions, 22.1% (59/267) had hydronephrosis on imaging, whereas 9.2% (47/511) of patients without such a diagnosis had hydronephrosis (P < 0.001).

Conversely, having a recent exposure to nephrotoxic medications was negatively correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with recent exposure to nephrotoxic medications, 7.1% (20/280) had hydronephrosis on imaging, whereas the prevalence of hydronephrosis was 17.3% (86/498) in patients without such an exposure (P < 0.001) (Table 1).

Adjusted for other covariates, the multiple variable model showed that a diagnosis predisposing patients for obstruction (odds ratio [OR]: 2.0, P = 0.004), history of hydronephrosis (OR: 7.4, P < 0.001), absence of a history of congestive heart failure (OR: 2.7, P = 0.009), and lack of exposure to nephrotoxic medications (OR: 1.9, P = 0.022) were statistically significant predictors for hydronephrosis (Table 2).

After applying the Licurse renal ultrasonography risk stratification model, 176 (22.6%), 190 (24.4%), and 412 (53.0%) patients were classified as low risk, medium risk, and high risk for hydronephrosis, respectively. The incidence rates for hydronephrosis in the pretest probability risk groups were 4.0%, 6.8%, and 20.9% for low‐, medium‐, and high‐risk patients, respectively (P < 0.0001). The rates for urologic interventions were 1.1%, 0.5%, and 4.9% in the risk groups from low to high (P < 0.0001) (Figure 2).

Figure 2

Overall, the Licurse model, using a cutoff between low‐risk and medium/high‐risk patients, had sensitivity of 91.3% (95% confidence interval [CI]: 73.2%‐97.6%) for HNRI and 93.4% (95% CI: 87.0%‐96.8%) for presence of HN. Specificity was low for both HNRI (23.0% [95% CI: 20.2%‐26.2%]) and HN (25.1% [95% CI: 22.0%‐28.6%]). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI, defined as the rate of imaging performed in the low‐risk group, was 22.6%. In the low‐risk group, the number needed to screen to find 1 case of HN was 25, and to find 1 case of HNRI it was 88. The negative predictive value for hydronephrosis was 96.0% (95% CI: 92.0%‐98.1%) and 98.9% for HNRI (95% CI: 96.0%‐99.7%) (Table 3).

Supporting Table 1, in the online version of this article, shows a sensitivity analysis using different cutoff thresholds in the Licurse model for classifying low‐risk patients. A lower threshold cutoff (ie, a cutoff of <1) significantly increases the sensitivity (98.1% [95% CI: 93.4%‐99.5%] for HN; 100% [95% CI: 85.7%‐100%]) for HNRI, but at the cost of a lower specificity (7.6% [95% CI: 5.8%‐9.8%] for HN and 7.0% [95% CI: 5.4%‐9.1%] for HNRI). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI would be 6.0%, the number needed to screen to find 1 case of HN would be 26, and 1 case of HNRI would be infinity.

DISCUSSION

In this prospective observational study, we found that the Licurse risk stratification model, using a cutoff between low‐ risk and medium/high‐risk patients, had 91.3% (95% CI: 73.2%‐97.6%) sensitivity for predicting patients who would require urologic intervention and 93.4% (95% CI: 87.0%‐96.8%) sensitivity for identifying patients with hydronephrosis. These findings were comparable to those found in the original validation cohort of the model, which showed sensitivity rates of 96.3% and 91.8%, respectively.[2] The negative predictive value for hydronephrosis and HNRI were sufficiently high, at 96.0% (95% CI: 92.0‐98.1) and 98.9% (95% CI: 96.0‐99.7), respectively.

Our results suggest that the Licurse model may be sufficient to rule out HN in the inpatient setting at our institution. The slight differences between the findings of our and the original studies may be due to differences in data extraction methodologies. In the original study, all data were retrospectively abstracted from medical records (discharge summaries and clinical notes) by 4 trained reviewers. However, such methodology is dependent on the quality of unstructured EHR data, which as noted in previous research, can be highly variable. Hogan and Wagner found that the correctness of EHR data can range from 44% to 100% and completeness from 1.1% to 100%, depending on the clinical concepts being studied.[4] Similarly, Thiru et al. found that the sensitivity of different types of EHR data ranged from 0.25 to 1.0.[5] Medical chart review can be labor intensive and time consuming. The lack of standardized methods for structured data capture has been a major limitation in decreasing research costs and speeding the rate of new medical discoveries through the secondary use of EHR data. By modifying our institutional clinical decision support (CDS) system to enable the necessary granular clinical data collection, we were able to obviate the need for resource intensive retrospective chart reviews. To our knowledge, this is the second example of a CDS tool specifically designed for capture of discrete data to validate a decision rule.[6] A similar process may also be useful to accelerate generation of new decision rules. With secondary use of EHR data becoming an increasingly important topic,[7] CDS may serve as an alternative method in the context of data reuse for clinical research. Based on a randomly selected chart review, it was noted that clinicians, overall, do try to communicate to the interpreting radiologists the clinical picture as accurately as they can, and rarely do providers drop their orders due to data entry.

Despite our data confirming Licurse's initial findings, it is important to note that as with any clinical prediction rules, there is a trade‐off between cost savings and potential missed diagnoses. Even the most accepted clinical decision rules, such as the Well's criteria for pulmonary embolism and deep vein thrombosis, has their inherent acceptable rates of false negative. What is considered to be acceptable may differ among providers and patients. Thus, a shared decision‐making model, in which the patient and provider actively engage in sharing of information regarding risks and benefits of both performing and bypassing the diagnostic testing, is preferred. For providers/patients who are more risk‐adverse, one could consider using a more sensitive cutoff (for example, using the <1 threshold), essentially increasing the sensitivity from 91.3% to 100% for HNRI and from 93.4% to 98.1% for HN.

Although one would not want to miss a hydronephrosis in a patient, a too aggressive imaging strategy is not without economic and downstream risks. At an estimated cost of $200 per renal ultrasonography,[2] a 22.6% reduction would result in an annual savings of nearly $20,000 at our institution. The financial costs of forgoing ultrasound studies at the risk of missing 1 case of HN or 1 case of HNRI would be $5000 and $17,600, respectively.

Data‐driven decision rules are becoming more commonly used in the current environment of increased emphasis on evidence‐based medicine.[8, 9, 10, 11, 12, 13] When applied appropriately, such prediction models can result in more efficient use of medical imaging while increasing value of care.[14, 15] However, prior to implementation in clinical practice, these models need to be externally validated across multiple institutions and in various practice settings. This is the largest study of which we are aware to validate the utility of a prediction model for AKI in the inpatient setting. Although we did find slightly smaller differences in hydronephrosis in inpatients across the low, moderate, and high pretest probability groups, this may be explained by the differences in methodology.

Our study has several limitations. First, it was performed at a single academic medical center, a similar setting as that of the original work. Thus, the generalizability of our findings in other settings is unclear. Second, it is possible that our ordering providers did not thoroughly and accurately enter data into the structured CPOE form. However, we randomly selected a sample for chart review and found 90% concordance between data captured and those in the EHR. Due to selection of our cohort that included only patients with AKI who underwent RUS, it is possible that some patients who were not imaged or imaged with other cross‐sectional modalities were excluded, resulting in differential test ordering bias. Finally, we did not include the potential benefits of RUS in affecting nonsurgical interventions of hydronephrosis (eg, Foley catheter insertion).

CONCLUSION

We found that the Licurse renal ultrasonography risk stratification model was sufficiently accurate in classifying patients at risk for ureteral obstruction among hospitalized patients with AKI.

Frailty is a state of vulnerability that encompasses a heterogeneous group of people.[1] Because it lacks a precise definition, multiple tools have been developed to identify frailty in both clinical and research settings.[2, 3, 4] Prevalence of frailty depends on the frailty assessment tool used and the population studied, ranging from 4% to 17% when the Fried score[5, 6, 7] is used and from 5% to 44%[5, 7, 8] when cumulative deficit models like the Frailty Index are utilized, with the lower prevalences being in younger community‐dwelling elderly populations and the higher proportions being in older institutionalized populations.

The Frailty Index, also called the Burden or Cumulative Deficit Model, comprises 70 domains that include mobility, mood, function, cognitive impairment, and disease states. It is multidimensional and allows for patients to be categorized on a continuum of frailty, but it is extremely difficult to apply in clinical practice. Recognizing this, Rockwood et al.[9] developed and validated the Clinical Frailty Scale (CFS) in the Canadian Study of Health and Aging. The CFS classifies patients into 1 of 9 categories: very fit, well, managing well, vulnerable, mildly frail (needs help with at least 1 instrumental activity of daily living such as shopping, finances, meal preparation, or housework), moderately frail (needs help with 1 or 2 activities of daily living such as bathing and dressing), severely frail (dependent for personal care), very severely frail (bedbound), and terminally ill. Although this tool is easy to use in clinical practice, it reflects a gestalt impression and requires some clinical judgement.

The Fried score[6] is a prototypical phenotype tool based on 5 criteria that include weight loss, self‐reported exhaustion, low energy expenditure, slowness of gait, and weakness. Recent evidence has suggested that slow gait (or dysmobility) alone may also be a potential screening test for frailty.[10] A recent systematic review[11] demonstrated an association between slow gait (dysmobility) and increased mortality. Dysmobility negatively impacts quality of life and has a strong association with disability resulting in the need for an increased level of care.[12] The Timed Up and Go Test (TUGT) is one method of assessing mobility which is relatively easy to perform, does not require special equipment, and is feasible to use in clinical settings.[13] However, whether impaired mobility predicts outcomes within the first 30 days after hospital discharge (a timeframe highlighted in the Affordable Care Act and used by the Centers for Medicare and Medicaid Services as an important hospital quality indicator) is still uncertain.

The aim of this study was to compare frailty assessments using the CFS and 2 of the most commonly used phenotypic tools (a modified Fried score and the TUGT as a proxy for mobility assessment) to determine which tools best predict postdischarge outcomes.

METHODS

RESULTS

Of 1124 potentially eligible patients, 626 were excluded because of patient refusal (n = 227); transfer to/from another hospital, long‐term care facility, or out of province (n = 189); moderate to severe cognitive impairment (n = 88); language barriers (n = 71); or foreshortened life expectancy (n = 51). Another 3 patients withdrew consent prior to outcome assessment. The 495 patients we recruited and had outcome data for had a mean age of 64 years, 19.6% were older than 80 years, 50% were women, and the patients had a mean of 4.2 comorbidities and mean Charlson score of 2.4. The 4 most common reasons for hospital admission were heart failure, pneumonia, chronic obstructive pulmonary disease, and urinary tract infection, and the median length of stay was 5 days (interquartile range: 49 days).

Prevalence of Frailty According to Different Definitions

Although the CFS assessment resulted in 162 (33%) patients being deemed frail, only 82 (51%) of those patients also met the phenotype frailty definition using either the Fried model or the TUGT, and 49 (10%) patients who were not classified as frail on the CFS met either of the phenotypic definitions of frailty (Figure 1). Overall, 211 (43%) patients were frail according to at least 1 assessment, and 46 (9%) met all 3 frailty definitions. In the subgroup of 245 patients older than 65 years, 137 (56%) were frail according to at least 1 assessment, 38 (16%) met all 3 frailty definitions, and 27 (11%) of those patients classified as not frail on the CFS met either phenotypic definition of frailty. Agreement between TUGT and CFS or CFS and Fried was relatively poor with kappas of 0.31 (95% confidence interval [CI]: 0.23‐0.40) and 0.33 (95% CI: 0.25‐0.42), respectively. It is noteworthy that some patients deemed nonfrail on the CFS had slow gait speeds, and most CFS‐frail patients had gait speeds in the nonfrail range (Figure 2).

Figure 1

Figure 2

DISCUSSION

In summary, we found that of patients being discharged from general medical wards who were frail according to at least 1 of the 3 tools we used, only 22% met all 3 frailty case definitions (including only 28% of elderly patients deemed frail by at least 1 definition). There was surprisingly poor correlation between phenotypic markers of frailty such as poor mobility (slow TUGT) or the modified Fried Index and the CFS, even amongt elderly patients. The most clinically useful of the frailty assessment tools (both overall and in those patients who are elderly) appears to be the CFS, because it more accurately identifies those at higher risk of adverse outcomes after discharge, does not require special equipment to conduct, and is faster to do than the phenotypic assessment models we tested. We have also previously demonstrated that the CFS, after a brief training period identical to that used in this study, is reproducible between observers[19] and remains an independent predictor of adverse 30‐day outcomes even after adjusting for age, sex, comorbidities, and the LACE (length of stay, acuity of the admission, comorbidity, emergency room visits during the previous 6 months) score.[14]

Although some[10] have advocated for the use of mobility assessments (such as gait speed) as a frailty marker due to its ease of measurement and objectivity, we found that slow TUGT (which is a marker for mobility and not just slow gait speed) was not an independent prognostic marker for postdischarge outcomes. We hypothesize that the phenotypic models of frailty performed less well than the CFS as they focus on the measurement of particular physical attributes and do not take into account cognitive or psychosocial characteristics or comorbidity burden that also influence postdischarge outcomes. As well, the CFS captures the patients' baseline status prior to acute illness, whereas the phenotypic measures were assessed just prior to discharge and thus may provide less information about eventual recovery potential. Some have suggested that repeating phenotype measures postdischarge might be more informative,[20] but this would reduce clinical applicability a great deal. Certainly, an analysis[21] of the Cardiovascular Health Study cohort demonstrated that cumulative deficit models of frailty (for which the CFS is an accurate proxy[9, 15]) better predicted risk of death than phenotypic models.

Although a number of published studies have shown similar results to ours in that frail patients are at greater risk for death and/or hospitalization,[22, 23, 24] there is surprisingly little literature on the comparative predictive performance of the different frailty instruments and the extent to which they overlap. Cigolle et al.[25] compared 3 frailty scales (the Functional Domain Model, the Burden Model, and the Fried score) in the Health and Retirement Study and, similarly to us, found that although 30.2% were frail on at least 1 of these scales, only 3.1% were deemed frail by all 3. The Conselice Study of Brain Aging[5] also reported that a deficit accumulation model defined a much higher prevalence of frailty (37.6%) than the 11.6% identified using the phenotypic Study of Osteoporotic Fractures (SOF) index based on weight loss, mobility, and level of energy. Another study[26] reported that risk models incorporating either the SOF index or the Fried score exhibited C statistics of only 0.61 for predicting falls in elderly females. A cohort study[27] from 2 English general medical units also found that none of the 5 frailty models was particularly accurate at predicting risk of readmission at 3 months, with C statistics ranging between 0.52 and 0.57. Although frailty assessment at time of hospital admission predicted in‐hospital mortality and length of stay in another English study, it was not independently associated with 30‐day outcomes after adjusting for age, sex, and comorbidities including dementia.[27] To our knowledge, these latter 2 are the only other studies reported to date performed in hospitalized patients to assess whether frailty assessment helps predict postdischarge outcomes. Thus, the poor C statistics we found for all of our frailty tools confirms prior literature that frailty assessment alone is inadequate to accurately identify those patients at highest risk for poor outcomes in the first 30 days after discharge. However, frailty assessment together with consideration of each individual's comorbidities, cognitive status, psychosocial circumstances, and environment can be useful to flag those individuals who may need extra attention postdischarge to optimize outcomes.

Frailty is a state of vulnerability that encompasses a heterogeneous group of people.[1] Because it lacks a precise definition, multiple tools have been developed to identify frailty in both clinical and research settings.[2, 3, 4] Prevalence of frailty depends on the frailty assessment tool used and the population studied, ranging from 4% to 17% when the Fried score[5, 6, 7] is used and from 5% to 44%[5, 7, 8] when cumulative deficit models like the Frailty Index are utilized, with the lower prevalences being in younger community‐dwelling elderly populations and the higher proportions being in older institutionalized populations.

The Frailty Index, also called the Burden or Cumulative Deficit Model, comprises 70 domains that include mobility, mood, function, cognitive impairment, and disease states. It is multidimensional and allows for patients to be categorized on a continuum of frailty, but it is extremely difficult to apply in clinical practice. Recognizing this, Rockwood et al.[9] developed and validated the Clinical Frailty Scale (CFS) in the Canadian Study of Health and Aging. The CFS classifies patients into 1 of 9 categories: very fit, well, managing well, vulnerable, mildly frail (needs help with at least 1 instrumental activity of daily living such as shopping, finances, meal preparation, or housework), moderately frail (needs help with 1 or 2 activities of daily living such as bathing and dressing), severely frail (dependent for personal care), very severely frail (bedbound), and terminally ill. Although this tool is easy to use in clinical practice, it reflects a gestalt impression and requires some clinical judgement.

The Fried score[6] is a prototypical phenotype tool based on 5 criteria that include weight loss, self‐reported exhaustion, low energy expenditure, slowness of gait, and weakness. Recent evidence has suggested that slow gait (or dysmobility) alone may also be a potential screening test for frailty.[10] A recent systematic review[11] demonstrated an association between slow gait (dysmobility) and increased mortality. Dysmobility negatively impacts quality of life and has a strong association with disability resulting in the need for an increased level of care.[12] The Timed Up and Go Test (TUGT) is one method of assessing mobility which is relatively easy to perform, does not require special equipment, and is feasible to use in clinical settings.[13] However, whether impaired mobility predicts outcomes within the first 30 days after hospital discharge (a timeframe highlighted in the Affordable Care Act and used by the Centers for Medicare and Medicaid Services as an important hospital quality indicator) is still uncertain.

The aim of this study was to compare frailty assessments using the CFS and 2 of the most commonly used phenotypic tools (a modified Fried score and the TUGT as a proxy for mobility assessment) to determine which tools best predict postdischarge outcomes.

METHODS

RESULTS

Of 1124 potentially eligible patients, 626 were excluded because of patient refusal (n = 227); transfer to/from another hospital, long‐term care facility, or out of province (n = 189); moderate to severe cognitive impairment (n = 88); language barriers (n = 71); or foreshortened life expectancy (n = 51). Another 3 patients withdrew consent prior to outcome assessment. The 495 patients we recruited and had outcome data for had a mean age of 64 years, 19.6% were older than 80 years, 50% were women, and the patients had a mean of 4.2 comorbidities and mean Charlson score of 2.4. The 4 most common reasons for hospital admission were heart failure, pneumonia, chronic obstructive pulmonary disease, and urinary tract infection, and the median length of stay was 5 days (interquartile range: 49 days).

Prevalence of Frailty According to Different Definitions

Although the CFS assessment resulted in 162 (33%) patients being deemed frail, only 82 (51%) of those patients also met the phenotype frailty definition using either the Fried model or the TUGT, and 49 (10%) patients who were not classified as frail on the CFS met either of the phenotypic definitions of frailty (Figure 1). Overall, 211 (43%) patients were frail according to at least 1 assessment, and 46 (9%) met all 3 frailty definitions. In the subgroup of 245 patients older than 65 years, 137 (56%) were frail according to at least 1 assessment, 38 (16%) met all 3 frailty definitions, and 27 (11%) of those patients classified as not frail on the CFS met either phenotypic definition of frailty. Agreement between TUGT and CFS or CFS and Fried was relatively poor with kappas of 0.31 (95% confidence interval [CI]: 0.23‐0.40) and 0.33 (95% CI: 0.25‐0.42), respectively. It is noteworthy that some patients deemed nonfrail on the CFS had slow gait speeds, and most CFS‐frail patients had gait speeds in the nonfrail range (Figure 2).

Figure 1

Figure 2

DISCUSSION

In summary, we found that of patients being discharged from general medical wards who were frail according to at least 1 of the 3 tools we used, only 22% met all 3 frailty case definitions (including only 28% of elderly patients deemed frail by at least 1 definition). There was surprisingly poor correlation between phenotypic markers of frailty such as poor mobility (slow TUGT) or the modified Fried Index and the CFS, even amongt elderly patients. The most clinically useful of the frailty assessment tools (both overall and in those patients who are elderly) appears to be the CFS, because it more accurately identifies those at higher risk of adverse outcomes after discharge, does not require special equipment to conduct, and is faster to do than the phenotypic assessment models we tested. We have also previously demonstrated that the CFS, after a brief training period identical to that used in this study, is reproducible between observers[19] and remains an independent predictor of adverse 30‐day outcomes even after adjusting for age, sex, comorbidities, and the LACE (length of stay, acuity of the admission, comorbidity, emergency room visits during the previous 6 months) score.[14]

Although some[10] have advocated for the use of mobility assessments (such as gait speed) as a frailty marker due to its ease of measurement and objectivity, we found that slow TUGT (which is a marker for mobility and not just slow gait speed) was not an independent prognostic marker for postdischarge outcomes. We hypothesize that the phenotypic models of frailty performed less well than the CFS as they focus on the measurement of particular physical attributes and do not take into account cognitive or psychosocial characteristics or comorbidity burden that also influence postdischarge outcomes. As well, the CFS captures the patients' baseline status prior to acute illness, whereas the phenotypic measures were assessed just prior to discharge and thus may provide less information about eventual recovery potential. Some have suggested that repeating phenotype measures postdischarge might be more informative,[20] but this would reduce clinical applicability a great deal. Certainly, an analysis[21] of the Cardiovascular Health Study cohort demonstrated that cumulative deficit models of frailty (for which the CFS is an accurate proxy[9, 15]) better predicted risk of death than phenotypic models.

Although a number of published studies have shown similar results to ours in that frail patients are at greater risk for death and/or hospitalization,[22, 23, 24] there is surprisingly little literature on the comparative predictive performance of the different frailty instruments and the extent to which they overlap. Cigolle et al.[25] compared 3 frailty scales (the Functional Domain Model, the Burden Model, and the Fried score) in the Health and Retirement Study and, similarly to us, found that although 30.2% were frail on at least 1 of these scales, only 3.1% were deemed frail by all 3. The Conselice Study of Brain Aging[5] also reported that a deficit accumulation model defined a much higher prevalence of frailty (37.6%) than the 11.6% identified using the phenotypic Study of Osteoporotic Fractures (SOF) index based on weight loss, mobility, and level of energy. Another study[26] reported that risk models incorporating either the SOF index or the Fried score exhibited C statistics of only 0.61 for predicting falls in elderly females. A cohort study[27] from 2 English general medical units also found that none of the 5 frailty models was particularly accurate at predicting risk of readmission at 3 months, with C statistics ranging between 0.52 and 0.57. Although frailty assessment at time of hospital admission predicted in‐hospital mortality and length of stay in another English study, it was not independently associated with 30‐day outcomes after adjusting for age, sex, and comorbidities including dementia.[27] To our knowledge, these latter 2 are the only other studies reported to date performed in hospitalized patients to assess whether frailty assessment helps predict postdischarge outcomes. Thus, the poor C statistics we found for all of our frailty tools confirms prior literature that frailty assessment alone is inadequate to accurately identify those patients at highest risk for poor outcomes in the first 30 days after discharge. However, frailty assessment together with consideration of each individual's comorbidities, cognitive status, psychosocial circumstances, and environment can be useful to flag those individuals who may need extra attention postdischarge to optimize outcomes.

Frailty is a state of vulnerability that encompasses a heterogeneous group of people.[1] Because it lacks a precise definition, multiple tools have been developed to identify frailty in both clinical and research settings.[2, 3, 4] Prevalence of frailty depends on the frailty assessment tool used and the population studied, ranging from 4% to 17% when the Fried score[5, 6, 7] is used and from 5% to 44%[5, 7, 8] when cumulative deficit models like the Frailty Index are utilized, with the lower prevalences being in younger community‐dwelling elderly populations and the higher proportions being in older institutionalized populations.

The Frailty Index, also called the Burden or Cumulative Deficit Model, comprises 70 domains that include mobility, mood, function, cognitive impairment, and disease states. It is multidimensional and allows for patients to be categorized on a continuum of frailty, but it is extremely difficult to apply in clinical practice. Recognizing this, Rockwood et al.[9] developed and validated the Clinical Frailty Scale (CFS) in the Canadian Study of Health and Aging. The CFS classifies patients into 1 of 9 categories: very fit, well, managing well, vulnerable, mildly frail (needs help with at least 1 instrumental activity of daily living such as shopping, finances, meal preparation, or housework), moderately frail (needs help with 1 or 2 activities of daily living such as bathing and dressing), severely frail (dependent for personal care), very severely frail (bedbound), and terminally ill. Although this tool is easy to use in clinical practice, it reflects a gestalt impression and requires some clinical judgement.

The Fried score[6] is a prototypical phenotype tool based on 5 criteria that include weight loss, self‐reported exhaustion, low energy expenditure, slowness of gait, and weakness. Recent evidence has suggested that slow gait (or dysmobility) alone may also be a potential screening test for frailty.[10] A recent systematic review[11] demonstrated an association between slow gait (dysmobility) and increased mortality. Dysmobility negatively impacts quality of life and has a strong association with disability resulting in the need for an increased level of care.[12] The Timed Up and Go Test (TUGT) is one method of assessing mobility which is relatively easy to perform, does not require special equipment, and is feasible to use in clinical settings.[13] However, whether impaired mobility predicts outcomes within the first 30 days after hospital discharge (a timeframe highlighted in the Affordable Care Act and used by the Centers for Medicare and Medicaid Services as an important hospital quality indicator) is still uncertain.

The aim of this study was to compare frailty assessments using the CFS and 2 of the most commonly used phenotypic tools (a modified Fried score and the TUGT as a proxy for mobility assessment) to determine which tools best predict postdischarge outcomes.

METHODS

RESULTS

Of 1124 potentially eligible patients, 626 were excluded because of patient refusal (n = 227); transfer to/from another hospital, long‐term care facility, or out of province (n = 189); moderate to severe cognitive impairment (n = 88); language barriers (n = 71); or foreshortened life expectancy (n = 51). Another 3 patients withdrew consent prior to outcome assessment. The 495 patients we recruited and had outcome data for had a mean age of 64 years, 19.6% were older than 80 years, 50% were women, and the patients had a mean of 4.2 comorbidities and mean Charlson score of 2.4. The 4 most common reasons for hospital admission were heart failure, pneumonia, chronic obstructive pulmonary disease, and urinary tract infection, and the median length of stay was 5 days (interquartile range: 49 days).

Prevalence of Frailty According to Different Definitions

Although the CFS assessment resulted in 162 (33%) patients being deemed frail, only 82 (51%) of those patients also met the phenotype frailty definition using either the Fried model or the TUGT, and 49 (10%) patients who were not classified as frail on the CFS met either of the phenotypic definitions of frailty (Figure 1). Overall, 211 (43%) patients were frail according to at least 1 assessment, and 46 (9%) met all 3 frailty definitions. In the subgroup of 245 patients older than 65 years, 137 (56%) were frail according to at least 1 assessment, 38 (16%) met all 3 frailty definitions, and 27 (11%) of those patients classified as not frail on the CFS met either phenotypic definition of frailty. Agreement between TUGT and CFS or CFS and Fried was relatively poor with kappas of 0.31 (95% confidence interval [CI]: 0.23‐0.40) and 0.33 (95% CI: 0.25‐0.42), respectively. It is noteworthy that some patients deemed nonfrail on the CFS had slow gait speeds, and most CFS‐frail patients had gait speeds in the nonfrail range (Figure 2).

Figure 1

Figure 2

DISCUSSION

In summary, we found that of patients being discharged from general medical wards who were frail according to at least 1 of the 3 tools we used, only 22% met all 3 frailty case definitions (including only 28% of elderly patients deemed frail by at least 1 definition). There was surprisingly poor correlation between phenotypic markers of frailty such as poor mobility (slow TUGT) or the modified Fried Index and the CFS, even amongt elderly patients. The most clinically useful of the frailty assessment tools (both overall and in those patients who are elderly) appears to be the CFS, because it more accurately identifies those at higher risk of adverse outcomes after discharge, does not require special equipment to conduct, and is faster to do than the phenotypic assessment models we tested. We have also previously demonstrated that the CFS, after a brief training period identical to that used in this study, is reproducible between observers[19] and remains an independent predictor of adverse 30‐day outcomes even after adjusting for age, sex, comorbidities, and the LACE (length of stay, acuity of the admission, comorbidity, emergency room visits during the previous 6 months) score.[14]

Although some[10] have advocated for the use of mobility assessments (such as gait speed) as a frailty marker due to its ease of measurement and objectivity, we found that slow TUGT (which is a marker for mobility and not just slow gait speed) was not an independent prognostic marker for postdischarge outcomes. We hypothesize that the phenotypic models of frailty performed less well than the CFS as they focus on the measurement of particular physical attributes and do not take into account cognitive or psychosocial characteristics or comorbidity burden that also influence postdischarge outcomes. As well, the CFS captures the patients' baseline status prior to acute illness, whereas the phenotypic measures were assessed just prior to discharge and thus may provide less information about eventual recovery potential. Some have suggested that repeating phenotype measures postdischarge might be more informative,[20] but this would reduce clinical applicability a great deal. Certainly, an analysis[21] of the Cardiovascular Health Study cohort demonstrated that cumulative deficit models of frailty (for which the CFS is an accurate proxy[9, 15]) better predicted risk of death than phenotypic models.

Although a number of published studies have shown similar results to ours in that frail patients are at greater risk for death and/or hospitalization,[22, 23, 24] there is surprisingly little literature on the comparative predictive performance of the different frailty instruments and the extent to which they overlap. Cigolle et al.[25] compared 3 frailty scales (the Functional Domain Model, the Burden Model, and the Fried score) in the Health and Retirement Study and, similarly to us, found that although 30.2% were frail on at least 1 of these scales, only 3.1% were deemed frail by all 3. The Conselice Study of Brain Aging[5] also reported that a deficit accumulation model defined a much higher prevalence of frailty (37.6%) than the 11.6% identified using the phenotypic Study of Osteoporotic Fractures (SOF) index based on weight loss, mobility, and level of energy. Another study[26] reported that risk models incorporating either the SOF index or the Fried score exhibited C statistics of only 0.61 for predicting falls in elderly females. A cohort study[27] from 2 English general medical units also found that none of the 5 frailty models was particularly accurate at predicting risk of readmission at 3 months, with C statistics ranging between 0.52 and 0.57. Although frailty assessment at time of hospital admission predicted in‐hospital mortality and length of stay in another English study, it was not independently associated with 30‐day outcomes after adjusting for age, sex, and comorbidities including dementia.[27] To our knowledge, these latter 2 are the only other studies reported to date performed in hospitalized patients to assess whether frailty assessment helps predict postdischarge outcomes. Thus, the poor C statistics we found for all of our frailty tools confirms prior literature that frailty assessment alone is inadequate to accurately identify those patients at highest risk for poor outcomes in the first 30 days after discharge. However, frailty assessment together with consideration of each individual's comorbidities, cognitive status, psychosocial circumstances, and environment can be useful to flag those individuals who may need extra attention postdischarge to optimize outcomes.

The hospital‐wide all‐cause 30‐day readmission rate is a key quality measure associated with patient outcomes, cost of care, and wasted hospital resources.[1] The estimated 20% readmission rate of Medicare patients and the associated $17 billion annual cost of readmissions led the Centers for Medicare and Medicaid Services (CMS) to implement policies that limit reimbursement for 30‐day unplanned readmissions and thus place hospitals with high readmission rates at financial risk.[1, 2]

The variation in readmission rates between hospitals is well documented in the literature.[3, 4] Singh et al. found that 9.3% of the variation in readmissions can be explained by hospital characteristics.[4] Hospital factors associated with lower readmission rates include not‐for‐profit ownership, hospital size, and nursing staffing levels.[5, 6, 7] Other studies found an association between environmental factors such as the percent of patients living under the poverty line and higher readmission rates.[7] The recent publicly available CMS data on readmission rates allows us to further our understanding of hospital characteristics that explain the variation in readmission rates. In this article, we are specifically interested in hospitalist staffing levels and hospital‐physician arrangements such as physician integration level and physician ownership. Moreover, we are interested in novel organizational variables, specifically, the adoption of a medical home model, which has been ignored by previous research. Medical homes are associated with better quality[8]; hospitals that already adopted the medical home model might be better equipped to coordinate care after the patients are discharged.

In recent years, the number of hospitals relying on hospitalists to provide inpatient care has been on the rise. As more hospitals employ hospitalists, it is important to understand how hospitalist staffing levels are associated with quality. Previous studies have linked hospitalists with lower hospital mortality rates,[7] lower cost of care,[9, 10] and lower readmission rates.[10, 11] Goodrich et al., on the other hand, did not find a significant relationship between the presence of hospitalists and mortality or readmission rates.[12] In a recent study, hospitalists indicated that heavy workloads limited the time they had available to communicate with patients, which negatively influenced quality and patient satisfaction, and resulted in delayed admissions and discharges.[13]

The main objective of this article was therefore to study the association between hospitalist staffing levels and hospital‐wide all‐cause readmission rates. Most empirical studies examining the relationship between hospitalist staffing and quality of inpatient care have predominantly focused on whether the presence of hospitalists who provided care at a hospital influenced mortality or readmissions.[11, 12] In this article, we contribute to the literature by examining how staffing levels measured by the ratio of hospitalists to general medical and surgical beds is associated with 30‐day readmission rates. We predict that there is a positive association between readmission rates and hospitalists per bed.

Hospitals have a broad range of contractual arrangements or integration levels with physicians, with employment being the highest level. A hospital can rely on physicians who have admitting privileges but are not salaried employees of the hospital to treat a large portion of its inpatient population. In the past few years, with the passage of the Patient Protection and Affordable Care Act (2010) and the shift in reimbursement towards Value Based Purchasing (VBP), more hospitals are choosing to ensure that physicians are strongly integrated within the hospital by adopting an employment‐based model. Moreover, hospitals view physician employment as a strategic move that will help ensure or expand their market share.[14] For instance, the number of surgeons who identified as self‐employed dropped from 48% in 2001 to 28% in 2011, and this reduction is attributed to the shift toward hospital employment of physicians.[15] Despite the evolving models of hospital‐physician arrangements, little is understood on how the adoption of the integrated salary model, in addition to the equity and foundation models, which are classified by Baker et al. as the highest level of integration, influence quality.[16] Therefore, another objective of this article was to examine the association between hospital‐physician arrangements and all‐cause unplanned readmission rates.

METHODS

RESULTS

Among our sample of 1756 hospitals, the median 30‐day all‐cause readmission rate was 16%, with the middle 50% of hospitals with readmission rates between 15.2% and 16.5%. All of the hospitals in this study reported that hospitalists provide care at the hospitals. The median Medicare share of hospital admissions was 48.46%, and the median Medicaid share of hospital admissions was 16.4%. Fifty‐one percent of the hospitals in our sample were fully integrated. Fifty percent of hospitals had 9 or fewer hospitalists per 100 general medical and surgical beds. Only 5% of the hospitals had partial or full physician ownership. Twenty‐nine percent of hospitals had an established medical home program. Table 1 provides summary statistics and the data sources of all the variables included in the study.

To compare readmission rates, we created a dummy variable that divided the sample into 2 categories: hospitals with low hospitalist staffing levels (hospitalists per general medical and surgical beds is less than the median) and high hospitalist staffing (hospitalists per general medical and surgical bed ratio is more than the median). We then used t tests to compare all‐cause readmission rates between hospitals with low and high hospitalist staffing levels, physician owned versus nonphysician owned, and fully integrated versus not fully integrated. We also used single‐factor analysis of variance (ANOVA) to compare readmission rates between nonteaching, minor teaching, and major teaching hospitals. Results are displayed in Table 2. There was a significant difference in the mean readmission rates between hospitals with low hospitalist staffing levels (mean readmission rate = 16.06%) versus high staffing levels (mean readmission rate = 15.72%). The mean readmission rate for physician‐owned hospitals was significantly lower than for nonphysician‐owned hospitals (15.46% vs 15.9%). Also, fully integrated hospitals had a lower readmission rate than hospitals where physicians were not fully integrated (15.93% vs 15.86%). Based on the ANOVA results, there was a significant difference between teaching levels. According to a Tukey honest significant difference post hoc test, there was no significant difference between nonteaching and minor teaching hospitals, but the readmission rate was significantly higher in major teaching hospitals (nonteaching = 15.83%, minor teaching = 15.76%, major teaching = 16.9%).

The OLS regression model was significant and explained 16% of the variability in readmission rates (Table 3). Higher hospitalists staffing levels were associated with lower 30‐day all cause readmission rates (P = 0.00). The addition of 1 hospitalist per general and surgical bed was associated with a 0.77 percentage points decrease in adjusted readmission rates. In terms of hospital‐physician arrangements, fully integrated hospitals had adjusted 30‐day all‐cause readmission rates 0.09 percentage points lower than nonfully integrated hospitals (P = 0.08). Physician partial or full ownership was significantly associated with lower readmission rates (P = 0.00); hospitals partially or fully owned by physicians had adjusted readmission rates 0.36 percentage points lower than nonphysician‐owned hospitals.

Based on the regression analysis, major teaching hospitals on average had adjusted readmission rates 1.03 percentage point higher than nonteaching hospitals (P = 0.000), whereas there was no significant difference between minor and nonteaching hospitals (P > 0.1). As the number of registered nurses (RNs) per 100 inpatient days increased by 1, readmission rates dropped by 0.17 (P = 0.00). Hospitals with higher Medicaid shares of admission had significantly higher readmission rates (P < 0.05). Hospitals located in counties with higher unemployment rates also had higher readmission rates (P = 0.000), whereas market competition had no significant association with readmissions. For‐profit hospitals had adjusted readmission rates 0.21 percentage points higher than not‐for‐profit hospitals (P = 0.01). Finally, hospitals that have adopted a medical home model had significantly lower readmission rates (P = 0.02); hospitals with an established medical home model had adjusted readmission rates 0.17 percentage points lower than their counterparts.

DISCUSSION

In the era of VBP and mounting pressures on hospitals to improve quality and lower cost, it is important to understand the association between modifiable hospital characteristics, such as hospitalist staffing levels, and unmodifiable characteristics, such as teaching status and size, with quality of care. There are many factors that can contribute to higher readmission rates. Some of these factors are hospital related and others are patient related, such as the environment in which a patient resides. Benbassat and Taragin argue that 9% to 48% of hospital readmissions are avoidable and are related to factors such as inadequate resolution of the problem the patient was admitted for and poor discharge care.[22] In this article, we have focused on hospital and market factors. Our main variables of interest were hospitalist staffing level, physician full integration, physician ownership, and the adoption of the medical home model at the hospital. Moreover, we examined the association between the hospital environment, specifically, market competition, and the patient environment, specifically, unemployment rates, with readmission rates.

Hospitalists' provision of inpatient care has been on the rise. From 1997 to 2006, the likelihood of receiving inpatient care from a hospitalist grew by 29.2% per year.[23] Based on AHA (2013) data, 65% of hospitals reported that hospitalists provided care at the hospital. The main driver behind the adoption of the hospitalists' model is the positive role hospitalists play in improving hospital efficiency and their familiarity and specialization in hospital care.[24] However, concerns exist that hospitalists might negatively influence patient outcomes given the discontinuity of care that occurs once the patient is discharged from the hospital and back to the care of their primary care physician.[25] Based on our analysis though, higher hospitalist staffing levels were associated with lower readmission rates. Therefore, to better understand the relationship between hospitalists and quality, it is important to account for staffing levels, not merely whether hospitalists provide care at the hospital or not. Higher patient load per hospitalist might still improve hospital efficiency by lowering costs, but is it likely to impede the quality of care provided by hospitalists. This is not surprising given similar findings, including in this article, which document a similar positive relationship between nursing staffing levels and quality.