Brief Reports

Respiratory rate (RR) is an essential vital sign that is routinely measured for hospitalized adults. It is a strong predictor of adverse events.^1,2 Therefore, RR is a key component of several widely used risk prediction scores, including the systemic inflammatory response syndrome (SIRS).³

Despite its clinical utility, RR is inaccurately measured.^4-7 One reason for the inaccurate measurement of RR is that RR measurement, in contrast to that of other vital signs, is not automated. The gold-standard technique for measuring RR is the visual assessment of a resting patient. Thus, RR measurement is perceived as time-consuming. Clinical staff instead frequently approximate RR through brief observation.^8-11

Given its clinical importance and widespread inaccuracy, we conducted a quality improvement (QI) initiative to improve RR accuracy.

Design and Setting

We conducted an interdisciplinary QI initiative by using the plan–do–study–act (PDSA) methodology from July 2017 to February 2018. The initiative was set in a single adult 28-bed medical inpatient unit of a large, urban, safety-net hospital consisting of general internal medicine and hematology/oncology patients. Routine vital sign measurements on this unit occur at four- or six-hour intervals per physician orders and are performed by patient-care assistants (PCAs) who are nonregistered nursing support staff. PCAs use a vital signs cart equipped with automated tools to measure vital signs except for RR, which is manually assessed. PCAs are trained on vital sign measurements during a two-day onboarding orientation and four to six weeks of on-the-job training by experienced PCAs. PCAs are directly supervised by nursing operations managers. Formal continuing education programs for PCAs or performance audits of their clinical duties did not exist prior to our QI initiative.

Intervention

Intervention development addressing several important barriers and workflow inefficiencies was based on the direct observation of PCA workflow and information gathering by engaging stakeholders, including PCAs, nursing operations management, nursing leadership, and hospital administration (PDSA cycles 1-7 in Table). Our modified PCA vital sign workflow incorporated RR measurement during the approximate 30 seconds needed to complete automated blood pressure measurement as previously described.¹² Nursing administration purchased three stopwatches (each $5 US) to attach to vital signs carts. One investigator (NK) participated in two monthly one-hour meetings, and three investigators (NK, KB, and SD) participated in 19 daily 15-minute huddles to conduct stakeholder engagement and educate and retrain PCAs on proper technique (total of 6.75 hours).

Evaluation

The primary aim of this QI initiative was to improve RR accuracy, which was evaluated using two distinct but complementary analyses: the prospective comparison of PCA-recorded RRs with gold-standard recorded RRs and the retrospective comparison of RRs recorded in electronic health records (EHR) on the intervention unit versus two control units. The secondary aims were to examine time to complete vital sign measurement and to assess whether the intervention was associated with a reduction in the incidence of SIRS specifically due to tachypnea.

Respiratory Rate Accuracy

PCA-recorded RRs were considered accurate if the RR was within ±2 breaths of a gold-standard RR measurement performed by a trained study member (NK or KB). We conducted gold-standard RR measurements for 100 observations pre- and postintervention within 30 minutes of PCA measurement to avoid Hawthorne bias.

We assessed the variability of recorded RRs in the EHR for all patients in the intervention unit as a proxy for accuracy. We hypothesized on the basis of prior research that improving the accuracy of RR measurement would increase the variability and normality of distribution in RRs.¹³ This is an approach that we have employed previously.⁷ The EHR cohort included consecutive hospitalizations by patients who were admitted to either the intervention unit or to one of two nonintervention general medicine inpatient units that served as concurrent controls. We grouped hospitalizations into a preintervention phase from March 1, 2017-July 22, 2017, a planning phase from July 23, 2017-December 3, 2017, and a postintervention phase from December 21, 2017-February 28, 2018. Hospitalizations during the two-week teaching phase from December 3, 2017-December 21, 2017 were excluded. We excluded vital signs obtained in the emergency department or in a location different from the patient’s admission unit. We qualitatively assessed RR distribution using histograms as we have done previously.⁷

We examined the distributions of RRs recorded in the EHR before and after intervention by individual PCAs on the intervention floor to assess for fidelity and adherence in the PCA uptake of the intervention.

Time

We compared the time to complete vital sign measurement among convenience samples of 50 unique observations pre- and postintervention using the Wilcoxon rank sum test.

SIRS Incidence

Since we hypothesized that improved RR accuracy would reduce falsely elevated RRs but have no impact on the other three SIRS criteria, we assessed changes in tachypnea-specific SIRS incidence, which was defined a priori as the presence of exactly two concurrent SIRS criteria, one of which was an elevated RR.³ We examined changes using a difference-in-differences approach with three different units of analysis (per vital sign measurement, hospital-day, and hospitalization; see footnote for Appendix Table 1 for methodological details. All analyses were conducted using STATA 12.0 (StataCorp, College Station, Texas).

Respiratory Rate Accuracy

Prior to the intervention, the median PCA RR was 18 (IQR 18-20) versus 12 (IQR 12-18) for the gold-standard RR (Appendix Figure 1), with only 36% of PCA measurements considered accurate. After the intervention, the median PCA-recorded RR was 14 (IQR 15-20) versus 14 (IQR 14-20) for the gold-standard RR and a RR accuracy of 58% (P < .001).

For our analyses on RR distribution using EHR data, we included 143,447 unique RRs (Appendix Table 2). After the intervention, the normality of the distribution of RRs on the intervention unit had increased, whereas those of RRs on the control units remained qualitatively similar pre- and postintervention (Appendix Figure 2).

Notable differences existed among the 11 individual PCAs (Figure) despite observing increased variability in PCA-recorded RRs postintervention. Some PCAs (numbers 2, 7, and 10) shifted their narrow RR interquartile range lower by several breaths/minute, whereas most other PCAs had a reduced median RR and widened interquartile range.

Time

Before the intervention, the median time to complete vital sign measurements was 2:36 (IQR 2:04-3:20). After the intervention, the time to complete vital signs decreased to 1:55 (IQR, 1:40-2:22; P < .001), which was 41 less seconds on average per vital sign set.

SIRS Incidence

The intervention was associated with a 3.3% reduction (95% CI, –6.4% to –0.005%) in tachypnea-specific SIRS incidence per hospital-day and a 7.8% reduction (95% CI, –13.5% to –2.2%) per hospitalization (Appendix Table 1). We also observed a modest reduction in overall SIRS incidence after the intervention (2.9% less per vital sign check, 4.6% less per hospital-day, and 3.2% less per hospitalization), although these reductions were not statistically significant.

Our QI initiative improved the absolute RR accuracy by 22%, saved PCAs 41 seconds on average per vital sign measurement, and decreased the absolute proportion of hospitalizations with tachypnea-specific SIRS by 7.8%. Our intervention is a novel, interdisciplinary, low-cost, low-effort, low-tech approach that addressed known challenges to accurate RR measurement,^8,9,11 as well as the key barriers identified in our initial PDSA cycles. Our approach includes adding a time-keeping device to vital sign carts and standardizing a PCA vital sign workflow with increased efficiency. Lastly, this intervention is potentially scalable because stakeholder engagement, education, and retraining of the entire PCA staff for the unit required only 6.75 hours.

While our primary goal was to improve RR accuracy, our QI initiative also improved vital sign efficiency. By extrapolating our findings to an eight-hour PCA shift caring for eight patients who require vital sign checks every four hours, we estimated that our intervention would save approximately 16:24 minutes per PCA shift. This newfound time could be repurposed for other patient-care tasks or could be spent ensuring the accuracy of other vital signs given that accurate monitoring may be neglected because of time constraints.¹¹ Additionally, the improvement in RR accuracy reduced falsely elevated RRs and thus lowered SIRS incidence specifically due to tachypnea. Given that EHR-based sepsis alerts are often based on SIRS criteria, improved RR accuracy may also improve alarm fatigue by reducing the rate of false-positive alerts.¹⁴

This initiative is not without limitations. Generalizability to other hospitals and even other units within the same hospital is uncertain. However, because this initiative was conducted within a safety-net hospital, we anticipate at least similar, if not increased, success in better-resourced hospitals. Second, the long-term durability of our intervention is unclear, although EHR RR variability remained steady for two months after our intervention (data not shown).

To ensure long-term sustainability and further improve RR accuracy, future PDSA cycles could include electing a PCA “vital signs champion” to reiterate the importance of RRs in clinical decision-making and ensure adherence to the modified workflow. Nursing champions act as persuasive change agents that disseminate and implement healthcare change,¹⁵ which may also be true of PCA champions. Additionally, future PDSA cycles can obviate the need for labor-intensive manual audits by leveraging EHR-based auditing to target education and retraining interventions to PCAs with minimal RR variability to optimize workflow adherence.

In conclusion, through a multipronged QI initiative we improved RR accuracy, increased the efficiency of vital sign measurement, and decreased SIRS incidence specifically due to tachypnea by reducing the number of falsely elevated RRs. This novel, low-cost, low-effort, low-tech approach can readily be implemented and disseminated in hospital inpatient settings.

Acknowledgments

The authors would like to acknowledge the meaningful contributions of Mr. Sudarshaan Pathak, RN, Ms. Shirly Koduvathu, RN, and Ms. Judy Herrington MSN, RN in this multidisciplinary initiative. We thank Mr. Christopher McKintosh, RN for his support in data acquisition. Lastly, the authors would like to acknowledge all of the patient-care assistants involved in this QI initiative.

Disclosures

Dr. Makam reports grants from NIA/NIH, during the conduct of the study. All other authors have nothing to disclose.

Funding

This work is supported in part by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24HS022418). OKN is funded by the National Heart, Lung, and Blood Institute (K23HL133441), and ANM is funded by the National Institute on Aging (K23AG052603).

Respiratory rate (RR) is an essential vital sign that is routinely measured for hospitalized adults. It is a strong predictor of adverse events.^1,2 Therefore, RR is a key component of several widely used risk prediction scores, including the systemic inflammatory response syndrome (SIRS).³

Despite its clinical utility, RR is inaccurately measured.^4-7 One reason for the inaccurate measurement of RR is that RR measurement, in contrast to that of other vital signs, is not automated. The gold-standard technique for measuring RR is the visual assessment of a resting patient. Thus, RR measurement is perceived as time-consuming. Clinical staff instead frequently approximate RR through brief observation.^8-11

Given its clinical importance and widespread inaccuracy, we conducted a quality improvement (QI) initiative to improve RR accuracy.

Design and Setting

We conducted an interdisciplinary QI initiative by using the plan–do–study–act (PDSA) methodology from July 2017 to February 2018. The initiative was set in a single adult 28-bed medical inpatient unit of a large, urban, safety-net hospital consisting of general internal medicine and hematology/oncology patients. Routine vital sign measurements on this unit occur at four- or six-hour intervals per physician orders and are performed by patient-care assistants (PCAs) who are nonregistered nursing support staff. PCAs use a vital signs cart equipped with automated tools to measure vital signs except for RR, which is manually assessed. PCAs are trained on vital sign measurements during a two-day onboarding orientation and four to six weeks of on-the-job training by experienced PCAs. PCAs are directly supervised by nursing operations managers. Formal continuing education programs for PCAs or performance audits of their clinical duties did not exist prior to our QI initiative.

Intervention

Intervention development addressing several important barriers and workflow inefficiencies was based on the direct observation of PCA workflow and information gathering by engaging stakeholders, including PCAs, nursing operations management, nursing leadership, and hospital administration (PDSA cycles 1-7 in Table). Our modified PCA vital sign workflow incorporated RR measurement during the approximate 30 seconds needed to complete automated blood pressure measurement as previously described.¹² Nursing administration purchased three stopwatches (each $5 US) to attach to vital signs carts. One investigator (NK) participated in two monthly one-hour meetings, and three investigators (NK, KB, and SD) participated in 19 daily 15-minute huddles to conduct stakeholder engagement and educate and retrain PCAs on proper technique (total of 6.75 hours).

Evaluation

The primary aim of this QI initiative was to improve RR accuracy, which was evaluated using two distinct but complementary analyses: the prospective comparison of PCA-recorded RRs with gold-standard recorded RRs and the retrospective comparison of RRs recorded in electronic health records (EHR) on the intervention unit versus two control units. The secondary aims were to examine time to complete vital sign measurement and to assess whether the intervention was associated with a reduction in the incidence of SIRS specifically due to tachypnea.

Respiratory Rate Accuracy

PCA-recorded RRs were considered accurate if the RR was within ±2 breaths of a gold-standard RR measurement performed by a trained study member (NK or KB). We conducted gold-standard RR measurements for 100 observations pre- and postintervention within 30 minutes of PCA measurement to avoid Hawthorne bias.

We assessed the variability of recorded RRs in the EHR for all patients in the intervention unit as a proxy for accuracy. We hypothesized on the basis of prior research that improving the accuracy of RR measurement would increase the variability and normality of distribution in RRs.¹³ This is an approach that we have employed previously.⁷ The EHR cohort included consecutive hospitalizations by patients who were admitted to either the intervention unit or to one of two nonintervention general medicine inpatient units that served as concurrent controls. We grouped hospitalizations into a preintervention phase from March 1, 2017-July 22, 2017, a planning phase from July 23, 2017-December 3, 2017, and a postintervention phase from December 21, 2017-February 28, 2018. Hospitalizations during the two-week teaching phase from December 3, 2017-December 21, 2017 were excluded. We excluded vital signs obtained in the emergency department or in a location different from the patient’s admission unit. We qualitatively assessed RR distribution using histograms as we have done previously.⁷

We examined the distributions of RRs recorded in the EHR before and after intervention by individual PCAs on the intervention floor to assess for fidelity and adherence in the PCA uptake of the intervention.

Time

We compared the time to complete vital sign measurement among convenience samples of 50 unique observations pre- and postintervention using the Wilcoxon rank sum test.

SIRS Incidence

Since we hypothesized that improved RR accuracy would reduce falsely elevated RRs but have no impact on the other three SIRS criteria, we assessed changes in tachypnea-specific SIRS incidence, which was defined a priori as the presence of exactly two concurrent SIRS criteria, one of which was an elevated RR.³ We examined changes using a difference-in-differences approach with three different units of analysis (per vital sign measurement, hospital-day, and hospitalization; see footnote for Appendix Table 1 for methodological details. All analyses were conducted using STATA 12.0 (StataCorp, College Station, Texas).

Respiratory Rate Accuracy

Prior to the intervention, the median PCA RR was 18 (IQR 18-20) versus 12 (IQR 12-18) for the gold-standard RR (Appendix Figure 1), with only 36% of PCA measurements considered accurate. After the intervention, the median PCA-recorded RR was 14 (IQR 15-20) versus 14 (IQR 14-20) for the gold-standard RR and a RR accuracy of 58% (P < .001).

For our analyses on RR distribution using EHR data, we included 143,447 unique RRs (Appendix Table 2). After the intervention, the normality of the distribution of RRs on the intervention unit had increased, whereas those of RRs on the control units remained qualitatively similar pre- and postintervention (Appendix Figure 2).

Notable differences existed among the 11 individual PCAs (Figure) despite observing increased variability in PCA-recorded RRs postintervention. Some PCAs (numbers 2, 7, and 10) shifted their narrow RR interquartile range lower by several breaths/minute, whereas most other PCAs had a reduced median RR and widened interquartile range.

Time

Before the intervention, the median time to complete vital sign measurements was 2:36 (IQR 2:04-3:20). After the intervention, the time to complete vital signs decreased to 1:55 (IQR, 1:40-2:22; P < .001), which was 41 less seconds on average per vital sign set.

SIRS Incidence

The intervention was associated with a 3.3% reduction (95% CI, –6.4% to –0.005%) in tachypnea-specific SIRS incidence per hospital-day and a 7.8% reduction (95% CI, –13.5% to –2.2%) per hospitalization (Appendix Table 1). We also observed a modest reduction in overall SIRS incidence after the intervention (2.9% less per vital sign check, 4.6% less per hospital-day, and 3.2% less per hospitalization), although these reductions were not statistically significant.

Our QI initiative improved the absolute RR accuracy by 22%, saved PCAs 41 seconds on average per vital sign measurement, and decreased the absolute proportion of hospitalizations with tachypnea-specific SIRS by 7.8%. Our intervention is a novel, interdisciplinary, low-cost, low-effort, low-tech approach that addressed known challenges to accurate RR measurement,^8,9,11 as well as the key barriers identified in our initial PDSA cycles. Our approach includes adding a time-keeping device to vital sign carts and standardizing a PCA vital sign workflow with increased efficiency. Lastly, this intervention is potentially scalable because stakeholder engagement, education, and retraining of the entire PCA staff for the unit required only 6.75 hours.

While our primary goal was to improve RR accuracy, our QI initiative also improved vital sign efficiency. By extrapolating our findings to an eight-hour PCA shift caring for eight patients who require vital sign checks every four hours, we estimated that our intervention would save approximately 16:24 minutes per PCA shift. This newfound time could be repurposed for other patient-care tasks or could be spent ensuring the accuracy of other vital signs given that accurate monitoring may be neglected because of time constraints.¹¹ Additionally, the improvement in RR accuracy reduced falsely elevated RRs and thus lowered SIRS incidence specifically due to tachypnea. Given that EHR-based sepsis alerts are often based on SIRS criteria, improved RR accuracy may also improve alarm fatigue by reducing the rate of false-positive alerts.¹⁴

This initiative is not without limitations. Generalizability to other hospitals and even other units within the same hospital is uncertain. However, because this initiative was conducted within a safety-net hospital, we anticipate at least similar, if not increased, success in better-resourced hospitals. Second, the long-term durability of our intervention is unclear, although EHR RR variability remained steady for two months after our intervention (data not shown).

To ensure long-term sustainability and further improve RR accuracy, future PDSA cycles could include electing a PCA “vital signs champion” to reiterate the importance of RRs in clinical decision-making and ensure adherence to the modified workflow. Nursing champions act as persuasive change agents that disseminate and implement healthcare change,¹⁵ which may also be true of PCA champions. Additionally, future PDSA cycles can obviate the need for labor-intensive manual audits by leveraging EHR-based auditing to target education and retraining interventions to PCAs with minimal RR variability to optimize workflow adherence.

In conclusion, through a multipronged QI initiative we improved RR accuracy, increased the efficiency of vital sign measurement, and decreased SIRS incidence specifically due to tachypnea by reducing the number of falsely elevated RRs. This novel, low-cost, low-effort, low-tech approach can readily be implemented and disseminated in hospital inpatient settings.

Acknowledgments

The authors would like to acknowledge the meaningful contributions of Mr. Sudarshaan Pathak, RN, Ms. Shirly Koduvathu, RN, and Ms. Judy Herrington MSN, RN in this multidisciplinary initiative. We thank Mr. Christopher McKintosh, RN for his support in data acquisition. Lastly, the authors would like to acknowledge all of the patient-care assistants involved in this QI initiative.

Disclosures

Dr. Makam reports grants from NIA/NIH, during the conduct of the study. All other authors have nothing to disclose.

Funding

This work is supported in part by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24HS022418). OKN is funded by the National Heart, Lung, and Blood Institute (K23HL133441), and ANM is funded by the National Institute on Aging (K23AG052603).

Respiratory rate (RR) is an essential vital sign that is routinely measured for hospitalized adults. It is a strong predictor of adverse events.^1,2 Therefore, RR is a key component of several widely used risk prediction scores, including the systemic inflammatory response syndrome (SIRS).³

Despite its clinical utility, RR is inaccurately measured.^4-7 One reason for the inaccurate measurement of RR is that RR measurement, in contrast to that of other vital signs, is not automated. The gold-standard technique for measuring RR is the visual assessment of a resting patient. Thus, RR measurement is perceived as time-consuming. Clinical staff instead frequently approximate RR through brief observation.^8-11

Given its clinical importance and widespread inaccuracy, we conducted a quality improvement (QI) initiative to improve RR accuracy.

Design and Setting

We conducted an interdisciplinary QI initiative by using the plan–do–study–act (PDSA) methodology from July 2017 to February 2018. The initiative was set in a single adult 28-bed medical inpatient unit of a large, urban, safety-net hospital consisting of general internal medicine and hematology/oncology patients. Routine vital sign measurements on this unit occur at four- or six-hour intervals per physician orders and are performed by patient-care assistants (PCAs) who are nonregistered nursing support staff. PCAs use a vital signs cart equipped with automated tools to measure vital signs except for RR, which is manually assessed. PCAs are trained on vital sign measurements during a two-day onboarding orientation and four to six weeks of on-the-job training by experienced PCAs. PCAs are directly supervised by nursing operations managers. Formal continuing education programs for PCAs or performance audits of their clinical duties did not exist prior to our QI initiative.

Intervention

Intervention development addressing several important barriers and workflow inefficiencies was based on the direct observation of PCA workflow and information gathering by engaging stakeholders, including PCAs, nursing operations management, nursing leadership, and hospital administration (PDSA cycles 1-7 in Table). Our modified PCA vital sign workflow incorporated RR measurement during the approximate 30 seconds needed to complete automated blood pressure measurement as previously described.¹² Nursing administration purchased three stopwatches (each $5 US) to attach to vital signs carts. One investigator (NK) participated in two monthly one-hour meetings, and three investigators (NK, KB, and SD) participated in 19 daily 15-minute huddles to conduct stakeholder engagement and educate and retrain PCAs on proper technique (total of 6.75 hours).

Evaluation

The primary aim of this QI initiative was to improve RR accuracy, which was evaluated using two distinct but complementary analyses: the prospective comparison of PCA-recorded RRs with gold-standard recorded RRs and the retrospective comparison of RRs recorded in electronic health records (EHR) on the intervention unit versus two control units. The secondary aims were to examine time to complete vital sign measurement and to assess whether the intervention was associated with a reduction in the incidence of SIRS specifically due to tachypnea.

Respiratory Rate Accuracy

PCA-recorded RRs were considered accurate if the RR was within ±2 breaths of a gold-standard RR measurement performed by a trained study member (NK or KB). We conducted gold-standard RR measurements for 100 observations pre- and postintervention within 30 minutes of PCA measurement to avoid Hawthorne bias.

We assessed the variability of recorded RRs in the EHR for all patients in the intervention unit as a proxy for accuracy. We hypothesized on the basis of prior research that improving the accuracy of RR measurement would increase the variability and normality of distribution in RRs.¹³ This is an approach that we have employed previously.⁷ The EHR cohort included consecutive hospitalizations by patients who were admitted to either the intervention unit or to one of two nonintervention general medicine inpatient units that served as concurrent controls. We grouped hospitalizations into a preintervention phase from March 1, 2017-July 22, 2017, a planning phase from July 23, 2017-December 3, 2017, and a postintervention phase from December 21, 2017-February 28, 2018. Hospitalizations during the two-week teaching phase from December 3, 2017-December 21, 2017 were excluded. We excluded vital signs obtained in the emergency department or in a location different from the patient’s admission unit. We qualitatively assessed RR distribution using histograms as we have done previously.⁷

We examined the distributions of RRs recorded in the EHR before and after intervention by individual PCAs on the intervention floor to assess for fidelity and adherence in the PCA uptake of the intervention.

Time

We compared the time to complete vital sign measurement among convenience samples of 50 unique observations pre- and postintervention using the Wilcoxon rank sum test.

SIRS Incidence

Since we hypothesized that improved RR accuracy would reduce falsely elevated RRs but have no impact on the other three SIRS criteria, we assessed changes in tachypnea-specific SIRS incidence, which was defined a priori as the presence of exactly two concurrent SIRS criteria, one of which was an elevated RR.³ We examined changes using a difference-in-differences approach with three different units of analysis (per vital sign measurement, hospital-day, and hospitalization; see footnote for Appendix Table 1 for methodological details. All analyses were conducted using STATA 12.0 (StataCorp, College Station, Texas).

Respiratory Rate Accuracy

Prior to the intervention, the median PCA RR was 18 (IQR 18-20) versus 12 (IQR 12-18) for the gold-standard RR (Appendix Figure 1), with only 36% of PCA measurements considered accurate. After the intervention, the median PCA-recorded RR was 14 (IQR 15-20) versus 14 (IQR 14-20) for the gold-standard RR and a RR accuracy of 58% (P < .001).

For our analyses on RR distribution using EHR data, we included 143,447 unique RRs (Appendix Table 2). After the intervention, the normality of the distribution of RRs on the intervention unit had increased, whereas those of RRs on the control units remained qualitatively similar pre- and postintervention (Appendix Figure 2).

Notable differences existed among the 11 individual PCAs (Figure) despite observing increased variability in PCA-recorded RRs postintervention. Some PCAs (numbers 2, 7, and 10) shifted their narrow RR interquartile range lower by several breaths/minute, whereas most other PCAs had a reduced median RR and widened interquartile range.

Time

Before the intervention, the median time to complete vital sign measurements was 2:36 (IQR 2:04-3:20). After the intervention, the time to complete vital signs decreased to 1:55 (IQR, 1:40-2:22; P < .001), which was 41 less seconds on average per vital sign set.

SIRS Incidence

The intervention was associated with a 3.3% reduction (95% CI, –6.4% to –0.005%) in tachypnea-specific SIRS incidence per hospital-day and a 7.8% reduction (95% CI, –13.5% to –2.2%) per hospitalization (Appendix Table 1). We also observed a modest reduction in overall SIRS incidence after the intervention (2.9% less per vital sign check, 4.6% less per hospital-day, and 3.2% less per hospitalization), although these reductions were not statistically significant.

Our QI initiative improved the absolute RR accuracy by 22%, saved PCAs 41 seconds on average per vital sign measurement, and decreased the absolute proportion of hospitalizations with tachypnea-specific SIRS by 7.8%. Our intervention is a novel, interdisciplinary, low-cost, low-effort, low-tech approach that addressed known challenges to accurate RR measurement,^8,9,11 as well as the key barriers identified in our initial PDSA cycles. Our approach includes adding a time-keeping device to vital sign carts and standardizing a PCA vital sign workflow with increased efficiency. Lastly, this intervention is potentially scalable because stakeholder engagement, education, and retraining of the entire PCA staff for the unit required only 6.75 hours.

While our primary goal was to improve RR accuracy, our QI initiative also improved vital sign efficiency. By extrapolating our findings to an eight-hour PCA shift caring for eight patients who require vital sign checks every four hours, we estimated that our intervention would save approximately 16:24 minutes per PCA shift. This newfound time could be repurposed for other patient-care tasks or could be spent ensuring the accuracy of other vital signs given that accurate monitoring may be neglected because of time constraints.¹¹ Additionally, the improvement in RR accuracy reduced falsely elevated RRs and thus lowered SIRS incidence specifically due to tachypnea. Given that EHR-based sepsis alerts are often based on SIRS criteria, improved RR accuracy may also improve alarm fatigue by reducing the rate of false-positive alerts.¹⁴

This initiative is not without limitations. Generalizability to other hospitals and even other units within the same hospital is uncertain. However, because this initiative was conducted within a safety-net hospital, we anticipate at least similar, if not increased, success in better-resourced hospitals. Second, the long-term durability of our intervention is unclear, although EHR RR variability remained steady for two months after our intervention (data not shown).

To ensure long-term sustainability and further improve RR accuracy, future PDSA cycles could include electing a PCA “vital signs champion” to reiterate the importance of RRs in clinical decision-making and ensure adherence to the modified workflow. Nursing champions act as persuasive change agents that disseminate and implement healthcare change,¹⁵ which may also be true of PCA champions. Additionally, future PDSA cycles can obviate the need for labor-intensive manual audits by leveraging EHR-based auditing to target education and retraining interventions to PCAs with minimal RR variability to optimize workflow adherence.

In conclusion, through a multipronged QI initiative we improved RR accuracy, increased the efficiency of vital sign measurement, and decreased SIRS incidence specifically due to tachypnea by reducing the number of falsely elevated RRs. This novel, low-cost, low-effort, low-tech approach can readily be implemented and disseminated in hospital inpatient settings.

Acknowledgments

The authors would like to acknowledge the meaningful contributions of Mr. Sudarshaan Pathak, RN, Ms. Shirly Koduvathu, RN, and Ms. Judy Herrington MSN, RN in this multidisciplinary initiative. We thank Mr. Christopher McKintosh, RN for his support in data acquisition. Lastly, the authors would like to acknowledge all of the patient-care assistants involved in this QI initiative.

Disclosures

Dr. Makam reports grants from NIA/NIH, during the conduct of the study. All other authors have nothing to disclose.

Funding

This work is supported in part by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24HS022418). OKN is funded by the National Heart, Lung, and Blood Institute (K23HL133441), and ANM is funded by the National Institute on Aging (K23AG052603).

Evidence exists that physicians consider what may be called “social” or “nonmedical” factors (lack of social support or barriers to access) in hospital admission decision-making and that patients are hospitalized even in the absence of a level of medical acuity warranting admission.^1-3 Although hospitalization is associated with the risk of adverse events (AEs),⁴ whether this risk is related to the medical acuity of admission remains unclear. Our study sought to quantify the AEs experienced by patients hospitalized without definite medical acuity compared with those experienced by patients hospitalized with a definite medically appropriate indication for admission.

Setting and Database Used for Analysis

This study was conducted at an urban, safety-net, public teaching hospital. At our site, calls for medical admissions are always answered by a hospital medicine attending physician (“triage physician”) who works collaboratively with the referring physician to facilitate appropriate disposition. Many of these discussions occur via telephone, but the triage physician may also assess the patient directly if needed. This study involved 24 triage physicians who directly assessed the patient in 65% of the cases.

At the time of each admission call, the triage physician logs the following information into a central triage database: date and time of call, patient location, reason for admission, assessment of appropriateness for medical floor, contributing factors to admission decision-making, and patient disposition.

Admission Appropriateness Group Designation

To be considered for inclusion in this study, calls must have originated from the emergency department and resulted in admission to the general medicine floor on either a resident teaching or hospitalist service from February 1, 2018 to June 1, 2018. This time frame was selected to avoid the start of a new academic cycle in late June that may confound AE rates.

The designation of appropriateness was determined by the triage physician’s logged response to triage database questions at the time of the admission call. Of the 748 admissions meeting inclusion criteria, 513 (68.6%) were considered definitely appropriate on the basis of the triage physician’s response to the question “Based ONLY on the medical reason for hospitalization, in your opinion, how appropriate is this admission to the medicine floor service?” Furthermore, 169 (22.6%) were considered without definite medical acuity on the basis of the triage physician’s indication that “severity of medical problems alone may not require inpatient hospitalization” (Appendix Figure 1).

Study Design

Following a retrospective cohort study design, we systematically sampled 150 admissions from those “admitted without definite medical acuity” to create the exposure group and 150 from the “definitely medically appropriate” admissions to create the nonexposure group. Our sampling method involved selecting every third record until reaching the target sample size. This method and group sizes were determined prior to beginning data collection. Given the expected incidence of 33% AEs in the unexposed group (consistent with previous reports of AEs using the trigger tool⁵), we anticipated that a total sample size of 300 would be appropriate to capture a relative risk of at least 1.5 with 80% power and 95% confidence level.

Chart review was performed to capture patient demographics, admission characteristics, and hospitalization outcomes. We captured emergency severity index (ESI)⁶, a validated, reliable triage assessment score assigned by our emergency department, as a measurement of acute illness and calculated the Charlson comorbidity index (CCI)⁷ as a measurement of chronic comorbidity.

Identification of Adverse Events

We measured AEs by using the Institute for Healthcare Improvement Global Trigger Tool,^8,9 which is estimated to identify up to 10 times more AEs than other methods, such as voluntary reporting.⁵ This protocol includes 28 triggers in the Cares and Medication Modules that serve as indicators that an AE may have occurred. The presence of a trigger is not necessarily an AE but a clue for further analysis. Two investigators (AS and CS) independently systematically searched for the presence of triggers within each patient chart. Trigger identification prompted in-depth analysis to confirm the occurrence of an AE and to characterize its severity by using the National Coordinating Council for Medication Error Reporting and Prevention categorization.¹⁰ An AE was coded when independent reviewers identified evidence of a preventable or nonpreventable “noxious and unintended event occurring in association with medical care.”⁹ By definition, any AEs identified were patient harms. Findings were reviewed weekly to ensure agreement, and discrepancies were adjudicated by a third investigator (MB).

All study data were collected by using REDCap electronic data capture tools hosted at the University of Washington.¹¹ The University of Washington Institutional Review Board granted approval for this study.

Study Outcome and Statistical Analysis

The primary outcome was AEs per group with results calculated in three ways: AEs per 1,000 patient-days, AEs per 100 admissions, and percent of admissions with an AE. The risk ratio (RR) for the percent of admissions with an AE and the incidence rate ratio (IRR) for AEs per 1,000 patient-days were calculated for the comparison of significance.

Other data were analyzed by using Pearson’s chi square for categorical data, Student t test for normally distributed quantitative data, and Wilcoxon rank-sum (Mann–Whitney) for the length of stay (due to skew). Analyses were conducted using STATA (version 15.1, College Station, TX).

This work follows standards for reporting observational students as outlined in the STROBE statement.¹²

Patient Demographics

Both groups were predominantly white/non-Hispanic, male, and English-speaking (Table 1). More patients without definite medical acuity were covered by public insurance (78.9% vs 69.8%, P = .010) and discharged to homelessness (34.8% vs 22.6%, P = .041).

Measures of Illness

Patients considered definitely medically appropriate had lower ESI scores, indicative of more acute presentation, than those without definite medical acuity (2.73 [95% CI 2.64-2.81] vs 2.87 [95% CI 2.78-2.95], P = .026). There was no difference in CCI scores (Table 1).

Reason for Admission and Outcomes

Admissions considered definitely medically appropriate more frequently had an identified diagnosis/syndrome (66% vs 53%) or objective measurement (8.7% vs 2.7%) listed as the reason for admission, whereas patients admitted without definite medical acuity more freuqently had undifferentiated symptoms (34.7% vs 24%) or other/disposition (6% vs 1.3%) listed. The most common factors that triage physicians cited as contributing to the decision to admit patients without definite medical acuity included homelessness (34%), lack of outpatient social support (32%), and substance use disorder (25%). More details are available in Appendix Tables 1 and 2.

Admissions without definite medical acuity were longer than those with definite medical acuity (6.6 vs 6.0 days, P = .038), but there was no difference in emergency department readmissions within 48 hours or hospital readmissions within 30 days (Table 1).

Adverse Events

We identified 76 AEs in 41 admissions without definite medical acuity (range 0-10 AEs per admission) and 63 AEs in 44 definitely medically appropriate admissions (range 0-4 AEs per admission). The percentage of admissions with AE (27.3% vs 29.3%; RR 0.93, 95% CI 0.65-1.34, P = .70) and the rate of AE/1,000 patient-days (76.8 vs 70.4; IRR 1.09, 95% CI 0.77-1.55, P = .61) did not show statistically significant differences. The distribution of AE severity was similar between the two groups (Table 2). Most identified AEs caused temporary harm to the patient and were rated at severity levels E or F. Severe AEs, including at least one level I (patient death), occurred in both groups. The complete listing of positive triggers leading to adverse event identification by group and severity is available in Appendix Table 3.

By using a robust, standardized method, we found that patients admitted without definite medical acuity experienced the same number of inpatient AEs as patients admitted for definitely medically appropriate reasons. While the groups were relatively similar overall in terms of demographics and chronic comorbidity, we found evidence of social vulnerability in the group admitted without definite medical acuity in the form of increased rates of homelessness, triage physician concern regarding the lack of outpatient social support, and disposition-related reasons for admission. That both groups suffered harm―including patient death―while admitted to the hospital is striking, in particular for those patients who were admitted because of the lack of suitable outpatient options.

The potential limitations to the generalizability of this work include the single-site, safety-net setting and the use of individual physician determination of admission appropriateness. The proportion of admissions without definite medical acuity reported here is similar to that reported by previously published admission decision-making studies,^2,3 and the rate of AEs observed is similar to rates measured in other studies using the trigger tool methodology.^5,13 These similarities suggest some commonality across settings. Our study treats triage physician assessment as the marker of difference in defining the two groups and is an inherently subjective assessment that is reflective of real-world, holistic decision-making. Notably, the triage physician assessment was corroborated by corresponding differences in the ESI score, an acute triage assessment completed by a clinician outside of our team.

This study adds foundational knowledge to the risk/benefit discussion surrounding the decision to admit. Physician admission decisions are likely influenced by concern for the safety of vulnerable patients. Our results suggest that considering the risk of hospitalization itself in this decision-making remains important.

Evidence exists that physicians consider what may be called “social” or “nonmedical” factors (lack of social support or barriers to access) in hospital admission decision-making and that patients are hospitalized even in the absence of a level of medical acuity warranting admission.^1-3 Although hospitalization is associated with the risk of adverse events (AEs),⁴ whether this risk is related to the medical acuity of admission remains unclear. Our study sought to quantify the AEs experienced by patients hospitalized without definite medical acuity compared with those experienced by patients hospitalized with a definite medically appropriate indication for admission.

Setting and Database Used for Analysis

This study was conducted at an urban, safety-net, public teaching hospital. At our site, calls for medical admissions are always answered by a hospital medicine attending physician (“triage physician”) who works collaboratively with the referring physician to facilitate appropriate disposition. Many of these discussions occur via telephone, but the triage physician may also assess the patient directly if needed. This study involved 24 triage physicians who directly assessed the patient in 65% of the cases.

At the time of each admission call, the triage physician logs the following information into a central triage database: date and time of call, patient location, reason for admission, assessment of appropriateness for medical floor, contributing factors to admission decision-making, and patient disposition.

Admission Appropriateness Group Designation

To be considered for inclusion in this study, calls must have originated from the emergency department and resulted in admission to the general medicine floor on either a resident teaching or hospitalist service from February 1, 2018 to June 1, 2018. This time frame was selected to avoid the start of a new academic cycle in late June that may confound AE rates.

The designation of appropriateness was determined by the triage physician’s logged response to triage database questions at the time of the admission call. Of the 748 admissions meeting inclusion criteria, 513 (68.6%) were considered definitely appropriate on the basis of the triage physician’s response to the question “Based ONLY on the medical reason for hospitalization, in your opinion, how appropriate is this admission to the medicine floor service?” Furthermore, 169 (22.6%) were considered without definite medical acuity on the basis of the triage physician’s indication that “severity of medical problems alone may not require inpatient hospitalization” (Appendix Figure 1).

Study Design

Following a retrospective cohort study design, we systematically sampled 150 admissions from those “admitted without definite medical acuity” to create the exposure group and 150 from the “definitely medically appropriate” admissions to create the nonexposure group. Our sampling method involved selecting every third record until reaching the target sample size. This method and group sizes were determined prior to beginning data collection. Given the expected incidence of 33% AEs in the unexposed group (consistent with previous reports of AEs using the trigger tool⁵), we anticipated that a total sample size of 300 would be appropriate to capture a relative risk of at least 1.5 with 80% power and 95% confidence level.

Chart review was performed to capture patient demographics, admission characteristics, and hospitalization outcomes. We captured emergency severity index (ESI)⁶, a validated, reliable triage assessment score assigned by our emergency department, as a measurement of acute illness and calculated the Charlson comorbidity index (CCI)⁷ as a measurement of chronic comorbidity.

Identification of Adverse Events

We measured AEs by using the Institute for Healthcare Improvement Global Trigger Tool,^8,9 which is estimated to identify up to 10 times more AEs than other methods, such as voluntary reporting.⁵ This protocol includes 28 triggers in the Cares and Medication Modules that serve as indicators that an AE may have occurred. The presence of a trigger is not necessarily an AE but a clue for further analysis. Two investigators (AS and CS) independently systematically searched for the presence of triggers within each patient chart. Trigger identification prompted in-depth analysis to confirm the occurrence of an AE and to characterize its severity by using the National Coordinating Council for Medication Error Reporting and Prevention categorization.¹⁰ An AE was coded when independent reviewers identified evidence of a preventable or nonpreventable “noxious and unintended event occurring in association with medical care.”⁹ By definition, any AEs identified were patient harms. Findings were reviewed weekly to ensure agreement, and discrepancies were adjudicated by a third investigator (MB).

All study data were collected by using REDCap electronic data capture tools hosted at the University of Washington.¹¹ The University of Washington Institutional Review Board granted approval for this study.

Study Outcome and Statistical Analysis

The primary outcome was AEs per group with results calculated in three ways: AEs per 1,000 patient-days, AEs per 100 admissions, and percent of admissions with an AE. The risk ratio (RR) for the percent of admissions with an AE and the incidence rate ratio (IRR) for AEs per 1,000 patient-days were calculated for the comparison of significance.

Other data were analyzed by using Pearson’s chi square for categorical data, Student t test for normally distributed quantitative data, and Wilcoxon rank-sum (Mann–Whitney) for the length of stay (due to skew). Analyses were conducted using STATA (version 15.1, College Station, TX).

This work follows standards for reporting observational students as outlined in the STROBE statement.¹²

Patient Demographics

Both groups were predominantly white/non-Hispanic, male, and English-speaking (Table 1). More patients without definite medical acuity were covered by public insurance (78.9% vs 69.8%, P = .010) and discharged to homelessness (34.8% vs 22.6%, P = .041).

Measures of Illness

Patients considered definitely medically appropriate had lower ESI scores, indicative of more acute presentation, than those without definite medical acuity (2.73 [95% CI 2.64-2.81] vs 2.87 [95% CI 2.78-2.95], P = .026). There was no difference in CCI scores (Table 1).

Reason for Admission and Outcomes

Admissions considered definitely medically appropriate more frequently had an identified diagnosis/syndrome (66% vs 53%) or objective measurement (8.7% vs 2.7%) listed as the reason for admission, whereas patients admitted without definite medical acuity more freuqently had undifferentiated symptoms (34.7% vs 24%) or other/disposition (6% vs 1.3%) listed. The most common factors that triage physicians cited as contributing to the decision to admit patients without definite medical acuity included homelessness (34%), lack of outpatient social support (32%), and substance use disorder (25%). More details are available in Appendix Tables 1 and 2.

Admissions without definite medical acuity were longer than those with definite medical acuity (6.6 vs 6.0 days, P = .038), but there was no difference in emergency department readmissions within 48 hours or hospital readmissions within 30 days (Table 1).

Adverse Events

We identified 76 AEs in 41 admissions without definite medical acuity (range 0-10 AEs per admission) and 63 AEs in 44 definitely medically appropriate admissions (range 0-4 AEs per admission). The percentage of admissions with AE (27.3% vs 29.3%; RR 0.93, 95% CI 0.65-1.34, P = .70) and the rate of AE/1,000 patient-days (76.8 vs 70.4; IRR 1.09, 95% CI 0.77-1.55, P = .61) did not show statistically significant differences. The distribution of AE severity was similar between the two groups (Table 2). Most identified AEs caused temporary harm to the patient and were rated at severity levels E or F. Severe AEs, including at least one level I (patient death), occurred in both groups. The complete listing of positive triggers leading to adverse event identification by group and severity is available in Appendix Table 3.

By using a robust, standardized method, we found that patients admitted without definite medical acuity experienced the same number of inpatient AEs as patients admitted for definitely medically appropriate reasons. While the groups were relatively similar overall in terms of demographics and chronic comorbidity, we found evidence of social vulnerability in the group admitted without definite medical acuity in the form of increased rates of homelessness, triage physician concern regarding the lack of outpatient social support, and disposition-related reasons for admission. That both groups suffered harm―including patient death―while admitted to the hospital is striking, in particular for those patients who were admitted because of the lack of suitable outpatient options.

The potential limitations to the generalizability of this work include the single-site, safety-net setting and the use of individual physician determination of admission appropriateness. The proportion of admissions without definite medical acuity reported here is similar to that reported by previously published admission decision-making studies,^2,3 and the rate of AEs observed is similar to rates measured in other studies using the trigger tool methodology.^5,13 These similarities suggest some commonality across settings. Our study treats triage physician assessment as the marker of difference in defining the two groups and is an inherently subjective assessment that is reflective of real-world, holistic decision-making. Notably, the triage physician assessment was corroborated by corresponding differences in the ESI score, an acute triage assessment completed by a clinician outside of our team.

This study adds foundational knowledge to the risk/benefit discussion surrounding the decision to admit. Physician admission decisions are likely influenced by concern for the safety of vulnerable patients. Our results suggest that considering the risk of hospitalization itself in this decision-making remains important.

Evidence exists that physicians consider what may be called “social” or “nonmedical” factors (lack of social support or barriers to access) in hospital admission decision-making and that patients are hospitalized even in the absence of a level of medical acuity warranting admission.^1-3 Although hospitalization is associated with the risk of adverse events (AEs),⁴ whether this risk is related to the medical acuity of admission remains unclear. Our study sought to quantify the AEs experienced by patients hospitalized without definite medical acuity compared with those experienced by patients hospitalized with a definite medically appropriate indication for admission.

Setting and Database Used for Analysis

This study was conducted at an urban, safety-net, public teaching hospital. At our site, calls for medical admissions are always answered by a hospital medicine attending physician (“triage physician”) who works collaboratively with the referring physician to facilitate appropriate disposition. Many of these discussions occur via telephone, but the triage physician may also assess the patient directly if needed. This study involved 24 triage physicians who directly assessed the patient in 65% of the cases.

At the time of each admission call, the triage physician logs the following information into a central triage database: date and time of call, patient location, reason for admission, assessment of appropriateness for medical floor, contributing factors to admission decision-making, and patient disposition.

Admission Appropriateness Group Designation

To be considered for inclusion in this study, calls must have originated from the emergency department and resulted in admission to the general medicine floor on either a resident teaching or hospitalist service from February 1, 2018 to June 1, 2018. This time frame was selected to avoid the start of a new academic cycle in late June that may confound AE rates.

The designation of appropriateness was determined by the triage physician’s logged response to triage database questions at the time of the admission call. Of the 748 admissions meeting inclusion criteria, 513 (68.6%) were considered definitely appropriate on the basis of the triage physician’s response to the question “Based ONLY on the medical reason for hospitalization, in your opinion, how appropriate is this admission to the medicine floor service?” Furthermore, 169 (22.6%) were considered without definite medical acuity on the basis of the triage physician’s indication that “severity of medical problems alone may not require inpatient hospitalization” (Appendix Figure 1).

Study Design

Following a retrospective cohort study design, we systematically sampled 150 admissions from those “admitted without definite medical acuity” to create the exposure group and 150 from the “definitely medically appropriate” admissions to create the nonexposure group. Our sampling method involved selecting every third record until reaching the target sample size. This method and group sizes were determined prior to beginning data collection. Given the expected incidence of 33% AEs in the unexposed group (consistent with previous reports of AEs using the trigger tool⁵), we anticipated that a total sample size of 300 would be appropriate to capture a relative risk of at least 1.5 with 80% power and 95% confidence level.

Chart review was performed to capture patient demographics, admission characteristics, and hospitalization outcomes. We captured emergency severity index (ESI)⁶, a validated, reliable triage assessment score assigned by our emergency department, as a measurement of acute illness and calculated the Charlson comorbidity index (CCI)⁷ as a measurement of chronic comorbidity.

Identification of Adverse Events

We measured AEs by using the Institute for Healthcare Improvement Global Trigger Tool,^8,9 which is estimated to identify up to 10 times more AEs than other methods, such as voluntary reporting.⁵ This protocol includes 28 triggers in the Cares and Medication Modules that serve as indicators that an AE may have occurred. The presence of a trigger is not necessarily an AE but a clue for further analysis. Two investigators (AS and CS) independently systematically searched for the presence of triggers within each patient chart. Trigger identification prompted in-depth analysis to confirm the occurrence of an AE and to characterize its severity by using the National Coordinating Council for Medication Error Reporting and Prevention categorization.¹⁰ An AE was coded when independent reviewers identified evidence of a preventable or nonpreventable “noxious and unintended event occurring in association with medical care.”⁹ By definition, any AEs identified were patient harms. Findings were reviewed weekly to ensure agreement, and discrepancies were adjudicated by a third investigator (MB).

All study data were collected by using REDCap electronic data capture tools hosted at the University of Washington.¹¹ The University of Washington Institutional Review Board granted approval for this study.

Study Outcome and Statistical Analysis

The primary outcome was AEs per group with results calculated in three ways: AEs per 1,000 patient-days, AEs per 100 admissions, and percent of admissions with an AE. The risk ratio (RR) for the percent of admissions with an AE and the incidence rate ratio (IRR) for AEs per 1,000 patient-days were calculated for the comparison of significance.

Other data were analyzed by using Pearson’s chi square for categorical data, Student t test for normally distributed quantitative data, and Wilcoxon rank-sum (Mann–Whitney) for the length of stay (due to skew). Analyses were conducted using STATA (version 15.1, College Station, TX).

This work follows standards for reporting observational students as outlined in the STROBE statement.¹²

Patient Demographics

Both groups were predominantly white/non-Hispanic, male, and English-speaking (Table 1). More patients without definite medical acuity were covered by public insurance (78.9% vs 69.8%, P = .010) and discharged to homelessness (34.8% vs 22.6%, P = .041).

Measures of Illness

Patients considered definitely medically appropriate had lower ESI scores, indicative of more acute presentation, than those without definite medical acuity (2.73 [95% CI 2.64-2.81] vs 2.87 [95% CI 2.78-2.95], P = .026). There was no difference in CCI scores (Table 1).

Reason for Admission and Outcomes

Admissions considered definitely medically appropriate more frequently had an identified diagnosis/syndrome (66% vs 53%) or objective measurement (8.7% vs 2.7%) listed as the reason for admission, whereas patients admitted without definite medical acuity more freuqently had undifferentiated symptoms (34.7% vs 24%) or other/disposition (6% vs 1.3%) listed. The most common factors that triage physicians cited as contributing to the decision to admit patients without definite medical acuity included homelessness (34%), lack of outpatient social support (32%), and substance use disorder (25%). More details are available in Appendix Tables 1 and 2.

Admissions without definite medical acuity were longer than those with definite medical acuity (6.6 vs 6.0 days, P = .038), but there was no difference in emergency department readmissions within 48 hours or hospital readmissions within 30 days (Table 1).

Adverse Events

We identified 76 AEs in 41 admissions without definite medical acuity (range 0-10 AEs per admission) and 63 AEs in 44 definitely medically appropriate admissions (range 0-4 AEs per admission). The percentage of admissions with AE (27.3% vs 29.3%; RR 0.93, 95% CI 0.65-1.34, P = .70) and the rate of AE/1,000 patient-days (76.8 vs 70.4; IRR 1.09, 95% CI 0.77-1.55, P = .61) did not show statistically significant differences. The distribution of AE severity was similar between the two groups (Table 2). Most identified AEs caused temporary harm to the patient and were rated at severity levels E or F. Severe AEs, including at least one level I (patient death), occurred in both groups. The complete listing of positive triggers leading to adverse event identification by group and severity is available in Appendix Table 3.

By using a robust, standardized method, we found that patients admitted without definite medical acuity experienced the same number of inpatient AEs as patients admitted for definitely medically appropriate reasons. While the groups were relatively similar overall in terms of demographics and chronic comorbidity, we found evidence of social vulnerability in the group admitted without definite medical acuity in the form of increased rates of homelessness, triage physician concern regarding the lack of outpatient social support, and disposition-related reasons for admission. That both groups suffered harm―including patient death―while admitted to the hospital is striking, in particular for those patients who were admitted because of the lack of suitable outpatient options.

The potential limitations to the generalizability of this work include the single-site, safety-net setting and the use of individual physician determination of admission appropriateness. The proportion of admissions without definite medical acuity reported here is similar to that reported by previously published admission decision-making studies,^2,3 and the rate of AEs observed is similar to rates measured in other studies using the trigger tool methodology.^5,13 These similarities suggest some commonality across settings. Our study treats triage physician assessment as the marker of difference in defining the two groups and is an inherently subjective assessment that is reflective of real-world, holistic decision-making. Notably, the triage physician assessment was corroborated by corresponding differences in the ESI score, an acute triage assessment completed by a clinician outside of our team.

This study adds foundational knowledge to the risk/benefit discussion surrounding the decision to admit. Physician admission decisions are likely influenced by concern for the safety of vulnerable patients. Our results suggest that considering the risk of hospitalization itself in this decision-making remains important.

Readmission rates have been used by payers to administer financial incentives or penalties to hospitals as a measure of quality. The Centers for Medicare and Medicaid Services (CMS) reduces payments to hospitals with excess readmissions for adult Medicare patients.¹ Although the Medicare readmission penalties do not apply to children, several state Medicaid agencies have adopted policies to reduce reimbursement for hospitals with higher than expected readmission rates. These Medicaid programs often use potentially preventable readmission (PPR) rates calculated with proprietary software.² As a result of these incentives and with a goal of improving care, many children’s hospitals have focused on reducing readmissions through participation in local, regional, and national collaboratives.³

Rates of unplanned readmissions in children are lower than in older adults, with all-cause 30-day pediatric readmission rates around 13%.^4-7 Even so, as many as 30% of pediatric readmissions may be potentially preventable, with the most common transition failure involving a hospital factor, such as failure to recognize worsening clinical status prior to discharge.⁸ While readmission metrics are often judged across peer institutions, little is known about national trends over time. Therefore, we sought to examine readmission rates at children’s hospitals over a six-year timeframe to determine if progress has been made toward reducing readmissions.

We utilized data from the Children’s Hospital Association Inpatient Essentials Database and included index hospitalizations from January 1, 2010 through June 30, 2016. This database contains demographic information, diagnosis and procedure codes, and All-Patient Refined Diagnosis-Related Groups (APR-DRGs; 3M Health Information Systems) to describe the principal reason for each hospitalization.⁹ We included 66 hospitals from 31 states plus the District of Columbia with complete data during the study period.

Seven-day all-cause (AC) readmission and PPR rates were calculated using the output from 3M potentially preventable readmission software (version 32). The PPR software utilizes a proprietary algorithm to designate potentially preventable readmissions based on diagnosis codes and the severity of illness (as measured by the APR-DRG severity of illness classification). We chose seven-day readmissions, as opposed to a longer window, as readmissions soon after discharge are more likely to be preventable⁸ and thus theoretically more amenable to prevention efforts. Quarterly rates were generated for each hospital and in aggregate across the population. We chose quarterly rates a priori to assess changes in rates without focusing on minor monthly fluctuations due to seasonal differences. We performed generalized linear mixed regression models with cluster adjustments at the hospital level to assess changes in readmission rates over time adjusted for case mix index, as admissions to children’s hospitals have increased in complexity over time.^10,11 We operationalized the case mix index as an average of pediatric admissions’ relative weights at each hospital for the quarter.¹² We assessed AC and PPR models separately. The average case mix index was a covariate in both regression models.

Finally, to determine if readmission reduction may be specific to particular conditions, we generated readmission rates for a select number of APR-DRGs. We focused on conditions with a very high percentage of AC readmissions classified as PPR (appendectomy, connective tissue disorders, ventricular shunt procedures, bronchiolitis, asthma, and sickle cell crisis) as well as those with a very low percentage of AC readmissions classified as PPR (gastrointestinal infections, hematologic disease, and bone marrow transplant [BMT]).⁵

We included 4.52 million admissions to the 66 included hospitals. Most hospitals (62%) were freestanding acute-care children’s hospitals. The hospitals were geographically diverse. Two-thirds had magnet status (Appendix Table 1). Appendix Table 2 displays patient/admission characteristics over time. Approximately 49% of children were non-Hispanic white, 19% were non-Hispanic black, and 19% were Hispanic. Half of the children were insured by Medicaid. These characteristics were stable over time, except case mix index, which increased during the study period (P = .04).

Across Diagnosis All-Cause and Potentially Preventable Readmission Rates

Over the study period, there were 227,378 AC seven-day readmissions (5.1% readmission rate), and 91,467 readmissions (40% of AC readmissions) were considered PPRs. Readmission rates did not vary over the study period (Figure, Panel A). The median AC seven-day readmission rate across all quarters was 5.1%, ranging from 4.3% to 5.3% (Figure, Panels A and B). The median seven-day PPR rate across all quarters was 2.5% and ranged from 2.1% to 2.5% (Figure, Panels A and C). When adjusted for case mix index, the AC rate increased slightly (on average 0.006% increase per quarter, P = .01) and PPR rates were unchanged over time (PPR model P = .14; Figure, Panel D).

Condition-Specific Readmission Rates

Of the condition-specific readmission rates, only the AC rate for BMT changed significantly, with a decrease of 0.1% per quarter, P = .048. None of the conditions had significant trends in increasing or decreasing readmission in PPR rates. Some conditions, including sickle cell and cerebrospinal fluid ventricular shunt procedures, had fluctuating readmission rates throughout the study period (Appendix Figure, Panels A-G).

Despite substantial national efforts to reduce pediatric readmissions,³ seven-day readmission rates at children’s hospitals have not decreased over six years. When individual conditions are examined, there are minor fluctuations of readmission rates over time but no clear trend of decreased readmission events.

Our results are contrary to findings in the Medicare population, where 30-day readmission rates have decreased over time.^13,14 In these analyses, we focused on seven-day readmission, as earlier pediatric readmissions are more likely to be preventable. Importantly, the majority of our included hospitals (88%) participate in the Solutions for Patient Safety collaborative, which focuses on reducing seven-day readmissions. Thus, we are confident that a concerted effort to decrease readmission has been ongoing. Further, our findings are contrary to recent analyses indicating an increase in pediatric readmission rates using the pediatric all-condition readmission rate in the National Readmission Database.¹⁵ Our analyses are distinctly different in that they allow a focus on hospital-level performance in children’s hospitals. Although in our analyses the all-cause adjusted readmission rate did increase significantly over time (0.006% a quarter or 0.024% per year), this small increase is unlikely to be clinically relevant.

There are several potential reasons for the lack of change in pediatric readmission rates despite concerted efforts to decrease readmissions. First, pediatric readmissions across all conditions are relatively infrequent compared with adult readmission rates. Extrapolating from the largest pediatric study on readmission preventability,⁸ it is estimated that only two in 100 pediatric hospitalizations results in a PPR.¹⁶ Given the lack of robust pediatric readmission prediction tools, the ability to prospectively identify children at high risk for readmission and target interventions is challenging. Second, as we have previously described, children are readmitted after hospitalization for a wide variety of conditions.⁵ Medicare readmission penalties are leveraged on specific conditions; yet, Medicaid policies include all conditions. In pediatrics, successful interventions to reduce readmissions have focused on hospitalizations for specific conditions.¹⁷ In the only two large pediatric readmission reduction trials across multiple conditions, postdischarge homecare nursing contact did not reduce reutilization.^18,19 It is challenging to decrease readmissions in heterogenous populations without a robust set of evidence-based interventions. Third, there are multiple ways to measure pediatric readmissions, and different institutions may focus on different methods. Given the proprietary nature and the reliance on retrospective administrative data, PPR rates cannot be assessed during admission and thus are not feasible as a real-time quality improvement outcome. Fourth, in contrast to other hospital quality metrics such as central line-associated bloodstream infections or catheter-associated urinary tract infection, the locus of control for readmission is not entirely within the purview of the hospital.

It is unclear what readmission rate in children is appropriate—or safe—and whether that level has already been met. National readmission prevention efforts may have collateral benefits such as improved communication, medication errors or adherence, and other important aspects of care during transitions. In this scenario, lower readmission rates may not reflect improved quality. Future research should focus on determining if and how readmission reduction efforts are helping to ease the transition to home. Alternatively, research should determine if there are better interventions to assist with transition challenges which should receive resources divested from failing readmission reduction efforts.

Using administrative data, we are limited in delineating truly preventable readmissions from nonpreventable readmissions. Nevertheless, we chose to focus on the PPR and AC metrics, as these are the most policy-relevant metrics. Additionally, we examined aggregate rates of readmission across a cohort of hospitals and did not assess for within-hospital changes in readmission rates. Thus, it is possible (and likely) that some hospitals saw improvements and others saw increases in readmission rates during the study period. We are unable to examine readmission rates at hospitals based on investment in readmission reduction efforts or individual state Medicaid reimbursement policies. Finally, we are unable to assess readmissions to other institutions; however, it is unlikely that readmissions to other hospitals have decreased significantly when readmissions to the discharging hospital have not changed.

Pediatric readmissions at children’s hospitals have not decreased in the past six years, despite widespread readmission reduction efforts. Readmission rates for individual conditions have fluctuated but have not decreased.

Disclosures

Dr. Auger reports grants from AHRQ, during the conduct of the study. Drs. Harris, Gay, Teufel, McLead, Neuman, Peltz, Morse, Del Beccaro, Simon, Argawal, and Fieldston have nothing to disclose. Dr. Shah is the Editor-in-Chief of the Journal of Hospital Medicine.

Funding

Dr. Auger’s research is funded by a K08 award from the Agency for Healthcare Research and Quality (1K08HS024735-01A).

Readmission rates have been used by payers to administer financial incentives or penalties to hospitals as a measure of quality. The Centers for Medicare and Medicaid Services (CMS) reduces payments to hospitals with excess readmissions for adult Medicare patients.¹ Although the Medicare readmission penalties do not apply to children, several state Medicaid agencies have adopted policies to reduce reimbursement for hospitals with higher than expected readmission rates. These Medicaid programs often use potentially preventable readmission (PPR) rates calculated with proprietary software.² As a result of these incentives and with a goal of improving care, many children’s hospitals have focused on reducing readmissions through participation in local, regional, and national collaboratives.³

Rates of unplanned readmissions in children are lower than in older adults, with all-cause 30-day pediatric readmission rates around 13%.^4-7 Even so, as many as 30% of pediatric readmissions may be potentially preventable, with the most common transition failure involving a hospital factor, such as failure to recognize worsening clinical status prior to discharge.⁸ While readmission metrics are often judged across peer institutions, little is known about national trends over time. Therefore, we sought to examine readmission rates at children’s hospitals over a six-year timeframe to determine if progress has been made toward reducing readmissions.

We utilized data from the Children’s Hospital Association Inpatient Essentials Database and included index hospitalizations from January 1, 2010 through June 30, 2016. This database contains demographic information, diagnosis and procedure codes, and All-Patient Refined Diagnosis-Related Groups (APR-DRGs; 3M Health Information Systems) to describe the principal reason for each hospitalization.⁹ We included 66 hospitals from 31 states plus the District of Columbia with complete data during the study period.

Seven-day all-cause (AC) readmission and PPR rates were calculated using the output from 3M potentially preventable readmission software (version 32). The PPR software utilizes a proprietary algorithm to designate potentially preventable readmissions based on diagnosis codes and the severity of illness (as measured by the APR-DRG severity of illness classification). We chose seven-day readmissions, as opposed to a longer window, as readmissions soon after discharge are more likely to be preventable⁸ and thus theoretically more amenable to prevention efforts. Quarterly rates were generated for each hospital and in aggregate across the population. We chose quarterly rates a priori to assess changes in rates without focusing on minor monthly fluctuations due to seasonal differences. We performed generalized linear mixed regression models with cluster adjustments at the hospital level to assess changes in readmission rates over time adjusted for case mix index, as admissions to children’s hospitals have increased in complexity over time.^10,11 We operationalized the case mix index as an average of pediatric admissions’ relative weights at each hospital for the quarter.¹² We assessed AC and PPR models separately. The average case mix index was a covariate in both regression models.

Finally, to determine if readmission reduction may be specific to particular conditions, we generated readmission rates for a select number of APR-DRGs. We focused on conditions with a very high percentage of AC readmissions classified as PPR (appendectomy, connective tissue disorders, ventricular shunt procedures, bronchiolitis, asthma, and sickle cell crisis) as well as those with a very low percentage of AC readmissions classified as PPR (gastrointestinal infections, hematologic disease, and bone marrow transplant [BMT]).⁵

We included 4.52 million admissions to the 66 included hospitals. Most hospitals (62%) were freestanding acute-care children’s hospitals. The hospitals were geographically diverse. Two-thirds had magnet status (Appendix Table 1). Appendix Table 2 displays patient/admission characteristics over time. Approximately 49% of children were non-Hispanic white, 19% were non-Hispanic black, and 19% were Hispanic. Half of the children were insured by Medicaid. These characteristics were stable over time, except case mix index, which increased during the study period (P = .04).

Across Diagnosis All-Cause and Potentially Preventable Readmission Rates

Over the study period, there were 227,378 AC seven-day readmissions (5.1% readmission rate), and 91,467 readmissions (40% of AC readmissions) were considered PPRs. Readmission rates did not vary over the study period (Figure, Panel A). The median AC seven-day readmission rate across all quarters was 5.1%, ranging from 4.3% to 5.3% (Figure, Panels A and B). The median seven-day PPR rate across all quarters was 2.5% and ranged from 2.1% to 2.5% (Figure, Panels A and C). When adjusted for case mix index, the AC rate increased slightly (on average 0.006% increase per quarter, P = .01) and PPR rates were unchanged over time (PPR model P = .14; Figure, Panel D).

Condition-Specific Readmission Rates

Of the condition-specific readmission rates, only the AC rate for BMT changed significantly, with a decrease of 0.1% per quarter, P = .048. None of the conditions had significant trends in increasing or decreasing readmission in PPR rates. Some conditions, including sickle cell and cerebrospinal fluid ventricular shunt procedures, had fluctuating readmission rates throughout the study period (Appendix Figure, Panels A-G).

Despite substantial national efforts to reduce pediatric readmissions,³ seven-day readmission rates at children’s hospitals have not decreased over six years. When individual conditions are examined, there are minor fluctuations of readmission rates over time but no clear trend of decreased readmission events.

Our results are contrary to findings in the Medicare population, where 30-day readmission rates have decreased over time.^13,14 In these analyses, we focused on seven-day readmission, as earlier pediatric readmissions are more likely to be preventable. Importantly, the majority of our included hospitals (88%) participate in the Solutions for Patient Safety collaborative, which focuses on reducing seven-day readmissions. Thus, we are confident that a concerted effort to decrease readmission has been ongoing. Further, our findings are contrary to recent analyses indicating an increase in pediatric readmission rates using the pediatric all-condition readmission rate in the National Readmission Database.¹⁵ Our analyses are distinctly different in that they allow a focus on hospital-level performance in children’s hospitals. Although in our analyses the all-cause adjusted readmission rate did increase significantly over time (0.006% a quarter or 0.024% per year), this small increase is unlikely to be clinically relevant.

There are several potential reasons for the lack of change in pediatric readmission rates despite concerted efforts to decrease readmissions. First, pediatric readmissions across all conditions are relatively infrequent compared with adult readmission rates. Extrapolating from the largest pediatric study on readmission preventability,⁸ it is estimated that only two in 100 pediatric hospitalizations results in a PPR.¹⁶ Given the lack of robust pediatric readmission prediction tools, the ability to prospectively identify children at high risk for readmission and target interventions is challenging. Second, as we have previously described, children are readmitted after hospitalization for a wide variety of conditions.⁵ Medicare readmission penalties are leveraged on specific conditions; yet, Medicaid policies include all conditions. In pediatrics, successful interventions to reduce readmissions have focused on hospitalizations for specific conditions.¹⁷ In the only two large pediatric readmission reduction trials across multiple conditions, postdischarge homecare nursing contact did not reduce reutilization.^18,19 It is challenging to decrease readmissions in heterogenous populations without a robust set of evidence-based interventions. Third, there are multiple ways to measure pediatric readmissions, and different institutions may focus on different methods. Given the proprietary nature and the reliance on retrospective administrative data, PPR rates cannot be assessed during admission and thus are not feasible as a real-time quality improvement outcome. Fourth, in contrast to other hospital quality metrics such as central line-associated bloodstream infections or catheter-associated urinary tract infection, the locus of control for readmission is not entirely within the purview of the hospital.

It is unclear what readmission rate in children is appropriate—or safe—and whether that level has already been met. National readmission prevention efforts may have collateral benefits such as improved communication, medication errors or adherence, and other important aspects of care during transitions. In this scenario, lower readmission rates may not reflect improved quality. Future research should focus on determining if and how readmission reduction efforts are helping to ease the transition to home. Alternatively, research should determine if there are better interventions to assist with transition challenges which should receive resources divested from failing readmission reduction efforts.

Using administrative data, we are limited in delineating truly preventable readmissions from nonpreventable readmissions. Nevertheless, we chose to focus on the PPR and AC metrics, as these are the most policy-relevant metrics. Additionally, we examined aggregate rates of readmission across a cohort of hospitals and did not assess for within-hospital changes in readmission rates. Thus, it is possible (and likely) that some hospitals saw improvements and others saw increases in readmission rates during the study period. We are unable to examine readmission rates at hospitals based on investment in readmission reduction efforts or individual state Medicaid reimbursement policies. Finally, we are unable to assess readmissions to other institutions; however, it is unlikely that readmissions to other hospitals have decreased significantly when readmissions to the discharging hospital have not changed.

Pediatric readmissions at children’s hospitals have not decreased in the past six years, despite widespread readmission reduction efforts. Readmission rates for individual conditions have fluctuated but have not decreased.

Disclosures

Dr. Auger reports grants from AHRQ, during the conduct of the study. Drs. Harris, Gay, Teufel, McLead, Neuman, Peltz, Morse, Del Beccaro, Simon, Argawal, and Fieldston have nothing to disclose. Dr. Shah is the Editor-in-Chief of the Journal of Hospital Medicine.

Funding

Dr. Auger’s research is funded by a K08 award from the Agency for Healthcare Research and Quality (1K08HS024735-01A).

Readmission rates have been used by payers to administer financial incentives or penalties to hospitals as a measure of quality. The Centers for Medicare and Medicaid Services (CMS) reduces payments to hospitals with excess readmissions for adult Medicare patients.¹ Although the Medicare readmission penalties do not apply to children, several state Medicaid agencies have adopted policies to reduce reimbursement for hospitals with higher than expected readmission rates. These Medicaid programs often use potentially preventable readmission (PPR) rates calculated with proprietary software.² As a result of these incentives and with a goal of improving care, many children’s hospitals have focused on reducing readmissions through participation in local, regional, and national collaboratives.³

Rates of unplanned readmissions in children are lower than in older adults, with all-cause 30-day pediatric readmission rates around 13%.^4-7 Even so, as many as 30% of pediatric readmissions may be potentially preventable, with the most common transition failure involving a hospital factor, such as failure to recognize worsening clinical status prior to discharge.⁸ While readmission metrics are often judged across peer institutions, little is known about national trends over time. Therefore, we sought to examine readmission rates at children’s hospitals over a six-year timeframe to determine if progress has been made toward reducing readmissions.

We utilized data from the Children’s Hospital Association Inpatient Essentials Database and included index hospitalizations from January 1, 2010 through June 30, 2016. This database contains demographic information, diagnosis and procedure codes, and All-Patient Refined Diagnosis-Related Groups (APR-DRGs; 3M Health Information Systems) to describe the principal reason for each hospitalization.⁹ We included 66 hospitals from 31 states plus the District of Columbia with complete data during the study period.

Seven-day all-cause (AC) readmission and PPR rates were calculated using the output from 3M potentially preventable readmission software (version 32). The PPR software utilizes a proprietary algorithm to designate potentially preventable readmissions based on diagnosis codes and the severity of illness (as measured by the APR-DRG severity of illness classification). We chose seven-day readmissions, as opposed to a longer window, as readmissions soon after discharge are more likely to be preventable⁸ and thus theoretically more amenable to prevention efforts. Quarterly rates were generated for each hospital and in aggregate across the population. We chose quarterly rates a priori to assess changes in rates without focusing on minor monthly fluctuations due to seasonal differences. We performed generalized linear mixed regression models with cluster adjustments at the hospital level to assess changes in readmission rates over time adjusted for case mix index, as admissions to children’s hospitals have increased in complexity over time.^10,11 We operationalized the case mix index as an average of pediatric admissions’ relative weights at each hospital for the quarter.¹² We assessed AC and PPR models separately. The average case mix index was a covariate in both regression models.

Finally, to determine if readmission reduction may be specific to particular conditions, we generated readmission rates for a select number of APR-DRGs. We focused on conditions with a very high percentage of AC readmissions classified as PPR (appendectomy, connective tissue disorders, ventricular shunt procedures, bronchiolitis, asthma, and sickle cell crisis) as well as those with a very low percentage of AC readmissions classified as PPR (gastrointestinal infections, hematologic disease, and bone marrow transplant [BMT]).⁵

We included 4.52 million admissions to the 66 included hospitals. Most hospitals (62%) were freestanding acute-care children’s hospitals. The hospitals were geographically diverse. Two-thirds had magnet status (Appendix Table 1). Appendix Table 2 displays patient/admission characteristics over time. Approximately 49% of children were non-Hispanic white, 19% were non-Hispanic black, and 19% were Hispanic. Half of the children were insured by Medicaid. These characteristics were stable over time, except case mix index, which increased during the study period (P = .04).

Across Diagnosis All-Cause and Potentially Preventable Readmission Rates

Over the study period, there were 227,378 AC seven-day readmissions (5.1% readmission rate), and 91,467 readmissions (40% of AC readmissions) were considered PPRs. Readmission rates did not vary over the study period (Figure, Panel A). The median AC seven-day readmission rate across all quarters was 5.1%, ranging from 4.3% to 5.3% (Figure, Panels A and B). The median seven-day PPR rate across all quarters was 2.5% and ranged from 2.1% to 2.5% (Figure, Panels A and C). When adjusted for case mix index, the AC rate increased slightly (on average 0.006% increase per quarter, P = .01) and PPR rates were unchanged over time (PPR model P = .14; Figure, Panel D).

Condition-Specific Readmission Rates

Of the condition-specific readmission rates, only the AC rate for BMT changed significantly, with a decrease of 0.1% per quarter, P = .048. None of the conditions had significant trends in increasing or decreasing readmission in PPR rates. Some conditions, including sickle cell and cerebrospinal fluid ventricular shunt procedures, had fluctuating readmission rates throughout the study period (Appendix Figure, Panels A-G).

Despite substantial national efforts to reduce pediatric readmissions,³ seven-day readmission rates at children’s hospitals have not decreased over six years. When individual conditions are examined, there are minor fluctuations of readmission rates over time but no clear trend of decreased readmission events.

Our results are contrary to findings in the Medicare population, where 30-day readmission rates have decreased over time.^13,14 In these analyses, we focused on seven-day readmission, as earlier pediatric readmissions are more likely to be preventable. Importantly, the majority of our included hospitals (88%) participate in the Solutions for Patient Safety collaborative, which focuses on reducing seven-day readmissions. Thus, we are confident that a concerted effort to decrease readmission has been ongoing. Further, our findings are contrary to recent analyses indicating an increase in pediatric readmission rates using the pediatric all-condition readmission rate in the National Readmission Database.¹⁵ Our analyses are distinctly different in that they allow a focus on hospital-level performance in children’s hospitals. Although in our analyses the all-cause adjusted readmission rate did increase significantly over time (0.006% a quarter or 0.024% per year), this small increase is unlikely to be clinically relevant.

There are several potential reasons for the lack of change in pediatric readmission rates despite concerted efforts to decrease readmissions. First, pediatric readmissions across all conditions are relatively infrequent compared with adult readmission rates. Extrapolating from the largest pediatric study on readmission preventability,⁸ it is estimated that only two in 100 pediatric hospitalizations results in a PPR.¹⁶ Given the lack of robust pediatric readmission prediction tools, the ability to prospectively identify children at high risk for readmission and target interventions is challenging. Second, as we have previously described, children are readmitted after hospitalization for a wide variety of conditions.⁵ Medicare readmission penalties are leveraged on specific conditions; yet, Medicaid policies include all conditions. In pediatrics, successful interventions to reduce readmissions have focused on hospitalizations for specific conditions.¹⁷ In the only two large pediatric readmission reduction trials across multiple conditions, postdischarge homecare nursing contact did not reduce reutilization.^18,19 It is challenging to decrease readmissions in heterogenous populations without a robust set of evidence-based interventions. Third, there are multiple ways to measure pediatric readmissions, and different institutions may focus on different methods. Given the proprietary nature and the reliance on retrospective administrative data, PPR rates cannot be assessed during admission and thus are not feasible as a real-time quality improvement outcome. Fourth, in contrast to other hospital quality metrics such as central line-associated bloodstream infections or catheter-associated urinary tract infection, the locus of control for readmission is not entirely within the purview of the hospital.

It is unclear what readmission rate in children is appropriate—or safe—and whether that level has already been met. National readmission prevention efforts may have collateral benefits such as improved communication, medication errors or adherence, and other important aspects of care during transitions. In this scenario, lower readmission rates may not reflect improved quality. Future research should focus on determining if and how readmission reduction efforts are helping to ease the transition to home. Alternatively, research should determine if there are better interventions to assist with transition challenges which should receive resources divested from failing readmission reduction efforts.

Using administrative data, we are limited in delineating truly preventable readmissions from nonpreventable readmissions. Nevertheless, we chose to focus on the PPR and AC metrics, as these are the most policy-relevant metrics. Additionally, we examined aggregate rates of readmission across a cohort of hospitals and did not assess for within-hospital changes in readmission rates. Thus, it is possible (and likely) that some hospitals saw improvements and others saw increases in readmission rates during the study period. We are unable to examine readmission rates at hospitals based on investment in readmission reduction efforts or individual state Medicaid reimbursement policies. Finally, we are unable to assess readmissions to other institutions; however, it is unlikely that readmissions to other hospitals have decreased significantly when readmissions to the discharging hospital have not changed.

Pediatric readmissions at children’s hospitals have not decreased in the past six years, despite widespread readmission reduction efforts. Readmission rates for individual conditions have fluctuated but have not decreased.

Disclosures

Dr. Auger reports grants from AHRQ, during the conduct of the study. Drs. Harris, Gay, Teufel, McLead, Neuman, Peltz, Morse, Del Beccaro, Simon, Argawal, and Fieldston have nothing to disclose. Dr. Shah is the Editor-in-Chief of the Journal of Hospital Medicine.

Funding

Dr. Auger’s research is funded by a K08 award from the Agency for Healthcare Research and Quality (1K08HS024735-01A).

Unrecognized in-hospital deterioration can result in tragic consequences for pediatric patients. The majority of deterioration events have antecedents such as increasingly abnormal vital signs and new concerns from nurses.¹ Recent meta-analyses have shown that rapid response systems (RRSs), which include trigger mechanisms such as a pediatric early warning score (PEWS), are associated with a reduced rate of arrests and in-hospital mortality.^2,3 Cardiopulmonary arrest rates are useful metrics to judge the effectiveness of the system to identify and respond to deteriorating adult patients; however, there are important challenges to their use as an outcome measure in pediatrics. Arrests, which have been relatively uncommon in pediatric patients, are now even less frequent since the adoption of a RRS in the majority of children’s hospitals.^4,5 Several innovations in these systems will be context-dependent and hence best first evaluated in a single center, where arrests outside of the intensive care unit (ICU) may occur rarely. Identification of valid, more frequent proximal measures to arrests may better identify the risk factors for deterioration. This could potentially inform quality improvement efforts to mitigate clinical deterioration.

Bonafide et al. at the Children’s Hospital of Philadelphia developed and validated the critical deterioration event (CDE) metric, demonstrating that children who were transferred to the ICU and who received noninvasive ventilation, intubation, or vasopressor initiation within 12 hours of transfer had a >13-fold increased risk of in-hospital mortality.⁶ At Cincinnati Children’s Hospital Medical Center, an additional proximal outcome measure was developed for unrecognized clinical deterioration, now termed emergency transfers (ETs).^7-9 An ET is defined as any patient transferred to the ICU where the patient received intubation, inotropes, or three or more fluid boluses in the first hour after arrival or before transfer.⁹ Improvement science work that aimed at increasing clinician situation awareness was associated with a reduction in ETs,⁸ but the association of ETs with mortality or other healthcare utilization outcomes is unknown. The objective of this study was to determine the predictive validity of an ET on in-hospital mortality, ICU length of stay (LOS), and overall hospital LOS.

We conducted a case–control study at Cincinnati Children’s Hospital, a free-standing tertiary care children’s hospital. Our center has had an ICU-based RRS in place since 2005. In 2009, we eliminated the ICU consult such that each floor-to-ICU transfer is evaluated by the RRS. Nurses calculate a Monaghan PEWS every four hours on the majority of nursing units.

Patients of all ages cared for outside of the ICU at any point in their hospitalization from January 1, 2013, to July 31, 2017, were eligible for inclusion. There were no other exclusion criteria. The ICU included both the pediatric ICU and the cardiac ICU.

Cases

We identified all ET cases from an existing situation awareness database in which each RRS call is entered by the hospital nursing supervisor, whose role includes responding to each RRS activation. If the patient transfer meets the ET criteria, the nurse indicates this in the database. Each ET entry is later confirmed for assurance purposes by the nurse leader of the RRS committee (RG). For the purposes of this study, all records were again reviewed and validated using manual chart review in the electronic health record (Epic Systems, Verona, Wisconsin).

Controls

We identified nonemergent ICU transfers to serve as controls and matched those to ET in cases to limit the impact of confounders that may increase the likelihood of both an ET and a negative outcome such as ICU mortality. We identified up to three controls for each case from our database and matched in terms of age group (within five years of age), hospital unit before transfer, and time of year (within three months of ET). These variables were chosen to adjust for the impact of age, diversity of disease (as hospital units are generally organized by organ system of illness), and seasonality on outcomes.

Outcome Measures

Posttransfer LOS in the ICU, posttransfer hospital LOS, and in-hospital mortality were the primary outcome measures. Patient demographics, specific diagnoses, and number of medical conditions were a priori defined as covariates of interest. Data for each case and control were entered into a secure, web-based Research Electronic Data Capture (REDCap) database.

Analysis

Descriptive data were summarized using counts and percentages for categorical variables and medians and ranges for continuous variables due to nonnormal distributions. Chi-square test was used to compare in-hospital mortality between the ETs and the controls. The Wilcoxon rank-sum test was used to compare LOS between ETs and controls. All data analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, North Carolina).

A total of 45 ETs were identified, and 110 controls were matched. Patient demographics were similar among all cases and controls (P > .05). Patients with ETs had a median age of seven years (interquartile range: 3-18 years), and 51% of them were males. The majority of patients among our examined cases were white (68%) and non-Hispanic (93%). There was no statistical difference in insurance between the ETs and the controls. When evaluating the hospital unit before the transfer, ETs occurred most commonly in the Cardiology (22%), Hematology/Oncology (22%), and Neuroscience (16%) units.

ETs stayed longer in the ICU than non-ETs [median of 4.9 days vs 2.2 days, P = .001; Figure (A)]. Similarly, ET cases had a significantly longer posttransfer hospital LOS [median of 35 days vs 21 days, P = .001; Figure (B)]. ETs had a 22% in-hospital mortality rate, compared with 9% in-hospital mortality in the matched controls (P = .02; Table).

Children who experienced an ET had a significantly longer ICU LOS, a longer posttransfer LOS, and a higher in-hospital mortality than the matched controls who were also transferred to the ICU. Researchers and improvement science teams at multiple hospitals have demonstrated that interventions targeting improved situation awareness can reduce ETs; we have demonstrated that reducing ETs may reduce subsequent adverse outcomes.^8,10

These findings provide additional support for the use of the ET metric in children’s hospitals as a proximal measure for significant clinical deterioration. We found mortality rates that were overall high for a children’s hospital (22% in ET cases and 9% among controls) compared with a national average mortality rate of 2.3% in pediatric ICUs.¹¹ This is likely due to the study sample containing a significant proportion of children with medical complexity.

Aoki et al. recently demonstrated that ETs, compared with non-ETs, were associated with longer LOS and higher mortality in a bivariate analysis.¹² In our study, we found similar results with the important addition that these findings were robust when ETs were compared with matched controls who were likely at a higher risk of poor outcomes than ICU transfers in general. In addition, we demonstrated that ETs were associated with adverse outcomes in a United States children’s hospital with a mature, long-standing RRS process. As ETs are considerably more common than cardiac and respiratory arrests, use of the ET metric in children’s hospitals may enable more rapid learning and systems improvement implementations. We also found that most of the children with ETs present from units that care for children with substantial medical complexity, including Cardiology, Hematology/Oncology, and Neurosciences. Future work should continue to examine the relationship between medical complexity and ET risk.

The ET metric is complementary to the CDE measure developed by Bonafide et al. Both metrics capture potential events of unrecognized clinical deterioration, and both offer researchers the opportunity to better understand and improve their RRSs. Both ETs and CDEs are more common than arrests, and CDEs are more common than ETs. ETs, which by definition occur in the first hour of ICU care, are likely a more specific measure of unrecognized clinical deterioration. CDEs will capture therapies that may have been started up to 12 hours after transfer and thus are possibly more sensitive to identify unrecognized clinical deterioration. However, CDEs also may encompass some patients who arrived at the ICU after prompt recognition and then had a subacute deterioration in the ICU.

The maturity of the RRS and the bandwidth of teams to collect data may inform which metric(s) are best for individual centers. As ETs are less common and likely more specific to unrecognized clinical deterioration, they might be the first tracked as a center improves its RRS through QI methods. Alternatively, CDEs may be a useful metric for centers where unrecognized clinical deterioration is less common or in research studies where this more common outcome would lead to more power to detect the effect of interventions to improve care.

Our study had several limitations. Data collection was confined to one tertiary care children’s hospital with a high burden of complex cardiac and oncology care. The results may not generalize well to children hospitalized in smaller or community hospitals or in hospitals without a mature RRS. There is also the possibility of misclassification of covariates and outcomes, but any misclassification would likely be nondifferential and bias toward the null. Matching was not possible based on exact diagnosis, and the unit is a good but imperfect proxy for diagnosis grouping. At our center, overflow of patients into the Cardiology and Hematology/Oncology units is uncommon, mitigating this partially, although residual confounding may remain. The finding that ETs are associated with adverse outcomes does not necessarily mean that these events were preventable; however, it is important and encouraging that the rate of ETs has been reduced at two centers using improvement science interventions.^8,10

Patients who experienced an ET had a significantly higher likelihood of in-hospital mortality, spent more time in the ICU, and had a longer hospital LOS posttransfer than matched controls. The use of the ET metric in children’s hospitals would allow for further analysis of such patients in hopes of identifying clinical characteristics that serve as predictors of deterioration. This may facilitate better risk stratification in the clinical system as well as enable more rapid learning and systems improvements targeted toward preventing unrecognized clinical deterioration.

Disclosures

Dr. Hussain, Dr. Sosa, Dr. Ambroggio, and Mrs. Gallagher have nothing to disclose. Patrick Brady reports grants from the Agency for Healthcare Research and Quality, outside the submitted work. The authors certify that this submission is not under review by any other publication. The author team has no conflicts of interest to disclose.

Funding

Ms. Hussain was supported by the Society of Hospital Medicine’s Student Hospitalist Scholar Grant Program in 2017. Dr. Brady receives career development support from AHRQ K08-HS023827. The project described was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health, under Award Number 5UL1TR001425-04. The content is solely the responsibility of the authors and does not necessarily represent the official views of the SHM, AHRQ, or NIH.

Unrecognized in-hospital deterioration can result in tragic consequences for pediatric patients. The majority of deterioration events have antecedents such as increasingly abnormal vital signs and new concerns from nurses.¹ Recent meta-analyses have shown that rapid response systems (RRSs), which include trigger mechanisms such as a pediatric early warning score (PEWS), are associated with a reduced rate of arrests and in-hospital mortality.^2,3 Cardiopulmonary arrest rates are useful metrics to judge the effectiveness of the system to identify and respond to deteriorating adult patients; however, there are important challenges to their use as an outcome measure in pediatrics. Arrests, which have been relatively uncommon in pediatric patients, are now even less frequent since the adoption of a RRS in the majority of children’s hospitals.^4,5 Several innovations in these systems will be context-dependent and hence best first evaluated in a single center, where arrests outside of the intensive care unit (ICU) may occur rarely. Identification of valid, more frequent proximal measures to arrests may better identify the risk factors for deterioration. This could potentially inform quality improvement efforts to mitigate clinical deterioration.

Bonafide et al. at the Children’s Hospital of Philadelphia developed and validated the critical deterioration event (CDE) metric, demonstrating that children who were transferred to the ICU and who received noninvasive ventilation, intubation, or vasopressor initiation within 12 hours of transfer had a >13-fold increased risk of in-hospital mortality.⁶ At Cincinnati Children’s Hospital Medical Center, an additional proximal outcome measure was developed for unrecognized clinical deterioration, now termed emergency transfers (ETs).^7-9 An ET is defined as any patient transferred to the ICU where the patient received intubation, inotropes, or three or more fluid boluses in the first hour after arrival or before transfer.⁹ Improvement science work that aimed at increasing clinician situation awareness was associated with a reduction in ETs,⁸ but the association of ETs with mortality or other healthcare utilization outcomes is unknown. The objective of this study was to determine the predictive validity of an ET on in-hospital mortality, ICU length of stay (LOS), and overall hospital LOS.

We conducted a case–control study at Cincinnati Children’s Hospital, a free-standing tertiary care children’s hospital. Our center has had an ICU-based RRS in place since 2005. In 2009, we eliminated the ICU consult such that each floor-to-ICU transfer is evaluated by the RRS. Nurses calculate a Monaghan PEWS every four hours on the majority of nursing units.

Patients of all ages cared for outside of the ICU at any point in their hospitalization from January 1, 2013, to July 31, 2017, were eligible for inclusion. There were no other exclusion criteria. The ICU included both the pediatric ICU and the cardiac ICU.

Cases

We identified all ET cases from an existing situation awareness database in which each RRS call is entered by the hospital nursing supervisor, whose role includes responding to each RRS activation. If the patient transfer meets the ET criteria, the nurse indicates this in the database. Each ET entry is later confirmed for assurance purposes by the nurse leader of the RRS committee (RG). For the purposes of this study, all records were again reviewed and validated using manual chart review in the electronic health record (Epic Systems, Verona, Wisconsin).

Controls

We identified nonemergent ICU transfers to serve as controls and matched those to ET in cases to limit the impact of confounders that may increase the likelihood of both an ET and a negative outcome such as ICU mortality. We identified up to three controls for each case from our database and matched in terms of age group (within five years of age), hospital unit before transfer, and time of year (within three months of ET). These variables were chosen to adjust for the impact of age, diversity of disease (as hospital units are generally organized by organ system of illness), and seasonality on outcomes.

Outcome Measures

Posttransfer LOS in the ICU, posttransfer hospital LOS, and in-hospital mortality were the primary outcome measures. Patient demographics, specific diagnoses, and number of medical conditions were a priori defined as covariates of interest. Data for each case and control were entered into a secure, web-based Research Electronic Data Capture (REDCap) database.

Analysis

Descriptive data were summarized using counts and percentages for categorical variables and medians and ranges for continuous variables due to nonnormal distributions. Chi-square test was used to compare in-hospital mortality between the ETs and the controls. The Wilcoxon rank-sum test was used to compare LOS between ETs and controls. All data analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, North Carolina).

A total of 45 ETs were identified, and 110 controls were matched. Patient demographics were similar among all cases and controls (P > .05). Patients with ETs had a median age of seven years (interquartile range: 3-18 years), and 51% of them were males. The majority of patients among our examined cases were white (68%) and non-Hispanic (93%). There was no statistical difference in insurance between the ETs and the controls. When evaluating the hospital unit before the transfer, ETs occurred most commonly in the Cardiology (22%), Hematology/Oncology (22%), and Neuroscience (16%) units.

ETs stayed longer in the ICU than non-ETs [median of 4.9 days vs 2.2 days, P = .001; Figure (A)]. Similarly, ET cases had a significantly longer posttransfer hospital LOS [median of 35 days vs 21 days, P = .001; Figure (B)]. ETs had a 22% in-hospital mortality rate, compared with 9% in-hospital mortality in the matched controls (P = .02; Table).

Children who experienced an ET had a significantly longer ICU LOS, a longer posttransfer LOS, and a higher in-hospital mortality than the matched controls who were also transferred to the ICU. Researchers and improvement science teams at multiple hospitals have demonstrated that interventions targeting improved situation awareness can reduce ETs; we have demonstrated that reducing ETs may reduce subsequent adverse outcomes.^8,10

These findings provide additional support for the use of the ET metric in children’s hospitals as a proximal measure for significant clinical deterioration. We found mortality rates that were overall high for a children’s hospital (22% in ET cases and 9% among controls) compared with a national average mortality rate of 2.3% in pediatric ICUs.¹¹ This is likely due to the study sample containing a significant proportion of children with medical complexity.

Aoki et al. recently demonstrated that ETs, compared with non-ETs, were associated with longer LOS and higher mortality in a bivariate analysis.¹² In our study, we found similar results with the important addition that these findings were robust when ETs were compared with matched controls who were likely at a higher risk of poor outcomes than ICU transfers in general. In addition, we demonstrated that ETs were associated with adverse outcomes in a United States children’s hospital with a mature, long-standing RRS process. As ETs are considerably more common than cardiac and respiratory arrests, use of the ET metric in children’s hospitals may enable more rapid learning and systems improvement implementations. We also found that most of the children with ETs present from units that care for children with substantial medical complexity, including Cardiology, Hematology/Oncology, and Neurosciences. Future work should continue to examine the relationship between medical complexity and ET risk.

The ET metric is complementary to the CDE measure developed by Bonafide et al. Both metrics capture potential events of unrecognized clinical deterioration, and both offer researchers the opportunity to better understand and improve their RRSs. Both ETs and CDEs are more common than arrests, and CDEs are more common than ETs. ETs, which by definition occur in the first hour of ICU care, are likely a more specific measure of unrecognized clinical deterioration. CDEs will capture therapies that may have been started up to 12 hours after transfer and thus are possibly more sensitive to identify unrecognized clinical deterioration. However, CDEs also may encompass some patients who arrived at the ICU after prompt recognition and then had a subacute deterioration in the ICU.

The maturity of the RRS and the bandwidth of teams to collect data may inform which metric(s) are best for individual centers. As ETs are less common and likely more specific to unrecognized clinical deterioration, they might be the first tracked as a center improves its RRS through QI methods. Alternatively, CDEs may be a useful metric for centers where unrecognized clinical deterioration is less common or in research studies where this more common outcome would lead to more power to detect the effect of interventions to improve care.

Our study had several limitations. Data collection was confined to one tertiary care children’s hospital with a high burden of complex cardiac and oncology care. The results may not generalize well to children hospitalized in smaller or community hospitals or in hospitals without a mature RRS. There is also the possibility of misclassification of covariates and outcomes, but any misclassification would likely be nondifferential and bias toward the null. Matching was not possible based on exact diagnosis, and the unit is a good but imperfect proxy for diagnosis grouping. At our center, overflow of patients into the Cardiology and Hematology/Oncology units is uncommon, mitigating this partially, although residual confounding may remain. The finding that ETs are associated with adverse outcomes does not necessarily mean that these events were preventable; however, it is important and encouraging that the rate of ETs has been reduced at two centers using improvement science interventions.^8,10

Patients who experienced an ET had a significantly higher likelihood of in-hospital mortality, spent more time in the ICU, and had a longer hospital LOS posttransfer than matched controls. The use of the ET metric in children’s hospitals would allow for further analysis of such patients in hopes of identifying clinical characteristics that serve as predictors of deterioration. This may facilitate better risk stratification in the clinical system as well as enable more rapid learning and systems improvements targeted toward preventing unrecognized clinical deterioration.

Disclosures

Dr. Hussain, Dr. Sosa, Dr. Ambroggio, and Mrs. Gallagher have nothing to disclose. Patrick Brady reports grants from the Agency for Healthcare Research and Quality, outside the submitted work. The authors certify that this submission is not under review by any other publication. The author team has no conflicts of interest to disclose.

Funding

Ms. Hussain was supported by the Society of Hospital Medicine’s Student Hospitalist Scholar Grant Program in 2017. Dr. Brady receives career development support from AHRQ K08-HS023827. The project described was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health, under Award Number 5UL1TR001425-04. The content is solely the responsibility of the authors and does not necessarily represent the official views of the SHM, AHRQ, or NIH.

Unrecognized in-hospital deterioration can result in tragic consequences for pediatric patients. The majority of deterioration events have antecedents such as increasingly abnormal vital signs and new concerns from nurses.¹ Recent meta-analyses have shown that rapid response systems (RRSs), which include trigger mechanisms such as a pediatric early warning score (PEWS), are associated with a reduced rate of arrests and in-hospital mortality.^2,3 Cardiopulmonary arrest rates are useful metrics to judge the effectiveness of the system to identify and respond to deteriorating adult patients; however, there are important challenges to their use as an outcome measure in pediatrics. Arrests, which have been relatively uncommon in pediatric patients, are now even less frequent since the adoption of a RRS in the majority of children’s hospitals.^4,5 Several innovations in these systems will be context-dependent and hence best first evaluated in a single center, where arrests outside of the intensive care unit (ICU) may occur rarely. Identification of valid, more frequent proximal measures to arrests may better identify the risk factors for deterioration. This could potentially inform quality improvement efforts to mitigate clinical deterioration.

Bonafide et al. at the Children’s Hospital of Philadelphia developed and validated the critical deterioration event (CDE) metric, demonstrating that children who were transferred to the ICU and who received noninvasive ventilation, intubation, or vasopressor initiation within 12 hours of transfer had a >13-fold increased risk of in-hospital mortality.⁶ At Cincinnati Children’s Hospital Medical Center, an additional proximal outcome measure was developed for unrecognized clinical deterioration, now termed emergency transfers (ETs).^7-9 An ET is defined as any patient transferred to the ICU where the patient received intubation, inotropes, or three or more fluid boluses in the first hour after arrival or before transfer.⁹ Improvement science work that aimed at increasing clinician situation awareness was associated with a reduction in ETs,⁸ but the association of ETs with mortality or other healthcare utilization outcomes is unknown. The objective of this study was to determine the predictive validity of an ET on in-hospital mortality, ICU length of stay (LOS), and overall hospital LOS.

We conducted a case–control study at Cincinnati Children’s Hospital, a free-standing tertiary care children’s hospital. Our center has had an ICU-based RRS in place since 2005. In 2009, we eliminated the ICU consult such that each floor-to-ICU transfer is evaluated by the RRS. Nurses calculate a Monaghan PEWS every four hours on the majority of nursing units.

Patients of all ages cared for outside of the ICU at any point in their hospitalization from January 1, 2013, to July 31, 2017, were eligible for inclusion. There were no other exclusion criteria. The ICU included both the pediatric ICU and the cardiac ICU.

Cases

We identified all ET cases from an existing situation awareness database in which each RRS call is entered by the hospital nursing supervisor, whose role includes responding to each RRS activation. If the patient transfer meets the ET criteria, the nurse indicates this in the database. Each ET entry is later confirmed for assurance purposes by the nurse leader of the RRS committee (RG). For the purposes of this study, all records were again reviewed and validated using manual chart review in the electronic health record (Epic Systems, Verona, Wisconsin).

Controls

We identified nonemergent ICU transfers to serve as controls and matched those to ET in cases to limit the impact of confounders that may increase the likelihood of both an ET and a negative outcome such as ICU mortality. We identified up to three controls for each case from our database and matched in terms of age group (within five years of age), hospital unit before transfer, and time of year (within three months of ET). These variables were chosen to adjust for the impact of age, diversity of disease (as hospital units are generally organized by organ system of illness), and seasonality on outcomes.

Outcome Measures

Posttransfer LOS in the ICU, posttransfer hospital LOS, and in-hospital mortality were the primary outcome measures. Patient demographics, specific diagnoses, and number of medical conditions were a priori defined as covariates of interest. Data for each case and control were entered into a secure, web-based Research Electronic Data Capture (REDCap) database.

Analysis

Descriptive data were summarized using counts and percentages for categorical variables and medians and ranges for continuous variables due to nonnormal distributions. Chi-square test was used to compare in-hospital mortality between the ETs and the controls. The Wilcoxon rank-sum test was used to compare LOS between ETs and controls. All data analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, North Carolina).

A total of 45 ETs were identified, and 110 controls were matched. Patient demographics were similar among all cases and controls (P > .05). Patients with ETs had a median age of seven years (interquartile range: 3-18 years), and 51% of them were males. The majority of patients among our examined cases were white (68%) and non-Hispanic (93%). There was no statistical difference in insurance between the ETs and the controls. When evaluating the hospital unit before the transfer, ETs occurred most commonly in the Cardiology (22%), Hematology/Oncology (22%), and Neuroscience (16%) units.

ETs stayed longer in the ICU than non-ETs [median of 4.9 days vs 2.2 days, P = .001; Figure (A)]. Similarly, ET cases had a significantly longer posttransfer hospital LOS [median of 35 days vs 21 days, P = .001; Figure (B)]. ETs had a 22% in-hospital mortality rate, compared with 9% in-hospital mortality in the matched controls (P = .02; Table).

Children who experienced an ET had a significantly longer ICU LOS, a longer posttransfer LOS, and a higher in-hospital mortality than the matched controls who were also transferred to the ICU. Researchers and improvement science teams at multiple hospitals have demonstrated that interventions targeting improved situation awareness can reduce ETs; we have demonstrated that reducing ETs may reduce subsequent adverse outcomes.^8,10

These findings provide additional support for the use of the ET metric in children’s hospitals as a proximal measure for significant clinical deterioration. We found mortality rates that were overall high for a children’s hospital (22% in ET cases and 9% among controls) compared with a national average mortality rate of 2.3% in pediatric ICUs.¹¹ This is likely due to the study sample containing a significant proportion of children with medical complexity.

Aoki et al. recently demonstrated that ETs, compared with non-ETs, were associated with longer LOS and higher mortality in a bivariate analysis.¹² In our study, we found similar results with the important addition that these findings were robust when ETs were compared with matched controls who were likely at a higher risk of poor outcomes than ICU transfers in general. In addition, we demonstrated that ETs were associated with adverse outcomes in a United States children’s hospital with a mature, long-standing RRS process. As ETs are considerably more common than cardiac and respiratory arrests, use of the ET metric in children’s hospitals may enable more rapid learning and systems improvement implementations. We also found that most of the children with ETs present from units that care for children with substantial medical complexity, including Cardiology, Hematology/Oncology, and Neurosciences. Future work should continue to examine the relationship between medical complexity and ET risk.

The ET metric is complementary to the CDE measure developed by Bonafide et al. Both metrics capture potential events of unrecognized clinical deterioration, and both offer researchers the opportunity to better understand and improve their RRSs. Both ETs and CDEs are more common than arrests, and CDEs are more common than ETs. ETs, which by definition occur in the first hour of ICU care, are likely a more specific measure of unrecognized clinical deterioration. CDEs will capture therapies that may have been started up to 12 hours after transfer and thus are possibly more sensitive to identify unrecognized clinical deterioration. However, CDEs also may encompass some patients who arrived at the ICU after prompt recognition and then had a subacute deterioration in the ICU.

The maturity of the RRS and the bandwidth of teams to collect data may inform which metric(s) are best for individual centers. As ETs are less common and likely more specific to unrecognized clinical deterioration, they might be the first tracked as a center improves its RRS through QI methods. Alternatively, CDEs may be a useful metric for centers where unrecognized clinical deterioration is less common or in research studies where this more common outcome would lead to more power to detect the effect of interventions to improve care.

Our study had several limitations. Data collection was confined to one tertiary care children’s hospital with a high burden of complex cardiac and oncology care. The results may not generalize well to children hospitalized in smaller or community hospitals or in hospitals without a mature RRS. There is also the possibility of misclassification of covariates and outcomes, but any misclassification would likely be nondifferential and bias toward the null. Matching was not possible based on exact diagnosis, and the unit is a good but imperfect proxy for diagnosis grouping. At our center, overflow of patients into the Cardiology and Hematology/Oncology units is uncommon, mitigating this partially, although residual confounding may remain. The finding that ETs are associated with adverse outcomes does not necessarily mean that these events were preventable; however, it is important and encouraging that the rate of ETs has been reduced at two centers using improvement science interventions.^8,10

Patients who experienced an ET had a significantly higher likelihood of in-hospital mortality, spent more time in the ICU, and had a longer hospital LOS posttransfer than matched controls. The use of the ET metric in children’s hospitals would allow for further analysis of such patients in hopes of identifying clinical characteristics that serve as predictors of deterioration. This may facilitate better risk stratification in the clinical system as well as enable more rapid learning and systems improvements targeted toward preventing unrecognized clinical deterioration.

Disclosures

Dr. Hussain, Dr. Sosa, Dr. Ambroggio, and Mrs. Gallagher have nothing to disclose. Patrick Brady reports grants from the Agency for Healthcare Research and Quality, outside the submitted work. The authors certify that this submission is not under review by any other publication. The author team has no conflicts of interest to disclose.

Funding

Ms. Hussain was supported by the Society of Hospital Medicine’s Student Hospitalist Scholar Grant Program in 2017. Dr. Brady receives career development support from AHRQ K08-HS023827. The project described was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health, under Award Number 5UL1TR001425-04. The content is solely the responsibility of the authors and does not necessarily represent the official views of the SHM, AHRQ, or NIH.

Recently released guidelines on safe opioid prescribing draw attention to the fact that physicians have the ability to curb the opioid epidemic through better adherence to prescribing guidelines and limiting opioid use when not clinically indicated.^1,2 A consensus statement from the Society of Hospital Medicine includes 16 recommendations for improving the safety of opioid use in hospitalized patients, one of which is to use the oral route of administration whenever possible, reserving intravenous (IV) administration for patients who cannot take food or medications by mouth, patients suspected of gastrointestinal (GI) malabsorption, or when immediate pain control and/or rapid dose titration is necessary.² This recommendation was based on an increased risk of side effects, adverse events, and medication errors with IV compared with oral formulations.^3-5 Furthermore, the reinforcement from opioids is inversely related to the rate of onset of action, and therefore opioids administered by an IV route may be more likely to lead to addiction.^6-8

Choosing oral over IV opioids has several additional advantages. The cost of the IV formulation is more than oral; at our institution, the cost of IV morphine is 2.5-4.6 times greater than oral. Additional costs associated with IV administration include nursing time and equipment. Overall, transitioning patients from IV to oral medications could considerably lower costs of care.⁹ Ongoing need for an IV line may also lead to avoidable complications, including patient discomfort, infection, and thrombophlebitis. In addition, the recent national shortage of IV opioids has necessitated better stewardship of IV opioids.

Despite this recommendation, our observations suggest that patients often continue receiving IV opioids longer than clinically indicated. The goal of this study was to identify the incidence of potentially inappropriate IV opioid use in hospitalized patients.

The present study was an observational study seeking to quantify the burden of potentially inappropriate IV opioid use and characteristics predicting potentially inappropriate use in the inpatient setting at a large academic medical center in Boston, Massachusetts, using retrospective review of medical records.

Definition of Potentially Inappropriate Use and Study Sample

We identified all hospitalizations during the month of February 2017 with any order for IV opioids using pharmacy charge data and performed chart reviews in this sample until we reached our prespecified study sample of 200 hospitalizations meeting inclusion/exclusion criteria further defined below.

We defined potentially inappropriate use of IV opioids as use of IV opioids for greater than 24 hours in a patient who could receive oral medications (evidenced by receipt of other orally administered medications during the same 24-hour period) and was not mechanically ventilated. This definition is consistent with recommendations in the recently released consensus statement from the Society of Hospital Medicine.² We selected a time frame of 24 hours because IV pain medications may be indicated for initial immediate pain control and rapid dose titration; however, 24 hours should be sufficient time to determine opioid needs and transition to an oral regimen in patients without contraindications. After an initial IV dose, additional IV doses within 24 hours were considered appropriate, whereas IV doses thereafter were considered potentially inappropriate unless the patient had nil per os status, including medications. All IV opioids administered within 24 hours of a surgery or procedure were considered appropriate. Because it may be appropriate to continue IV opioids beyond 24 hours in patients with an active cancer diagnosis, in patients who have chosen comfort measures only, or in patients with GI dysfunction (including conditions such as small bowel obstruction, colitis, pancreatitis), we excluded these populations from the study sample. Patients admitted to the hospital for less than 24 hours were also excluded from the study, because they would not be at risk for the outcome of potentially inappropriate use. Doses of IV opioids administered for respiratory distress were considered to be appropriate. Given difficulty in identifying the appropriate time to transition from patient-controlled analgesia (PCA) to IV or per os (PO) opioids, days spent receiving opioids by PCA or continuous IV drip were excluded from the analysis.

We used Fisher’s exact test or the Chi-square test (in the setting of a multicategory variable) to calculate bivariable P values. We used multivariable logistic regression to identify independent predictors of receipt of at least one dose of potentially inappropriate IV opioids, using the hospitalization as the unit of analysis.

Of 630 hospitalizations with at least one order for IV opioids over a one-month period, we reviewed 502 charts, from which we excluded 76 hospitalizations with an active cancer diagnosis, 30 with comfort-focused care, 115 with GI dysfunction, and 108 with a hospitalization less than 24 hours in duration, resulting in 200 hospitalizations included in this analysis (some patients met multiple exclusion criteria). Table 1 outlines characteristics of the study population, stratified by appropriateness of IV opioid use. The study population was predominately white and had an average age of 56.3 years. The majority of patients were on a surgical service. Hydromorphone was the most commonly administered opioid. There were significant differences in the percentage of doses considered inappropriate between different types of opioids (P < .001), with morphine having the highest proportion of doses considered potentially inappropriate (Table 2).

Thirty-one percent of the cohort was administered at least one potentially inappropriate dose of IV opioids. A total of 432 of 1,319 (33%) IV doses were considered potentially inappropriate.

Predictors of Potentially Inappropriate Use

No significant associations were observed between potentially inappropriate IV opioid administration and age, sex, or admitting service (Table 1). Patients with an ethnicity described as other, unknown, or declined were less likely to have potentially inappropriate use.

In this cohort of medical and surgical inpatients, we found that almost one-third received at least one potentially inappropriate IV opioid administration during their hospitalization, and one-third of all IV opioid administrations were potentially inappropriate based on current recommendations defining the appropriate use of IV versus oral opioids. Although this is a single-center analysis, to our knowledge, this is the first study to ascertain the rate of potentially inappropriate IV opioid administration in hospitalized patients. Our findings suggest that quality improvement initiatives are necessary to promote more guideline-concordant care in this realm.

Several factors may contribute to overuse. Requests from patients for immediate pain relief may at times drive prescription of the IV formulation. In addition, patients may expect the IV formulation because of precedents from prior interactions with the healthcare system. Both of these situations may be opportunities for patient education about the equivalent bioavailability of oral and IV formulations in patients with a functioning GI tract, as well as the relatively small difference in rate of onset between the two routes of administration (generally 15-20 minutes). When a patient’s pain is well controlled with IV medications, physicians may also fail to recognize the need to transition to PO medications, further prolonging unnecessary use. Finally, in patients with multiple, complex, or deteriorating medical conditions, transitioning to oral opioids may be deprioritized for the sake of addressing more urgent medical concerns.

This study highlights the potential for transitioning more patients to oral opioids, which should be feasible in the inpatient setting, where pain needs can often be anticipated in advance and oral medications can be administered earlier to overcome the short delay in the onset of action between the oral and IV routes. Oral medications also have the advantage of a longer duration of effect, which may provide overall improved pain control. At our institution, a recent shortage of IV opioids (which occurred after the data collection period for this study) and subsequent efforts to limit IV opioid use (via computerized prompts and active pharmacist consultation) resulted in an immediate 50% reduction in the daily number of IV opioid administrations, further supporting our conclusion that there is an opportunity to decrease inappropriate use of IV opioids.

There were no specific patient factors that contributed to potentially inappropriate use. Although the ethnicity category of other/unknown/declined was significantly less likely to receive opioids potentially inappropriately, given the heterogeneity of this group, it is difficult to draw conclusions on the clinical significance of this finding. Morphine was significantly more likely than other opioids to be administered inappropriately.

There are several limitations of this study. Because this was a retrospective review, our criteria for appropriate use may have resulted in some misclassification; as a result, we can comment only on potentially inappropriate use rather than on definitively inappropriate use. We attempted to use a conservative definition of appropriateness by automatically assuming all doses in the first 24 hours of administration to be appropriate, which could have resulted in underestimating potentially inappropriate use. Nonetheless, there may be instances in which a patient had suspected malabsorption that was not captured or a fluctuating ability to receive oral medications within a given 24-hour period (due to nausea, for example), resulting in outcome misclassification. In addition, we did not correlate findings with patient-reported pain scores. Because there is no clearly defined pain threshold at which IV opioids are indicated, we did not believe that would be useful in clarifying appropriate versus inappropriate use. That said, we believe that, most of the time, pain medications should be able to be titrated appropriately within 24 hours to avoid the need for immediate pain relief with IV opioids thereafter. Although there may be instances of patients who have breakthrough pain severe enough to require IV opioids despite adequate titration of oral medications, we believe this is likely to represent a small number of our population that received potentially inappropriate use. It is worth noting that even if we overestimated by 50%, such that the true rate of potentially inappropriate IV administrations is 15%, we believe this would still be a ripe target for quality improvement initiatives, given that tens of millions of hospitalized patients receive opioids each year in the United States.¹⁰ Finally, we were unable to quantify the number of providers involved in decision making for these patients, and the single-center nature and short time frame of the study limit generalizability; our analysis should be replicated at other hospitals.

In conclusion, in this sample of 200 medical and surgical hospitalizations receiving IV opioids at a large academic medical center, we identified potentially inappropriate IV administration in 31%, suggesting potential to improve value through improving prescribing practices.

Disclosures

None of the authors have conflicts to disclose.

Funding

Dr. Herzig is funded by grant number K23AG042459 from the National Institute on Aging and R01HS026215 from the Agency for Healthcare Research and Quality. The manuscript contents are solely the responsibility of the authors and do not necessarily represent the views of the funding organizations.

Recently released guidelines on safe opioid prescribing draw attention to the fact that physicians have the ability to curb the opioid epidemic through better adherence to prescribing guidelines and limiting opioid use when not clinically indicated.^1,2 A consensus statement from the Society of Hospital Medicine includes 16 recommendations for improving the safety of opioid use in hospitalized patients, one of which is to use the oral route of administration whenever possible, reserving intravenous (IV) administration for patients who cannot take food or medications by mouth, patients suspected of gastrointestinal (GI) malabsorption, or when immediate pain control and/or rapid dose titration is necessary.² This recommendation was based on an increased risk of side effects, adverse events, and medication errors with IV compared with oral formulations.^3-5 Furthermore, the reinforcement from opioids is inversely related to the rate of onset of action, and therefore opioids administered by an IV route may be more likely to lead to addiction.^6-8

Choosing oral over IV opioids has several additional advantages. The cost of the IV formulation is more than oral; at our institution, the cost of IV morphine is 2.5-4.6 times greater than oral. Additional costs associated with IV administration include nursing time and equipment. Overall, transitioning patients from IV to oral medications could considerably lower costs of care.⁹ Ongoing need for an IV line may also lead to avoidable complications, including patient discomfort, infection, and thrombophlebitis. In addition, the recent national shortage of IV opioids has necessitated better stewardship of IV opioids.

Despite this recommendation, our observations suggest that patients often continue receiving IV opioids longer than clinically indicated. The goal of this study was to identify the incidence of potentially inappropriate IV opioid use in hospitalized patients.

The present study was an observational study seeking to quantify the burden of potentially inappropriate IV opioid use and characteristics predicting potentially inappropriate use in the inpatient setting at a large academic medical center in Boston, Massachusetts, using retrospective review of medical records.

Definition of Potentially Inappropriate Use and Study Sample

We identified all hospitalizations during the month of February 2017 with any order for IV opioids using pharmacy charge data and performed chart reviews in this sample until we reached our prespecified study sample of 200 hospitalizations meeting inclusion/exclusion criteria further defined below.

We defined potentially inappropriate use of IV opioids as use of IV opioids for greater than 24 hours in a patient who could receive oral medications (evidenced by receipt of other orally administered medications during the same 24-hour period) and was not mechanically ventilated. This definition is consistent with recommendations in the recently released consensus statement from the Society of Hospital Medicine.² We selected a time frame of 24 hours because IV pain medications may be indicated for initial immediate pain control and rapid dose titration; however, 24 hours should be sufficient time to determine opioid needs and transition to an oral regimen in patients without contraindications. After an initial IV dose, additional IV doses within 24 hours were considered appropriate, whereas IV doses thereafter were considered potentially inappropriate unless the patient had nil per os status, including medications. All IV opioids administered within 24 hours of a surgery or procedure were considered appropriate. Because it may be appropriate to continue IV opioids beyond 24 hours in patients with an active cancer diagnosis, in patients who have chosen comfort measures only, or in patients with GI dysfunction (including conditions such as small bowel obstruction, colitis, pancreatitis), we excluded these populations from the study sample. Patients admitted to the hospital for less than 24 hours were also excluded from the study, because they would not be at risk for the outcome of potentially inappropriate use. Doses of IV opioids administered for respiratory distress were considered to be appropriate. Given difficulty in identifying the appropriate time to transition from patient-controlled analgesia (PCA) to IV or per os (PO) opioids, days spent receiving opioids by PCA or continuous IV drip were excluded from the analysis.

We used Fisher’s exact test or the Chi-square test (in the setting of a multicategory variable) to calculate bivariable P values. We used multivariable logistic regression to identify independent predictors of receipt of at least one dose of potentially inappropriate IV opioids, using the hospitalization as the unit of analysis.

Of 630 hospitalizations with at least one order for IV opioids over a one-month period, we reviewed 502 charts, from which we excluded 76 hospitalizations with an active cancer diagnosis, 30 with comfort-focused care, 115 with GI dysfunction, and 108 with a hospitalization less than 24 hours in duration, resulting in 200 hospitalizations included in this analysis (some patients met multiple exclusion criteria). Table 1 outlines characteristics of the study population, stratified by appropriateness of IV opioid use. The study population was predominately white and had an average age of 56.3 years. The majority of patients were on a surgical service. Hydromorphone was the most commonly administered opioid. There were significant differences in the percentage of doses considered inappropriate between different types of opioids (P < .001), with morphine having the highest proportion of doses considered potentially inappropriate (Table 2).

Thirty-one percent of the cohort was administered at least one potentially inappropriate dose of IV opioids. A total of 432 of 1,319 (33%) IV doses were considered potentially inappropriate.

Predictors of Potentially Inappropriate Use

No significant associations were observed between potentially inappropriate IV opioid administration and age, sex, or admitting service (Table 1). Patients with an ethnicity described as other, unknown, or declined were less likely to have potentially inappropriate use.

In this cohort of medical and surgical inpatients, we found that almost one-third received at least one potentially inappropriate IV opioid administration during their hospitalization, and one-third of all IV opioid administrations were potentially inappropriate based on current recommendations defining the appropriate use of IV versus oral opioids. Although this is a single-center analysis, to our knowledge, this is the first study to ascertain the rate of potentially inappropriate IV opioid administration in hospitalized patients. Our findings suggest that quality improvement initiatives are necessary to promote more guideline-concordant care in this realm.

Several factors may contribute to overuse. Requests from patients for immediate pain relief may at times drive prescription of the IV formulation. In addition, patients may expect the IV formulation because of precedents from prior interactions with the healthcare system. Both of these situations may be opportunities for patient education about the equivalent bioavailability of oral and IV formulations in patients with a functioning GI tract, as well as the relatively small difference in rate of onset between the two routes of administration (generally 15-20 minutes). When a patient’s pain is well controlled with IV medications, physicians may also fail to recognize the need to transition to PO medications, further prolonging unnecessary use. Finally, in patients with multiple, complex, or deteriorating medical conditions, transitioning to oral opioids may be deprioritized for the sake of addressing more urgent medical concerns.

This study highlights the potential for transitioning more patients to oral opioids, which should be feasible in the inpatient setting, where pain needs can often be anticipated in advance and oral medications can be administered earlier to overcome the short delay in the onset of action between the oral and IV routes. Oral medications also have the advantage of a longer duration of effect, which may provide overall improved pain control. At our institution, a recent shortage of IV opioids (which occurred after the data collection period for this study) and subsequent efforts to limit IV opioid use (via computerized prompts and active pharmacist consultation) resulted in an immediate 50% reduction in the daily number of IV opioid administrations, further supporting our conclusion that there is an opportunity to decrease inappropriate use of IV opioids.

There were no specific patient factors that contributed to potentially inappropriate use. Although the ethnicity category of other/unknown/declined was significantly less likely to receive opioids potentially inappropriately, given the heterogeneity of this group, it is difficult to draw conclusions on the clinical significance of this finding. Morphine was significantly more likely than other opioids to be administered inappropriately.

There are several limitations of this study. Because this was a retrospective review, our criteria for appropriate use may have resulted in some misclassification; as a result, we can comment only on potentially inappropriate use rather than on definitively inappropriate use. We attempted to use a conservative definition of appropriateness by automatically assuming all doses in the first 24 hours of administration to be appropriate, which could have resulted in underestimating potentially inappropriate use. Nonetheless, there may be instances in which a patient had suspected malabsorption that was not captured or a fluctuating ability to receive oral medications within a given 24-hour period (due to nausea, for example), resulting in outcome misclassification. In addition, we did not correlate findings with patient-reported pain scores. Because there is no clearly defined pain threshold at which IV opioids are indicated, we did not believe that would be useful in clarifying appropriate versus inappropriate use. That said, we believe that, most of the time, pain medications should be able to be titrated appropriately within 24 hours to avoid the need for immediate pain relief with IV opioids thereafter. Although there may be instances of patients who have breakthrough pain severe enough to require IV opioids despite adequate titration of oral medications, we believe this is likely to represent a small number of our population that received potentially inappropriate use. It is worth noting that even if we overestimated by 50%, such that the true rate of potentially inappropriate IV administrations is 15%, we believe this would still be a ripe target for quality improvement initiatives, given that tens of millions of hospitalized patients receive opioids each year in the United States.¹⁰ Finally, we were unable to quantify the number of providers involved in decision making for these patients, and the single-center nature and short time frame of the study limit generalizability; our analysis should be replicated at other hospitals.

In conclusion, in this sample of 200 medical and surgical hospitalizations receiving IV opioids at a large academic medical center, we identified potentially inappropriate IV administration in 31%, suggesting potential to improve value through improving prescribing practices.

Disclosures

None of the authors have conflicts to disclose.

Funding

Dr. Herzig is funded by grant number K23AG042459 from the National Institute on Aging and R01HS026215 from the Agency for Healthcare Research and Quality. The manuscript contents are solely the responsibility of the authors and do not necessarily represent the views of the funding organizations.

Recently released guidelines on safe opioid prescribing draw attention to the fact that physicians have the ability to curb the opioid epidemic through better adherence to prescribing guidelines and limiting opioid use when not clinically indicated.^1,2 A consensus statement from the Society of Hospital Medicine includes 16 recommendations for improving the safety of opioid use in hospitalized patients, one of which is to use the oral route of administration whenever possible, reserving intravenous (IV) administration for patients who cannot take food or medications by mouth, patients suspected of gastrointestinal (GI) malabsorption, or when immediate pain control and/or rapid dose titration is necessary.² This recommendation was based on an increased risk of side effects, adverse events, and medication errors with IV compared with oral formulations.^3-5 Furthermore, the reinforcement from opioids is inversely related to the rate of onset of action, and therefore opioids administered by an IV route may be more likely to lead to addiction.^6-8

Choosing oral over IV opioids has several additional advantages. The cost of the IV formulation is more than oral; at our institution, the cost of IV morphine is 2.5-4.6 times greater than oral. Additional costs associated with IV administration include nursing time and equipment. Overall, transitioning patients from IV to oral medications could considerably lower costs of care.⁹ Ongoing need for an IV line may also lead to avoidable complications, including patient discomfort, infection, and thrombophlebitis. In addition, the recent national shortage of IV opioids has necessitated better stewardship of IV opioids.

Despite this recommendation, our observations suggest that patients often continue receiving IV opioids longer than clinically indicated. The goal of this study was to identify the incidence of potentially inappropriate IV opioid use in hospitalized patients.

The present study was an observational study seeking to quantify the burden of potentially inappropriate IV opioid use and characteristics predicting potentially inappropriate use in the inpatient setting at a large academic medical center in Boston, Massachusetts, using retrospective review of medical records.

Definition of Potentially Inappropriate Use and Study Sample

We identified all hospitalizations during the month of February 2017 with any order for IV opioids using pharmacy charge data and performed chart reviews in this sample until we reached our prespecified study sample of 200 hospitalizations meeting inclusion/exclusion criteria further defined below.

We defined potentially inappropriate use of IV opioids as use of IV opioids for greater than 24 hours in a patient who could receive oral medications (evidenced by receipt of other orally administered medications during the same 24-hour period) and was not mechanically ventilated. This definition is consistent with recommendations in the recently released consensus statement from the Society of Hospital Medicine.² We selected a time frame of 24 hours because IV pain medications may be indicated for initial immediate pain control and rapid dose titration; however, 24 hours should be sufficient time to determine opioid needs and transition to an oral regimen in patients without contraindications. After an initial IV dose, additional IV doses within 24 hours were considered appropriate, whereas IV doses thereafter were considered potentially inappropriate unless the patient had nil per os status, including medications. All IV opioids administered within 24 hours of a surgery or procedure were considered appropriate. Because it may be appropriate to continue IV opioids beyond 24 hours in patients with an active cancer diagnosis, in patients who have chosen comfort measures only, or in patients with GI dysfunction (including conditions such as small bowel obstruction, colitis, pancreatitis), we excluded these populations from the study sample. Patients admitted to the hospital for less than 24 hours were also excluded from the study, because they would not be at risk for the outcome of potentially inappropriate use. Doses of IV opioids administered for respiratory distress were considered to be appropriate. Given difficulty in identifying the appropriate time to transition from patient-controlled analgesia (PCA) to IV or per os (PO) opioids, days spent receiving opioids by PCA or continuous IV drip were excluded from the analysis.

We used Fisher’s exact test or the Chi-square test (in the setting of a multicategory variable) to calculate bivariable P values. We used multivariable logistic regression to identify independent predictors of receipt of at least one dose of potentially inappropriate IV opioids, using the hospitalization as the unit of analysis.

Of 630 hospitalizations with at least one order for IV opioids over a one-month period, we reviewed 502 charts, from which we excluded 76 hospitalizations with an active cancer diagnosis, 30 with comfort-focused care, 115 with GI dysfunction, and 108 with a hospitalization less than 24 hours in duration, resulting in 200 hospitalizations included in this analysis (some patients met multiple exclusion criteria). Table 1 outlines characteristics of the study population, stratified by appropriateness of IV opioid use. The study population was predominately white and had an average age of 56.3 years. The majority of patients were on a surgical service. Hydromorphone was the most commonly administered opioid. There were significant differences in the percentage of doses considered inappropriate between different types of opioids (P < .001), with morphine having the highest proportion of doses considered potentially inappropriate (Table 2).

Thirty-one percent of the cohort was administered at least one potentially inappropriate dose of IV opioids. A total of 432 of 1,319 (33%) IV doses were considered potentially inappropriate.

Predictors of Potentially Inappropriate Use

No significant associations were observed between potentially inappropriate IV opioid administration and age, sex, or admitting service (Table 1). Patients with an ethnicity described as other, unknown, or declined were less likely to have potentially inappropriate use.

In this cohort of medical and surgical inpatients, we found that almost one-third received at least one potentially inappropriate IV opioid administration during their hospitalization, and one-third of all IV opioid administrations were potentially inappropriate based on current recommendations defining the appropriate use of IV versus oral opioids. Although this is a single-center analysis, to our knowledge, this is the first study to ascertain the rate of potentially inappropriate IV opioid administration in hospitalized patients. Our findings suggest that quality improvement initiatives are necessary to promote more guideline-concordant care in this realm.

Several factors may contribute to overuse. Requests from patients for immediate pain relief may at times drive prescription of the IV formulation. In addition, patients may expect the IV formulation because of precedents from prior interactions with the healthcare system. Both of these situations may be opportunities for patient education about the equivalent bioavailability of oral and IV formulations in patients with a functioning GI tract, as well as the relatively small difference in rate of onset between the two routes of administration (generally 15-20 minutes). When a patient’s pain is well controlled with IV medications, physicians may also fail to recognize the need to transition to PO medications, further prolonging unnecessary use. Finally, in patients with multiple, complex, or deteriorating medical conditions, transitioning to oral opioids may be deprioritized for the sake of addressing more urgent medical concerns.

This study highlights the potential for transitioning more patients to oral opioids, which should be feasible in the inpatient setting, where pain needs can often be anticipated in advance and oral medications can be administered earlier to overcome the short delay in the onset of action between the oral and IV routes. Oral medications also have the advantage of a longer duration of effect, which may provide overall improved pain control. At our institution, a recent shortage of IV opioids (which occurred after the data collection period for this study) and subsequent efforts to limit IV opioid use (via computerized prompts and active pharmacist consultation) resulted in an immediate 50% reduction in the daily number of IV opioid administrations, further supporting our conclusion that there is an opportunity to decrease inappropriate use of IV opioids.

There were no specific patient factors that contributed to potentially inappropriate use. Although the ethnicity category of other/unknown/declined was significantly less likely to receive opioids potentially inappropriately, given the heterogeneity of this group, it is difficult to draw conclusions on the clinical significance of this finding. Morphine was significantly more likely than other opioids to be administered inappropriately.

There are several limitations of this study. Because this was a retrospective review, our criteria for appropriate use may have resulted in some misclassification; as a result, we can comment only on potentially inappropriate use rather than on definitively inappropriate use. We attempted to use a conservative definition of appropriateness by automatically assuming all doses in the first 24 hours of administration to be appropriate, which could have resulted in underestimating potentially inappropriate use. Nonetheless, there may be instances in which a patient had suspected malabsorption that was not captured or a fluctuating ability to receive oral medications within a given 24-hour period (due to nausea, for example), resulting in outcome misclassification. In addition, we did not correlate findings with patient-reported pain scores. Because there is no clearly defined pain threshold at which IV opioids are indicated, we did not believe that would be useful in clarifying appropriate versus inappropriate use. That said, we believe that, most of the time, pain medications should be able to be titrated appropriately within 24 hours to avoid the need for immediate pain relief with IV opioids thereafter. Although there may be instances of patients who have breakthrough pain severe enough to require IV opioids despite adequate titration of oral medications, we believe this is likely to represent a small number of our population that received potentially inappropriate use. It is worth noting that even if we overestimated by 50%, such that the true rate of potentially inappropriate IV administrations is 15%, we believe this would still be a ripe target for quality improvement initiatives, given that tens of millions of hospitalized patients receive opioids each year in the United States.¹⁰ Finally, we were unable to quantify the number of providers involved in decision making for these patients, and the single-center nature and short time frame of the study limit generalizability; our analysis should be replicated at other hospitals.

In conclusion, in this sample of 200 medical and surgical hospitalizations receiving IV opioids at a large academic medical center, we identified potentially inappropriate IV administration in 31%, suggesting potential to improve value through improving prescribing practices.

Disclosures

None of the authors have conflicts to disclose.

Funding

Dr. Herzig is funded by grant number K23AG042459 from the National Institute on Aging and R01HS026215 from the Agency for Healthcare Research and Quality. The manuscript contents are solely the responsibility of the authors and do not necessarily represent the views of the funding organizations.

Prior healthcare utilization predicts future utilization;¹ thus, providers should know when a child has had a recent healthcare visit. Healthcare providers typically obtain this information from parents and caregivers, who may not always provide accurate information.^2-4

The Hospital to Home Outcomes study (H2O) was a randomized controlled trial conducted to assess the effects of a one-time home nurse visit following discharge on unplanned healthcare reutilization.⁵ We assessed reutilization through two sources: parent report via a postdischarge telephone call and administrative data. In this analysis, we sought to understand differences in reutilization rates by source by comparing parent report with administrative data.

The H2O trial included children (<18 years) hospitalized on the hospital medicine (HM) or neuroscience (Neurology/Neurosurgery) services at Cincinnati Children’s Hospital Medical Center (CCHMC) from February 2015 to April 2016; they had an English-speaking parent and were discharged to home without skilled nursing care.⁶ For this analysis, we restricted the sample to children randomized to the control arm (discharge without a home visit), which reflects typical clinical care.

We used administrative data to capture 14-day reutilization (unplanned hospital readmissions, emergency department [ED] visits, or urgent care visits). CCHMC is the only pediatric admitting facility in the region and includes two pediatric EDs and five urgent care centers. We supplemented hospital data with a dataset (The Health Collaborative⁷) that included utilization at other regional facilities. Parent report was assessed via a research coordinator phone call 14-23 days after discharge. Parents were asked: “I’m going to [ask] about your child’s health since [discharge date]. Has s/he been hospitalized overnight? Has s/he been taken to the Emergency Room/Emergency Department (didn’t stay overnight)? Has s/he been taken to an urgent care?” We report 14-day reutilization rates by source (parent and/or administrative) and visit type.

We considered administrative data the gold standard for documentation of reutilization events for two reasons. First, all healthcare encounters generate billing and are therefore documented with verifiable coding. Second, we had access to data from our center and other regional healthcare facilities. Any parent-reported utilization to a facility not documented in either dataset was considered an unverifiable event (eg, outside our catchment region). Agreement between administrative and parent report of 14-day reutilization was summarized as positive agreement (reutilization documented in both administrative and parent report), negative agreement (no reutilization reported in either administrative or parent report), and overall agreement (combination of positive and negative agreement). We classified discrepancies as reutilization events in administrative data without parent report of reutilization or vice versa. We performed medical record review of discrepancies in our institutional data.

We summarized agreement by using the Cohen’s kappa statistic by reuse type (hospital readmission, ED, and urgent care visit) and overall (any reutilization event). Strength of agreement based on the kappa statistics was classified as poor (<0.20), fair (0.21-0.40), moderate (0.41-0.60), good (0.61-0.80), and very good (0.81-1.00).⁸ We used McNemar’s test to evaluate marginal homogeneity.

Of 749 children randomized to the standard of care arm, 723 parents completed the 14-day follow-up call and were included in this analysis. The median child age was two years (interquartile range: 0.4, 6.9), the median length of stay (LOS) was two days (1, 3), and the majority were white (62%). Payer mix varied, with 44% privately insured and 54% publicly insured. Most patients (83%) were admitted to the HM service, and the most common diagnoses groups for index admission were respiratory (35%), neurologic (14%), and gastrointestinal (9%) diseases.

Administrative data showed 63 children with any reutilization event; parents reported 63 with any reutilization event; 48 children had events reported by both sources. The overall agreement was high, ranging from 95.9% to 98.5% (Table 1) depending on visit type. The positive agreement (ie, parent and administrative data indicated reutilization) ranged from 47.6% to 76.2%. Negative agreement (ie, parent and administrative data agreed no reutilization) was very high, 97.7% to 99.2%. Parents reported three ED visits and four urgent care visits that were unverifiable due to lack of access to administrative data (sites of care reported were not included in our datasets).

By comparing parent report of reutilization after hospital discharge through postdischarge phone calls with administrative data, we demonstrated high overall agreement between sources (95.9%); this finding is similar to prior research investigating the relationship between an established medical home and reutilization.⁹ However, this agreement is largely due to both sources reporting no reutilization. When revisits did occur, the agreement was notably lower, especially with regard to urgent care visits.

Discrepancies between sources have several possible explanations. First, parents may be confused by the framing of reutilization questions, perhaps lacking clarity around which visit we were referencing. Second, parents may experience limitations in health literacy^10,11 with a lack of familiarity with healthcare language, such as the ability to delineate location types (for example, a parent may identify an urgent care visit as an ED visit, given their close proximity at our facility). Finally, our prior work identified that the “fog” of hospitalization,¹² which is often a stressful and disruptive time for families, may linger after admission and could lead to difficulty in recalling detailed events.

Our findings have implications for effective care in a complex healthcare system where parent report may be the most practical method to obtain historical information, both within clinical care and in the context of research or quality measures, such as postdischarge utilization. Given that one of the greatest risk factors for readmission is prior utilization,¹ the knowledge that a patient experienced a reutilization after a prior discharge might prompt the inpatient provider to better prepare families for subsequent transition to home.

To apply our findings practically, it is important to realize that a parent report may be sufficient when reporting that no revisit occurred, if there is also no record of a visit in accessible administrative data (such as an electronic health record). However, further questions or investigation should be considered when parents report a visit did occur or when administrative data indicate a visit occurred that the parent does not recall. Providers and researchers alike should remember to use health literacy universal precautions with all families, employing plain language without medical jargon.¹³ As linked electronic health record use becomes more prevalent, administrative data may be accessible in real-time, allowing for verification of family interview information. Administrative data beyond a single hospital system should be considered to effectively capture reutilization for research or quality efforts.

Our study has several limitations. Similar to most studies using reutilization outcomes, our data may miss a few unverifiable reuse events. By supplementing with additional regional data,⁷ we likely captured most events. Second, we did not include patients with limited English proficiency, although it is unclear how this might have biased our results. Third, while relatively few families did not complete the calls, it is possible that more discrepancies would have been noted in nonresponders. Fourth, research coordinators administering the calls followed a script to determine reutilization information; in clinical practice, a practitioner might not ask questions as clearly, which could negatively impact recall or might add clarifying follow-up questions to enhance recall. Finally, the analysis occurred in the setting of a randomized controlled trial that included children with relatively noncomplex health conditions with short LOS;⁶ thus, the results may not apply to other populations.

In conclusion, parent report and administrative data of reutilization following hospital discharge were usually in agreement when no reutilization occurred; however, discrepancies were noted more often when reutilizations occurred and may have care implications.

Collaborators

On behalf of the H2O Trial study group including: Joanne Bachus, BSN, RN; Andrew F. Beck, MD, MPH; Monica L. Borell, BSN, RN; Lenisa V. Chang, MA, PhD; Patricia Crawford, RN; Jennifer M. Gold, MSN, RN; Judy A. Heilman BSN, RN; Jane C. Khoury, PhD; Pierce Kuhnell, MS; Karen Lawley, BSN, RN; Allison Loechtenfeldt, BS; Colleen Mangeot, MS; Lynn O’Donnell, BSN, RN; Rita H. Pickler, PhD, RN; Hadley S. Sauers-Ford, MPH; Anita N. Shah, DO, MPH; Susan N. Sherman, DPA; Lauren G. Solan, MD, MEd; Karen P. Sullivan, BSN, RN; Susan Wade-Murphy, MSN, RN

Disclosures

Hospital to Home Outcomes team reports grants from the Patient Centered Outcomes Research Institute during the conduct of the study. Dr. White reports personal fees from the Institute for Health Care Improvement, outside the submitted work.

Funding

This work was supported by the Patient Centered Outcomes Research Institute (IHS-1306-0081 to Dr. S. Shah). All statements in this report, including findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute, its Board of Governors, or the Methodology Committee. Dr Auger’s research is funded by the Agency for Healthcare Research and Quality (1K08HS024735).

Prior healthcare utilization predicts future utilization;¹ thus, providers should know when a child has had a recent healthcare visit. Healthcare providers typically obtain this information from parents and caregivers, who may not always provide accurate information.^2-4

The Hospital to Home Outcomes study (H2O) was a randomized controlled trial conducted to assess the effects of a one-time home nurse visit following discharge on unplanned healthcare reutilization.⁵ We assessed reutilization through two sources: parent report via a postdischarge telephone call and administrative data. In this analysis, we sought to understand differences in reutilization rates by source by comparing parent report with administrative data.

The H2O trial included children (<18 years) hospitalized on the hospital medicine (HM) or neuroscience (Neurology/Neurosurgery) services at Cincinnati Children’s Hospital Medical Center (CCHMC) from February 2015 to April 2016; they had an English-speaking parent and were discharged to home without skilled nursing care.⁶ For this analysis, we restricted the sample to children randomized to the control arm (discharge without a home visit), which reflects typical clinical care.

We used administrative data to capture 14-day reutilization (unplanned hospital readmissions, emergency department [ED] visits, or urgent care visits). CCHMC is the only pediatric admitting facility in the region and includes two pediatric EDs and five urgent care centers. We supplemented hospital data with a dataset (The Health Collaborative⁷) that included utilization at other regional facilities. Parent report was assessed via a research coordinator phone call 14-23 days after discharge. Parents were asked: “I’m going to [ask] about your child’s health since [discharge date]. Has s/he been hospitalized overnight? Has s/he been taken to the Emergency Room/Emergency Department (didn’t stay overnight)? Has s/he been taken to an urgent care?” We report 14-day reutilization rates by source (parent and/or administrative) and visit type.

We considered administrative data the gold standard for documentation of reutilization events for two reasons. First, all healthcare encounters generate billing and are therefore documented with verifiable coding. Second, we had access to data from our center and other regional healthcare facilities. Any parent-reported utilization to a facility not documented in either dataset was considered an unverifiable event (eg, outside our catchment region). Agreement between administrative and parent report of 14-day reutilization was summarized as positive agreement (reutilization documented in both administrative and parent report), negative agreement (no reutilization reported in either administrative or parent report), and overall agreement (combination of positive and negative agreement). We classified discrepancies as reutilization events in administrative data without parent report of reutilization or vice versa. We performed medical record review of discrepancies in our institutional data.

We summarized agreement by using the Cohen’s kappa statistic by reuse type (hospital readmission, ED, and urgent care visit) and overall (any reutilization event). Strength of agreement based on the kappa statistics was classified as poor (<0.20), fair (0.21-0.40), moderate (0.41-0.60), good (0.61-0.80), and very good (0.81-1.00).⁸ We used McNemar’s test to evaluate marginal homogeneity.

Of 749 children randomized to the standard of care arm, 723 parents completed the 14-day follow-up call and were included in this analysis. The median child age was two years (interquartile range: 0.4, 6.9), the median length of stay (LOS) was two days (1, 3), and the majority were white (62%). Payer mix varied, with 44% privately insured and 54% publicly insured. Most patients (83%) were admitted to the HM service, and the most common diagnoses groups for index admission were respiratory (35%), neurologic (14%), and gastrointestinal (9%) diseases.

Administrative data showed 63 children with any reutilization event; parents reported 63 with any reutilization event; 48 children had events reported by both sources. The overall agreement was high, ranging from 95.9% to 98.5% (Table 1) depending on visit type. The positive agreement (ie, parent and administrative data indicated reutilization) ranged from 47.6% to 76.2%. Negative agreement (ie, parent and administrative data agreed no reutilization) was very high, 97.7% to 99.2%. Parents reported three ED visits and four urgent care visits that were unverifiable due to lack of access to administrative data (sites of care reported were not included in our datasets).

By comparing parent report of reutilization after hospital discharge through postdischarge phone calls with administrative data, we demonstrated high overall agreement between sources (95.9%); this finding is similar to prior research investigating the relationship between an established medical home and reutilization.⁹ However, this agreement is largely due to both sources reporting no reutilization. When revisits did occur, the agreement was notably lower, especially with regard to urgent care visits.

Discrepancies between sources have several possible explanations. First, parents may be confused by the framing of reutilization questions, perhaps lacking clarity around which visit we were referencing. Second, parents may experience limitations in health literacy^10,11 with a lack of familiarity with healthcare language, such as the ability to delineate location types (for example, a parent may identify an urgent care visit as an ED visit, given their close proximity at our facility). Finally, our prior work identified that the “fog” of hospitalization,¹² which is often a stressful and disruptive time for families, may linger after admission and could lead to difficulty in recalling detailed events.

Our findings have implications for effective care in a complex healthcare system where parent report may be the most practical method to obtain historical information, both within clinical care and in the context of research or quality measures, such as postdischarge utilization. Given that one of the greatest risk factors for readmission is prior utilization,¹ the knowledge that a patient experienced a reutilization after a prior discharge might prompt the inpatient provider to better prepare families for subsequent transition to home.

To apply our findings practically, it is important to realize that a parent report may be sufficient when reporting that no revisit occurred, if there is also no record of a visit in accessible administrative data (such as an electronic health record). However, further questions or investigation should be considered when parents report a visit did occur or when administrative data indicate a visit occurred that the parent does not recall. Providers and researchers alike should remember to use health literacy universal precautions with all families, employing plain language without medical jargon.¹³ As linked electronic health record use becomes more prevalent, administrative data may be accessible in real-time, allowing for verification of family interview information. Administrative data beyond a single hospital system should be considered to effectively capture reutilization for research or quality efforts.

Our study has several limitations. Similar to most studies using reutilization outcomes, our data may miss a few unverifiable reuse events. By supplementing with additional regional data,⁷ we likely captured most events. Second, we did not include patients with limited English proficiency, although it is unclear how this might have biased our results. Third, while relatively few families did not complete the calls, it is possible that more discrepancies would have been noted in nonresponders. Fourth, research coordinators administering the calls followed a script to determine reutilization information; in clinical practice, a practitioner might not ask questions as clearly, which could negatively impact recall or might add clarifying follow-up questions to enhance recall. Finally, the analysis occurred in the setting of a randomized controlled trial that included children with relatively noncomplex health conditions with short LOS;⁶ thus, the results may not apply to other populations.

In conclusion, parent report and administrative data of reutilization following hospital discharge were usually in agreement when no reutilization occurred; however, discrepancies were noted more often when reutilizations occurred and may have care implications.

Collaborators

On behalf of the H2O Trial study group including: Joanne Bachus, BSN, RN; Andrew F. Beck, MD, MPH; Monica L. Borell, BSN, RN; Lenisa V. Chang, MA, PhD; Patricia Crawford, RN; Jennifer M. Gold, MSN, RN; Judy A. Heilman BSN, RN; Jane C. Khoury, PhD; Pierce Kuhnell, MS; Karen Lawley, BSN, RN; Allison Loechtenfeldt, BS; Colleen Mangeot, MS; Lynn O’Donnell, BSN, RN; Rita H. Pickler, PhD, RN; Hadley S. Sauers-Ford, MPH; Anita N. Shah, DO, MPH; Susan N. Sherman, DPA; Lauren G. Solan, MD, MEd; Karen P. Sullivan, BSN, RN; Susan Wade-Murphy, MSN, RN

Disclosures

Hospital to Home Outcomes team reports grants from the Patient Centered Outcomes Research Institute during the conduct of the study. Dr. White reports personal fees from the Institute for Health Care Improvement, outside the submitted work.

Funding

This work was supported by the Patient Centered Outcomes Research Institute (IHS-1306-0081 to Dr. S. Shah). All statements in this report, including findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute, its Board of Governors, or the Methodology Committee. Dr Auger’s research is funded by the Agency for Healthcare Research and Quality (1K08HS024735).

Prior healthcare utilization predicts future utilization;¹ thus, providers should know when a child has had a recent healthcare visit. Healthcare providers typically obtain this information from parents and caregivers, who may not always provide accurate information.^2-4

The Hospital to Home Outcomes study (H2O) was a randomized controlled trial conducted to assess the effects of a one-time home nurse visit following discharge on unplanned healthcare reutilization.⁵ We assessed reutilization through two sources: parent report via a postdischarge telephone call and administrative data. In this analysis, we sought to understand differences in reutilization rates by source by comparing parent report with administrative data.

The H2O trial included children (<18 years) hospitalized on the hospital medicine (HM) or neuroscience (Neurology/Neurosurgery) services at Cincinnati Children’s Hospital Medical Center (CCHMC) from February 2015 to April 2016; they had an English-speaking parent and were discharged to home without skilled nursing care.⁶ For this analysis, we restricted the sample to children randomized to the control arm (discharge without a home visit), which reflects typical clinical care.

We used administrative data to capture 14-day reutilization (unplanned hospital readmissions, emergency department [ED] visits, or urgent care visits). CCHMC is the only pediatric admitting facility in the region and includes two pediatric EDs and five urgent care centers. We supplemented hospital data with a dataset (The Health Collaborative⁷) that included utilization at other regional facilities. Parent report was assessed via a research coordinator phone call 14-23 days after discharge. Parents were asked: “I’m going to [ask] about your child’s health since [discharge date]. Has s/he been hospitalized overnight? Has s/he been taken to the Emergency Room/Emergency Department (didn’t stay overnight)? Has s/he been taken to an urgent care?” We report 14-day reutilization rates by source (parent and/or administrative) and visit type.

We considered administrative data the gold standard for documentation of reutilization events for two reasons. First, all healthcare encounters generate billing and are therefore documented with verifiable coding. Second, we had access to data from our center and other regional healthcare facilities. Any parent-reported utilization to a facility not documented in either dataset was considered an unverifiable event (eg, outside our catchment region). Agreement between administrative and parent report of 14-day reutilization was summarized as positive agreement (reutilization documented in both administrative and parent report), negative agreement (no reutilization reported in either administrative or parent report), and overall agreement (combination of positive and negative agreement). We classified discrepancies as reutilization events in administrative data without parent report of reutilization or vice versa. We performed medical record review of discrepancies in our institutional data.

We summarized agreement by using the Cohen’s kappa statistic by reuse type (hospital readmission, ED, and urgent care visit) and overall (any reutilization event). Strength of agreement based on the kappa statistics was classified as poor (<0.20), fair (0.21-0.40), moderate (0.41-0.60), good (0.61-0.80), and very good (0.81-1.00).⁸ We used McNemar’s test to evaluate marginal homogeneity.

Of 749 children randomized to the standard of care arm, 723 parents completed the 14-day follow-up call and were included in this analysis. The median child age was two years (interquartile range: 0.4, 6.9), the median length of stay (LOS) was two days (1, 3), and the majority were white (62%). Payer mix varied, with 44% privately insured and 54% publicly insured. Most patients (83%) were admitted to the HM service, and the most common diagnoses groups for index admission were respiratory (35%), neurologic (14%), and gastrointestinal (9%) diseases.

Administrative data showed 63 children with any reutilization event; parents reported 63 with any reutilization event; 48 children had events reported by both sources. The overall agreement was high, ranging from 95.9% to 98.5% (Table 1) depending on visit type. The positive agreement (ie, parent and administrative data indicated reutilization) ranged from 47.6% to 76.2%. Negative agreement (ie, parent and administrative data agreed no reutilization) was very high, 97.7% to 99.2%. Parents reported three ED visits and four urgent care visits that were unverifiable due to lack of access to administrative data (sites of care reported were not included in our datasets).

By comparing parent report of reutilization after hospital discharge through postdischarge phone calls with administrative data, we demonstrated high overall agreement between sources (95.9%); this finding is similar to prior research investigating the relationship between an established medical home and reutilization.⁹ However, this agreement is largely due to both sources reporting no reutilization. When revisits did occur, the agreement was notably lower, especially with regard to urgent care visits.

Discrepancies between sources have several possible explanations. First, parents may be confused by the framing of reutilization questions, perhaps lacking clarity around which visit we were referencing. Second, parents may experience limitations in health literacy^10,11 with a lack of familiarity with healthcare language, such as the ability to delineate location types (for example, a parent may identify an urgent care visit as an ED visit, given their close proximity at our facility). Finally, our prior work identified that the “fog” of hospitalization,¹² which is often a stressful and disruptive time for families, may linger after admission and could lead to difficulty in recalling detailed events.

Our findings have implications for effective care in a complex healthcare system where parent report may be the most practical method to obtain historical information, both within clinical care and in the context of research or quality measures, such as postdischarge utilization. Given that one of the greatest risk factors for readmission is prior utilization,¹ the knowledge that a patient experienced a reutilization after a prior discharge might prompt the inpatient provider to better prepare families for subsequent transition to home.

To apply our findings practically, it is important to realize that a parent report may be sufficient when reporting that no revisit occurred, if there is also no record of a visit in accessible administrative data (such as an electronic health record). However, further questions or investigation should be considered when parents report a visit did occur or when administrative data indicate a visit occurred that the parent does not recall. Providers and researchers alike should remember to use health literacy universal precautions with all families, employing plain language without medical jargon.¹³ As linked electronic health record use becomes more prevalent, administrative data may be accessible in real-time, allowing for verification of family interview information. Administrative data beyond a single hospital system should be considered to effectively capture reutilization for research or quality efforts.

Our study has several limitations. Similar to most studies using reutilization outcomes, our data may miss a few unverifiable reuse events. By supplementing with additional regional data,⁷ we likely captured most events. Second, we did not include patients with limited English proficiency, although it is unclear how this might have biased our results. Third, while relatively few families did not complete the calls, it is possible that more discrepancies would have been noted in nonresponders. Fourth, research coordinators administering the calls followed a script to determine reutilization information; in clinical practice, a practitioner might not ask questions as clearly, which could negatively impact recall or might add clarifying follow-up questions to enhance recall. Finally, the analysis occurred in the setting of a randomized controlled trial that included children with relatively noncomplex health conditions with short LOS;⁶ thus, the results may not apply to other populations.

In conclusion, parent report and administrative data of reutilization following hospital discharge were usually in agreement when no reutilization occurred; however, discrepancies were noted more often when reutilizations occurred and may have care implications.

Collaborators

On behalf of the H2O Trial study group including: Joanne Bachus, BSN, RN; Andrew F. Beck, MD, MPH; Monica L. Borell, BSN, RN; Lenisa V. Chang, MA, PhD; Patricia Crawford, RN; Jennifer M. Gold, MSN, RN; Judy A. Heilman BSN, RN; Jane C. Khoury, PhD; Pierce Kuhnell, MS; Karen Lawley, BSN, RN; Allison Loechtenfeldt, BS; Colleen Mangeot, MS; Lynn O’Donnell, BSN, RN; Rita H. Pickler, PhD, RN; Hadley S. Sauers-Ford, MPH; Anita N. Shah, DO, MPH; Susan N. Sherman, DPA; Lauren G. Solan, MD, MEd; Karen P. Sullivan, BSN, RN; Susan Wade-Murphy, MSN, RN

Disclosures

Hospital to Home Outcomes team reports grants from the Patient Centered Outcomes Research Institute during the conduct of the study. Dr. White reports personal fees from the Institute for Health Care Improvement, outside the submitted work.

Funding

This work was supported by the Patient Centered Outcomes Research Institute (IHS-1306-0081 to Dr. S. Shah). All statements in this report, including findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute, its Board of Governors, or the Methodology Committee. Dr Auger’s research is funded by the Agency for Healthcare Research and Quality (1K08HS024735).

Hospital readmission rates are a common metric for defining, evaluating, and benchmarking quality of care. The Centers for Medicare and Medicaid Services (CMS) publicly report hospital readmission rates for common adult conditions and reduces payments to hospitals with excessive readmissions.¹ Recently, the focus on pediatric readmission rates has increased and the National Quality Forum (NQF) has endorsed at least two pediatric readmission-specific quality indicators which could be used by public and private payers in pay-for-performance programs aimed at institutions caring for children.² While preventability of readmissions and their value as a marker of quality remains debated, their acceptance by the NQF and CMS has led public and private payers to propose readmission-related penalties for hospitals caring for children. ^3-5

All-cause 30-day same-hospital readmission rates for pediatric conditions are half of the adult readmission rates, around 6% in most studies, compared to 12% in adults.^6,7 The lower rates of pediatric readmissions makes it difficult to only use mean readmission rates to stratify hospitals into high- or low-performers and set target goals for improvement.⁸ While adult readmissions have been studied in depth, there are no consistent measures used to benchmark pediatric readmissions across hospital types.

Given the emphasis placed on readmissions, it is essential to understand patterns in pediatric readmission rates to determine optimal and achievable targets for improvement. Achievable Benchmarks of Care (ABCs) are one approach to understanding readmission rates and have an advantage over using mean or medians in performance improvement as they can stratify performance for conditions with low readmission rates and low volumes.⁹ When creating benchmarks, it is important that hospitals performance is evaluated among peer hospitals with similar patient populations, not just a cumulative average from all hospital types which may punish hospitals with a more complex patient case mix.¹⁰ The goal of this study was to calculate the readmission rates and the ABCs for common pediatric diagnoses by hospital type to identify priority conditions for quality improvement efforts using a previously published methodology.^11-13

Data Source

We conducted a retrospective analysis of patients less than 18 years of age in the Healthcare Utilization Project 2014 Nationwide Readmissions Database (NRD). The NRD includes public hospitals; academic medical centers; and specialty hospitals in obstetrics and gynecology, otolaryngology, orthopedics, and cancer; and pediatric, public, and academic medical hospitals. Excluded are long-term care facilities such as rehabilitation, long-term acute care, psychiatric, alcoholism, and chemical dependency hospitals. The readmissions data contains information from hospitals grouped by region, population census, and teaching status.¹⁴ Three hospital type classifications used in this study were metropolitan teaching hospitals, metropolitan nonteaching hospitals, and nonmetropolitan hospitals. These three hospital type classifications follow the reporting format in the NRD.

Study Population

Patients less than 18 years old were included if they were discharged from January 1, 2014 through November 30, 2014 and had a readmission to the index hospital within 30 days. We limited inclusion to discharges through November 30 so we could identify patients with a 30-day readmission as patient identifiers do not link across years in the NRD.

Exposure

We included 30-day, all-cause, same-hospital readmissions to the index acute care hospital, excluding labor and delivery, normal newborn care, chemotherapy, transfers, and mortalities. Intrahospital discharge and admissions within the same hospital system were not defined as a readmission, but rather as a “same-day event.”¹⁵ For example, institutions with inpatient mental health facilities, medical unit discharges and admission to the mental health unit were not identified as a readmission in this dataset.

Outcome

For each hospital type, we measured same-hospital, all-cause, 30-day readmission rates and achievable benchmark of care for the 17 most commonly readmitted pediatric discharge diagnoses. To identify the target readmission diagnoses and all-cause, 30-day readmissions based on their index hospitalizations, All-Patient Refined Diagnosis-Related Groups (APR-DRG), version 25 (3M Health Information Systems, Salt Lake City, Utah) were ordered by frequency for each hospital type. The 20 most common APR-DRGs were the same across all hospital types. The authors then evaluated these 20 APR-DRGs for clinical consistency of included diagnoses identified by the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes within each APR-DRG. Three diagnosis-related groups were excluded from the analysis (major hematologic/immunologic disease except for sickle cell, other anemia and disorders of blood and blood forming organs, and other digestive system diagnoses) due to the heterogeneity of the diagnoses identified by the ICD-9-CM codes within each APR-DRG. We refer to each APR-DRG as a “diagnosis” throughout the article.

Analysis

The demographic characteristics of the patients seen at the three hospital types were summarized using frequencies and percentages. Reports were generated for patient age, gender, payer source, patient residence, median household income, patient complexity, and discharge disposition. Patient complexity was defined using complex chronic condition (CCC) and the number of chronic conditions (CCI).^16,17 As previously defined in the literature, a complex chronic condition is “any medical condition that can be reasonably expected to last at least 12 months (unless death intervenes) and to involve either several different organ systems or one organ system severely enough to require specialty pediatric care and probably some period of hospitalization in a tertiary care center.”¹⁶ Whereas, the Agency for Healthcare Research and Quality’s Chronic Condition Indicator (CCI) defines single, non-CCCs (eg, allergic rhinitis).¹⁷

For each diagnosis, we calculated the mean readmission rate for hospitals in each hospital type category. We then calculated an ABC for each diagnosis in each hospital type using a four-step process.^13,18 First, to control for hospitals with small sample sizes, we adjusted all readmission rates using an adjusted performance fraction ([numerator+1]/[denominator +2]), where the numerator is the number of all-cause 30-day readmissions and the denominator is the number of discharges for the selected diagnosis. Then the hospitals were ordered from lowest (best performing) to highest (worst performing) using the adjusted readmission rate. Third, the number of discharges from the best performing hospital to the worst performing hospital was summed until at least 10% of the total discharges had been accounted for. Finally, we computed the ABC as the average of these best performing hospitals. We only report ABCs for which at least three hospitals were included as best performers in the calculation.¹³

To evaluate hospital performance on ABCs for each diagnosis, we identified the percent of hospitals in each setting that were outliers. We defined an outlier as any hospital whose 95% confidence interval for their readmission rate for a given diagnosis did not contain the ABC for their hospital type. All the statistical analyses were performed using SAS version 9.3 (SAS Institute, Inc, Cary, North Carolina).

This project was reviewed by the Cincinnati Children’s Hospital Medical Center Institutional Review Board and determined to be nonhuman subjects research.

Hospital-Type Demographics

The 690,949 discharges from 1,664 hospitals were categorized into 525 metropolitan teaching (550,039 discharges, 79.6% of discharges), 552 metropolitan nonteaching (97,207 discharges, 14% of discharges), and 587 nonmetropolitan hospitals (43,703 discharges, 6.3% of discharges; Table 1). There were significant differences in the patient composition among the three hospital settings. Nonmetropolitan hospitals had a larger percentage of younger patients (aged 0-4 years, P < .001), prominence of first and second quartile median household income, and fewer medically complex patients (48.3% No CCC/No CCI versus 25.5% metropolitan teaching and 33.7% nonteaching, P < .001). Disposition home was over 96% in all three hospital types; however, the metropolitan teaching had a greater percentage of patients discharged to home health versus metropolitan nonteaching and nonmetropolitan hospitals (2.3% versus 0.5%; P < .001).

Readmission Rates

The 17 most common diagnoses based on the number of all-cause 30-day same-hospital readmissions, were categorized into two surgical, seven acute/infectious, four chronic, and four mental health diagnoses (Table 2). Readmission rates varied based on diagnosis and hospital type (Table 2). Overall, mean readmission rates were low, especially in acute respiratory tract related diseases. For chronic diseases, asthma readmissions were consistently low in all three hospital types, whereas sickle cell disease had the highest readmission rate in all three hospital types.

Achievable Benchmarks of Care by Hospital Type

The diagnoses for which ABC could be calculated across all three hospital types included appendectomy and four acute conditions (bronchiolitis, pneumonia, nonbacterial gastroenteritis, and kidney/urinary tract infections). For these conditions, metropolitan teaching hospitals had a more significant percentage of outlier hospitals compared to metropolitan nonteaching and nonmetropolitan hospitals. The percent of outlier hospitals varied by diagnosis and hospital type (Figure).

Metropolitan Teaching

The readmission ABC was calculated for all 17 diagnoses (Table 2). The ABC ranged from 0.4% in acute kidney and urinary tract infection to 7.0% in sickle cell anemia crisis. Bipolar disorder, major depressive disorders and other psychoses, and sickle cell disease (SCD) had the highest percent of outlier hospitals whose mean readmission rates confidence interval did not contain the ABC; tonsil and adenoid procedures and viral illness had the lowest.¹

Metropolitan Nonteaching

The ABC was calculated for 13 of the 17 diagnoses because ABCs were not calculated when there were fewer than three best practicing hospitals. This was the case for tonsil and adenoid procedures, diabetes, seizures, and depression except for major depressive disorder (Table 2). Seven of the 13 diagnoses had an ABC of 0.0%: viral illness, infections of the upper respiratory tract, bronchiolitis, gastroenteritis, hypovolemia and electrolyte disorders, asthma, and childhood behavioral disorders. Like the findings at the metropolitan teaching hospitals, ABCs were lowest for surgical and acute conditions while bipolar disorder, major depressive disorders and other psychoses, and SCD had the highest percent of outlier hospitals with readmission rates beyond the 95% confidence interval of their hospital type’s ABC.

Nonmetropolitan

There was a sufficient number of best practicing hospitals to calculate the ABC for six of the 17 diagnoses (Table 2). For conditions where readmission ABCs could be calculated, they were low: 0.0% for appendectomy, bronchiolitis, gastroenteritis, and seizure; 0.3% for pneumonia; and 1.3% in kidney and urinary tract disorders. None of the conditions with the highest ABCs in other hospital settings (bipolar disease, sickle cell anemia crisis, and major depressive disorders and other psychoses) could be calculated in this setting. Seizure-related readmissions exhibited the most outlier hospitals yet were less than 5%.¹

Among a nationally representative sample of different hospital types that deliver care to children, we report the mean readmission rates and ABCs for 30-day all-cause, same-hospital readmissions for the most commonly readmitted pediatric diagnoses based on hospital type. Previous studies have shown patient variables such as race, ethnicity, and insurance type influencing readmission rates.^19,20 However, hospital type has also been associated with a higher risk of readmission due to the varying complexity of patients at different hospital types.^21,22 Our analyses provide hospital-type specific national estimates of pediatric readmission ABCs for medical and surgical conditions, many less than 1%. While commonly encountered pediatric conditions like asthma and bronchiolitis had low mean readmission rates and ABCs across all hospital types, the mean rates and ABCs for SCD and mental health disorders were much higher with more hospitals performing far from the ABCs.

Diagnoses with a larger percentage of outlier hospitals may represent a national opportunity to improve care for children. Conditions such as SCD and mental illnesses have the highest percentage of hospitals whose readmission rates fall outside of the ABCs in both metropolitan teaching and metropolitan nonteaching hospitals. Hospital performance on SCD and mental health disorders may not reflect deficits in hospital quality or poor adherence to evidence-based best practices, but rather the complex interplay of factors on various levels from government policy and insurance plans, to patient and family resources, to access and availability of medical and mental health specific care. Most importantly, these diseases may represent a significant opportunity for quality improvementin hospitals across the United States.

Sickle cell disease is predominantly a disease among African-Americans, a demographic risk factor for decreased access to care and limited patient and family resources.^23-26 In previous studies evaluating the disparity in readmission rates for Black children with asthma, socioeconomic variables explained 53% of the observed disparity and readmission rates were inversely related to the childhood opportunity index of the patient’s census tract and positively related with geographic social risk.^27,28 Likewise, with SCD affecting a specific demographic and being a chronic disease, best practice policies need to account for the child’s medical needs and include the patient and family resources to ensure access to care and enhanced case management for chronic disease if we aim to improve performance among the outlier hospitals.

Similarly, barriers to care for children with mental illnesses in the United States need attention.^29,30 While there is a paucity of data on the prevalence of mental health disorders in children, one national report estimates that one in 10 American adolescents have depression.^29,31 The American Academy of Pediatrics has developed a policy statement on mental health competencies and a mental health tool-kit for primary care pediatricians; however, no such guidelines or policy statements exist for hospitalized patients with acute or chronic psychiatric conditions.^32,33 Moreover, hospitals are increasingly facing “boarding” of children with acute psychiatric illness in inpatient units and emergency departments.³⁴ The American Medical Association and the American College of Emergency Physicians have expressed concerns regarding the boarding of children with acute psychiatric illness because nonpsychiatric hospitals do not have adequate resources to evaluate, manage, and place these children who deserve appropriate facilities for further management. Coordinated case management and “bundled” discharge planning in other chronic illnesses have shown benefit in cost reduction and readmission.^35-37 Evidence-based practices around pediatric readmissions in other diagnoses should be explored as possible interventions in these conditions.³⁸

There are several limitations to this study. Our data is limited to one calendar year; therefore, admissions in January do not account for potential readmissions from December of the previous year, as patient identifiers do not link across years in the NRD. We also limited our evaluation to the conventional 30-day readmission window, but recent publications may indicate that readmission windows with different timelines could be a more accurate reflection of medically preventable readmissions versus a reflection of social determinants of health leading to readmissions.²⁴ Newborn index admissions were not an allowable index admission; therefore, we may be underreporting readmissions in the neonatal age group. We also chose to include all-cause readmissions, a conventional method to evaluate readmission within an institution, but which may not reflect the quality of care delivered in the index admission. For example, an asthmatic discharged after an acute exacerbation readmitted for dehydration secondary to gastroenteritis may not reflect a lack of quality in asthma inpatient care. Readmissions were limited to the same hospital; therefore, this study cannot account for readmissions at other institutions, which may cause us to underestimate readmission rates. However, end-users of our findings most likely have access only to their own institution’s data. The inclusion of observation status admissions in the database varies from state to state; therefore, this percent of admissions in the database is unknown.

The use of the ABC methodology has some inherent limitations. One hospital with a significant volume diagnosis and low readmission rate within a hospital type may prohibit the reporting of an ABC if less than three hospitals composed the total of the ‘best performing’ hospitals. This was a significant limitation leading to the exclusion of many ABCs in nonmetropolitan institutions. The limitation of calculating and reporting an ABC then prohibits the calculation of outlier hospitals within a hospital type for a given diagnosis. However, when the ABCs are not available, we do provide the mean readmission rate for the diagnosis within the hospital type. While the hospital groupings by population and teaching status for ABCs provide meaningful comparisons for within each hospital setting, it should be noted that there may be vast differences among hospitals within each type (eg, tertiary children’s hospitals compared to teaching hospitals with a pediatric floor in the metropolitan teaching hospital category).^39,40

As healthcare moves from a fee-for-service model to a population-health centered, value-based model, reduction in readmission rates will be more than a quality measure and will have potential financial implications.⁴¹ In the Medicare fee-for-service patients, the Hospital Readmission Reduction Program (HRRP) penalize hospitals with excess readmissions for acute myocardial infarction, heart failure, and pneumonia. The hospitals subject to penalties in the HRRP had greater reduction in readmission rates in the targeted, and even nontargeted conditions, compared with hospitals not subject to penalties.⁴² Similarly, we believe that our data on low readmission rates and ABCs for conditions such as asthma, bronchiolitis, and appendicitis could represent decades of quality improvement work for the most common pediatric conditions among hospitalized children. Sickle cell disease and mental health problems remain as outliers and merit further attention. To move to a true population-health model, hospitals will need to explore outlier conditions including evaluating patient-level readmission patterns across institutions. This moves readmission from a hospital quality measure to a patient-centric quality measure, and perhaps will provide value to the patient and the healthcare system alike.

The readmission ABCs for the most commonly readmitted pediatric diagnoses are low, regardless of the hospital setting. The highest pediatric readmission rates in SCD, bipolar disorders, and major depressive disorder were lower than the most common adult readmission diagnoses. However, mental health conditions and SCD remain as outliers for pediatric readmissions, burden hospital systems, and perhaps warrant national-level attention. The ABCs stratified by hospital type in this study facilitate comparisons and identify opportunities for population-level interventions to meaningfully improve patient care.

Disclosures

The authors have nothing to disclose.

Hospital readmission rates are a common metric for defining, evaluating, and benchmarking quality of care. The Centers for Medicare and Medicaid Services (CMS) publicly report hospital readmission rates for common adult conditions and reduces payments to hospitals with excessive readmissions.¹ Recently, the focus on pediatric readmission rates has increased and the National Quality Forum (NQF) has endorsed at least two pediatric readmission-specific quality indicators which could be used by public and private payers in pay-for-performance programs aimed at institutions caring for children.² While preventability of readmissions and their value as a marker of quality remains debated, their acceptance by the NQF and CMS has led public and private payers to propose readmission-related penalties for hospitals caring for children. ^3-5

All-cause 30-day same-hospital readmission rates for pediatric conditions are half of the adult readmission rates, around 6% in most studies, compared to 12% in adults.^6,7 The lower rates of pediatric readmissions makes it difficult to only use mean readmission rates to stratify hospitals into high- or low-performers and set target goals for improvement.⁸ While adult readmissions have been studied in depth, there are no consistent measures used to benchmark pediatric readmissions across hospital types.

Given the emphasis placed on readmissions, it is essential to understand patterns in pediatric readmission rates to determine optimal and achievable targets for improvement. Achievable Benchmarks of Care (ABCs) are one approach to understanding readmission rates and have an advantage over using mean or medians in performance improvement as they can stratify performance for conditions with low readmission rates and low volumes.⁹ When creating benchmarks, it is important that hospitals performance is evaluated among peer hospitals with similar patient populations, not just a cumulative average from all hospital types which may punish hospitals with a more complex patient case mix.¹⁰ The goal of this study was to calculate the readmission rates and the ABCs for common pediatric diagnoses by hospital type to identify priority conditions for quality improvement efforts using a previously published methodology.^11-13

Data Source

We conducted a retrospective analysis of patients less than 18 years of age in the Healthcare Utilization Project 2014 Nationwide Readmissions Database (NRD). The NRD includes public hospitals; academic medical centers; and specialty hospitals in obstetrics and gynecology, otolaryngology, orthopedics, and cancer; and pediatric, public, and academic medical hospitals. Excluded are long-term care facilities such as rehabilitation, long-term acute care, psychiatric, alcoholism, and chemical dependency hospitals. The readmissions data contains information from hospitals grouped by region, population census, and teaching status.¹⁴ Three hospital type classifications used in this study were metropolitan teaching hospitals, metropolitan nonteaching hospitals, and nonmetropolitan hospitals. These three hospital type classifications follow the reporting format in the NRD.

Study Population

Patients less than 18 years old were included if they were discharged from January 1, 2014 through November 30, 2014 and had a readmission to the index hospital within 30 days. We limited inclusion to discharges through November 30 so we could identify patients with a 30-day readmission as patient identifiers do not link across years in the NRD.

Exposure

We included 30-day, all-cause, same-hospital readmissions to the index acute care hospital, excluding labor and delivery, normal newborn care, chemotherapy, transfers, and mortalities. Intrahospital discharge and admissions within the same hospital system were not defined as a readmission, but rather as a “same-day event.”¹⁵ For example, institutions with inpatient mental health facilities, medical unit discharges and admission to the mental health unit were not identified as a readmission in this dataset.

Outcome

For each hospital type, we measured same-hospital, all-cause, 30-day readmission rates and achievable benchmark of care for the 17 most commonly readmitted pediatric discharge diagnoses. To identify the target readmission diagnoses and all-cause, 30-day readmissions based on their index hospitalizations, All-Patient Refined Diagnosis-Related Groups (APR-DRG), version 25 (3M Health Information Systems, Salt Lake City, Utah) were ordered by frequency for each hospital type. The 20 most common APR-DRGs were the same across all hospital types. The authors then evaluated these 20 APR-DRGs for clinical consistency of included diagnoses identified by the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes within each APR-DRG. Three diagnosis-related groups were excluded from the analysis (major hematologic/immunologic disease except for sickle cell, other anemia and disorders of blood and blood forming organs, and other digestive system diagnoses) due to the heterogeneity of the diagnoses identified by the ICD-9-CM codes within each APR-DRG. We refer to each APR-DRG as a “diagnosis” throughout the article.

Analysis

The demographic characteristics of the patients seen at the three hospital types were summarized using frequencies and percentages. Reports were generated for patient age, gender, payer source, patient residence, median household income, patient complexity, and discharge disposition. Patient complexity was defined using complex chronic condition (CCC) and the number of chronic conditions (CCI).^16,17 As previously defined in the literature, a complex chronic condition is “any medical condition that can be reasonably expected to last at least 12 months (unless death intervenes) and to involve either several different organ systems or one organ system severely enough to require specialty pediatric care and probably some period of hospitalization in a tertiary care center.”¹⁶ Whereas, the Agency for Healthcare Research and Quality’s Chronic Condition Indicator (CCI) defines single, non-CCCs (eg, allergic rhinitis).¹⁷

For each diagnosis, we calculated the mean readmission rate for hospitals in each hospital type category. We then calculated an ABC for each diagnosis in each hospital type using a four-step process.^13,18 First, to control for hospitals with small sample sizes, we adjusted all readmission rates using an adjusted performance fraction ([numerator+1]/[denominator +2]), where the numerator is the number of all-cause 30-day readmissions and the denominator is the number of discharges for the selected diagnosis. Then the hospitals were ordered from lowest (best performing) to highest (worst performing) using the adjusted readmission rate. Third, the number of discharges from the best performing hospital to the worst performing hospital was summed until at least 10% of the total discharges had been accounted for. Finally, we computed the ABC as the average of these best performing hospitals. We only report ABCs for which at least three hospitals were included as best performers in the calculation.¹³

To evaluate hospital performance on ABCs for each diagnosis, we identified the percent of hospitals in each setting that were outliers. We defined an outlier as any hospital whose 95% confidence interval for their readmission rate for a given diagnosis did not contain the ABC for their hospital type. All the statistical analyses were performed using SAS version 9.3 (SAS Institute, Inc, Cary, North Carolina).

This project was reviewed by the Cincinnati Children’s Hospital Medical Center Institutional Review Board and determined to be nonhuman subjects research.

Hospital-Type Demographics

The 690,949 discharges from 1,664 hospitals were categorized into 525 metropolitan teaching (550,039 discharges, 79.6% of discharges), 552 metropolitan nonteaching (97,207 discharges, 14% of discharges), and 587 nonmetropolitan hospitals (43,703 discharges, 6.3% of discharges; Table 1). There were significant differences in the patient composition among the three hospital settings. Nonmetropolitan hospitals had a larger percentage of younger patients (aged 0-4 years, P < .001), prominence of first and second quartile median household income, and fewer medically complex patients (48.3% No CCC/No CCI versus 25.5% metropolitan teaching and 33.7% nonteaching, P < .001). Disposition home was over 96% in all three hospital types; however, the metropolitan teaching had a greater percentage of patients discharged to home health versus metropolitan nonteaching and nonmetropolitan hospitals (2.3% versus 0.5%; P < .001).

Readmission Rates

The 17 most common diagnoses based on the number of all-cause 30-day same-hospital readmissions, were categorized into two surgical, seven acute/infectious, four chronic, and four mental health diagnoses (Table 2). Readmission rates varied based on diagnosis and hospital type (Table 2). Overall, mean readmission rates were low, especially in acute respiratory tract related diseases. For chronic diseases, asthma readmissions were consistently low in all three hospital types, whereas sickle cell disease had the highest readmission rate in all three hospital types.

Achievable Benchmarks of Care by Hospital Type

The diagnoses for which ABC could be calculated across all three hospital types included appendectomy and four acute conditions (bronchiolitis, pneumonia, nonbacterial gastroenteritis, and kidney/urinary tract infections). For these conditions, metropolitan teaching hospitals had a more significant percentage of outlier hospitals compared to metropolitan nonteaching and nonmetropolitan hospitals. The percent of outlier hospitals varied by diagnosis and hospital type (Figure).

Metropolitan Teaching

The readmission ABC was calculated for all 17 diagnoses (Table 2). The ABC ranged from 0.4% in acute kidney and urinary tract infection to 7.0% in sickle cell anemia crisis. Bipolar disorder, major depressive disorders and other psychoses, and sickle cell disease (SCD) had the highest percent of outlier hospitals whose mean readmission rates confidence interval did not contain the ABC; tonsil and adenoid procedures and viral illness had the lowest.¹

Metropolitan Nonteaching

The ABC was calculated for 13 of the 17 diagnoses because ABCs were not calculated when there were fewer than three best practicing hospitals. This was the case for tonsil and adenoid procedures, diabetes, seizures, and depression except for major depressive disorder (Table 2). Seven of the 13 diagnoses had an ABC of 0.0%: viral illness, infections of the upper respiratory tract, bronchiolitis, gastroenteritis, hypovolemia and electrolyte disorders, asthma, and childhood behavioral disorders. Like the findings at the metropolitan teaching hospitals, ABCs were lowest for surgical and acute conditions while bipolar disorder, major depressive disorders and other psychoses, and SCD had the highest percent of outlier hospitals with readmission rates beyond the 95% confidence interval of their hospital type’s ABC.

Nonmetropolitan

There was a sufficient number of best practicing hospitals to calculate the ABC for six of the 17 diagnoses (Table 2). For conditions where readmission ABCs could be calculated, they were low: 0.0% for appendectomy, bronchiolitis, gastroenteritis, and seizure; 0.3% for pneumonia; and 1.3% in kidney and urinary tract disorders. None of the conditions with the highest ABCs in other hospital settings (bipolar disease, sickle cell anemia crisis, and major depressive disorders and other psychoses) could be calculated in this setting. Seizure-related readmissions exhibited the most outlier hospitals yet were less than 5%.¹

Among a nationally representative sample of different hospital types that deliver care to children, we report the mean readmission rates and ABCs for 30-day all-cause, same-hospital readmissions for the most commonly readmitted pediatric diagnoses based on hospital type. Previous studies have shown patient variables such as race, ethnicity, and insurance type influencing readmission rates.^19,20 However, hospital type has also been associated with a higher risk of readmission due to the varying complexity of patients at different hospital types.^21,22 Our analyses provide hospital-type specific national estimates of pediatric readmission ABCs for medical and surgical conditions, many less than 1%. While commonly encountered pediatric conditions like asthma and bronchiolitis had low mean readmission rates and ABCs across all hospital types, the mean rates and ABCs for SCD and mental health disorders were much higher with more hospitals performing far from the ABCs.

Diagnoses with a larger percentage of outlier hospitals may represent a national opportunity to improve care for children. Conditions such as SCD and mental illnesses have the highest percentage of hospitals whose readmission rates fall outside of the ABCs in both metropolitan teaching and metropolitan nonteaching hospitals. Hospital performance on SCD and mental health disorders may not reflect deficits in hospital quality or poor adherence to evidence-based best practices, but rather the complex interplay of factors on various levels from government policy and insurance plans, to patient and family resources, to access and availability of medical and mental health specific care. Most importantly, these diseases may represent a significant opportunity for quality improvementin hospitals across the United States.

Sickle cell disease is predominantly a disease among African-Americans, a demographic risk factor for decreased access to care and limited patient and family resources.^23-26 In previous studies evaluating the disparity in readmission rates for Black children with asthma, socioeconomic variables explained 53% of the observed disparity and readmission rates were inversely related to the childhood opportunity index of the patient’s census tract and positively related with geographic social risk.^27,28 Likewise, with SCD affecting a specific demographic and being a chronic disease, best practice policies need to account for the child’s medical needs and include the patient and family resources to ensure access to care and enhanced case management for chronic disease if we aim to improve performance among the outlier hospitals.

Similarly, barriers to care for children with mental illnesses in the United States need attention.^29,30 While there is a paucity of data on the prevalence of mental health disorders in children, one national report estimates that one in 10 American adolescents have depression.^29,31 The American Academy of Pediatrics has developed a policy statement on mental health competencies and a mental health tool-kit for primary care pediatricians; however, no such guidelines or policy statements exist for hospitalized patients with acute or chronic psychiatric conditions.^32,33 Moreover, hospitals are increasingly facing “boarding” of children with acute psychiatric illness in inpatient units and emergency departments.³⁴ The American Medical Association and the American College of Emergency Physicians have expressed concerns regarding the boarding of children with acute psychiatric illness because nonpsychiatric hospitals do not have adequate resources to evaluate, manage, and place these children who deserve appropriate facilities for further management. Coordinated case management and “bundled” discharge planning in other chronic illnesses have shown benefit in cost reduction and readmission.^35-37 Evidence-based practices around pediatric readmissions in other diagnoses should be explored as possible interventions in these conditions.³⁸

There are several limitations to this study. Our data is limited to one calendar year; therefore, admissions in January do not account for potential readmissions from December of the previous year, as patient identifiers do not link across years in the NRD. We also limited our evaluation to the conventional 30-day readmission window, but recent publications may indicate that readmission windows with different timelines could be a more accurate reflection of medically preventable readmissions versus a reflection of social determinants of health leading to readmissions.²⁴ Newborn index admissions were not an allowable index admission; therefore, we may be underreporting readmissions in the neonatal age group. We also chose to include all-cause readmissions, a conventional method to evaluate readmission within an institution, but which may not reflect the quality of care delivered in the index admission. For example, an asthmatic discharged after an acute exacerbation readmitted for dehydration secondary to gastroenteritis may not reflect a lack of quality in asthma inpatient care. Readmissions were limited to the same hospital; therefore, this study cannot account for readmissions at other institutions, which may cause us to underestimate readmission rates. However, end-users of our findings most likely have access only to their own institution’s data. The inclusion of observation status admissions in the database varies from state to state; therefore, this percent of admissions in the database is unknown.

The use of the ABC methodology has some inherent limitations. One hospital with a significant volume diagnosis and low readmission rate within a hospital type may prohibit the reporting of an ABC if less than three hospitals composed the total of the ‘best performing’ hospitals. This was a significant limitation leading to the exclusion of many ABCs in nonmetropolitan institutions. The limitation of calculating and reporting an ABC then prohibits the calculation of outlier hospitals within a hospital type for a given diagnosis. However, when the ABCs are not available, we do provide the mean readmission rate for the diagnosis within the hospital type. While the hospital groupings by population and teaching status for ABCs provide meaningful comparisons for within each hospital setting, it should be noted that there may be vast differences among hospitals within each type (eg, tertiary children’s hospitals compared to teaching hospitals with a pediatric floor in the metropolitan teaching hospital category).^39,40

As healthcare moves from a fee-for-service model to a population-health centered, value-based model, reduction in readmission rates will be more than a quality measure and will have potential financial implications.⁴¹ In the Medicare fee-for-service patients, the Hospital Readmission Reduction Program (HRRP) penalize hospitals with excess readmissions for acute myocardial infarction, heart failure, and pneumonia. The hospitals subject to penalties in the HRRP had greater reduction in readmission rates in the targeted, and even nontargeted conditions, compared with hospitals not subject to penalties.⁴² Similarly, we believe that our data on low readmission rates and ABCs for conditions such as asthma, bronchiolitis, and appendicitis could represent decades of quality improvement work for the most common pediatric conditions among hospitalized children. Sickle cell disease and mental health problems remain as outliers and merit further attention. To move to a true population-health model, hospitals will need to explore outlier conditions including evaluating patient-level readmission patterns across institutions. This moves readmission from a hospital quality measure to a patient-centric quality measure, and perhaps will provide value to the patient and the healthcare system alike.

The readmission ABCs for the most commonly readmitted pediatric diagnoses are low, regardless of the hospital setting. The highest pediatric readmission rates in SCD, bipolar disorders, and major depressive disorder were lower than the most common adult readmission diagnoses. However, mental health conditions and SCD remain as outliers for pediatric readmissions, burden hospital systems, and perhaps warrant national-level attention. The ABCs stratified by hospital type in this study facilitate comparisons and identify opportunities for population-level interventions to meaningfully improve patient care.

Disclosures

The authors have nothing to disclose.

Hospital readmission rates are a common metric for defining, evaluating, and benchmarking quality of care. The Centers for Medicare and Medicaid Services (CMS) publicly report hospital readmission rates for common adult conditions and reduces payments to hospitals with excessive readmissions.¹ Recently, the focus on pediatric readmission rates has increased and the National Quality Forum (NQF) has endorsed at least two pediatric readmission-specific quality indicators which could be used by public and private payers in pay-for-performance programs aimed at institutions caring for children.² While preventability of readmissions and their value as a marker of quality remains debated, their acceptance by the NQF and CMS has led public and private payers to propose readmission-related penalties for hospitals caring for children. ^3-5

All-cause 30-day same-hospital readmission rates for pediatric conditions are half of the adult readmission rates, around 6% in most studies, compared to 12% in adults.^6,7 The lower rates of pediatric readmissions makes it difficult to only use mean readmission rates to stratify hospitals into high- or low-performers and set target goals for improvement.⁸ While adult readmissions have been studied in depth, there are no consistent measures used to benchmark pediatric readmissions across hospital types.

Given the emphasis placed on readmissions, it is essential to understand patterns in pediatric readmission rates to determine optimal and achievable targets for improvement. Achievable Benchmarks of Care (ABCs) are one approach to understanding readmission rates and have an advantage over using mean or medians in performance improvement as they can stratify performance for conditions with low readmission rates and low volumes.⁹ When creating benchmarks, it is important that hospitals performance is evaluated among peer hospitals with similar patient populations, not just a cumulative average from all hospital types which may punish hospitals with a more complex patient case mix.¹⁰ The goal of this study was to calculate the readmission rates and the ABCs for common pediatric diagnoses by hospital type to identify priority conditions for quality improvement efforts using a previously published methodology.^11-13

Data Source

We conducted a retrospective analysis of patients less than 18 years of age in the Healthcare Utilization Project 2014 Nationwide Readmissions Database (NRD). The NRD includes public hospitals; academic medical centers; and specialty hospitals in obstetrics and gynecology, otolaryngology, orthopedics, and cancer; and pediatric, public, and academic medical hospitals. Excluded are long-term care facilities such as rehabilitation, long-term acute care, psychiatric, alcoholism, and chemical dependency hospitals. The readmissions data contains information from hospitals grouped by region, population census, and teaching status.¹⁴ Three hospital type classifications used in this study were metropolitan teaching hospitals, metropolitan nonteaching hospitals, and nonmetropolitan hospitals. These three hospital type classifications follow the reporting format in the NRD.

Study Population

Patients less than 18 years old were included if they were discharged from January 1, 2014 through November 30, 2014 and had a readmission to the index hospital within 30 days. We limited inclusion to discharges through November 30 so we could identify patients with a 30-day readmission as patient identifiers do not link across years in the NRD.

Exposure

We included 30-day, all-cause, same-hospital readmissions to the index acute care hospital, excluding labor and delivery, normal newborn care, chemotherapy, transfers, and mortalities. Intrahospital discharge and admissions within the same hospital system were not defined as a readmission, but rather as a “same-day event.”¹⁵ For example, institutions with inpatient mental health facilities, medical unit discharges and admission to the mental health unit were not identified as a readmission in this dataset.

Outcome

For each hospital type, we measured same-hospital, all-cause, 30-day readmission rates and achievable benchmark of care for the 17 most commonly readmitted pediatric discharge diagnoses. To identify the target readmission diagnoses and all-cause, 30-day readmissions based on their index hospitalizations, All-Patient Refined Diagnosis-Related Groups (APR-DRG), version 25 (3M Health Information Systems, Salt Lake City, Utah) were ordered by frequency for each hospital type. The 20 most common APR-DRGs were the same across all hospital types. The authors then evaluated these 20 APR-DRGs for clinical consistency of included diagnoses identified by the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes within each APR-DRG. Three diagnosis-related groups were excluded from the analysis (major hematologic/immunologic disease except for sickle cell, other anemia and disorders of blood and blood forming organs, and other digestive system diagnoses) due to the heterogeneity of the diagnoses identified by the ICD-9-CM codes within each APR-DRG. We refer to each APR-DRG as a “diagnosis” throughout the article.

Analysis

The demographic characteristics of the patients seen at the three hospital types were summarized using frequencies and percentages. Reports were generated for patient age, gender, payer source, patient residence, median household income, patient complexity, and discharge disposition. Patient complexity was defined using complex chronic condition (CCC) and the number of chronic conditions (CCI).^16,17 As previously defined in the literature, a complex chronic condition is “any medical condition that can be reasonably expected to last at least 12 months (unless death intervenes) and to involve either several different organ systems or one organ system severely enough to require specialty pediatric care and probably some period of hospitalization in a tertiary care center.”¹⁶ Whereas, the Agency for Healthcare Research and Quality’s Chronic Condition Indicator (CCI) defines single, non-CCCs (eg, allergic rhinitis).¹⁷

For each diagnosis, we calculated the mean readmission rate for hospitals in each hospital type category. We then calculated an ABC for each diagnosis in each hospital type using a four-step process.^13,18 First, to control for hospitals with small sample sizes, we adjusted all readmission rates using an adjusted performance fraction ([numerator+1]/[denominator +2]), where the numerator is the number of all-cause 30-day readmissions and the denominator is the number of discharges for the selected diagnosis. Then the hospitals were ordered from lowest (best performing) to highest (worst performing) using the adjusted readmission rate. Third, the number of discharges from the best performing hospital to the worst performing hospital was summed until at least 10% of the total discharges had been accounted for. Finally, we computed the ABC as the average of these best performing hospitals. We only report ABCs for which at least three hospitals were included as best performers in the calculation.¹³

To evaluate hospital performance on ABCs for each diagnosis, we identified the percent of hospitals in each setting that were outliers. We defined an outlier as any hospital whose 95% confidence interval for their readmission rate for a given diagnosis did not contain the ABC for their hospital type. All the statistical analyses were performed using SAS version 9.3 (SAS Institute, Inc, Cary, North Carolina).

This project was reviewed by the Cincinnati Children’s Hospital Medical Center Institutional Review Board and determined to be nonhuman subjects research.

Hospital-Type Demographics

The 690,949 discharges from 1,664 hospitals were categorized into 525 metropolitan teaching (550,039 discharges, 79.6% of discharges), 552 metropolitan nonteaching (97,207 discharges, 14% of discharges), and 587 nonmetropolitan hospitals (43,703 discharges, 6.3% of discharges; Table 1). There were significant differences in the patient composition among the three hospital settings. Nonmetropolitan hospitals had a larger percentage of younger patients (aged 0-4 years, P < .001), prominence of first and second quartile median household income, and fewer medically complex patients (48.3% No CCC/No CCI versus 25.5% metropolitan teaching and 33.7% nonteaching, P < .001). Disposition home was over 96% in all three hospital types; however, the metropolitan teaching had a greater percentage of patients discharged to home health versus metropolitan nonteaching and nonmetropolitan hospitals (2.3% versus 0.5%; P < .001).

Readmission Rates

The 17 most common diagnoses based on the number of all-cause 30-day same-hospital readmissions, were categorized into two surgical, seven acute/infectious, four chronic, and four mental health diagnoses (Table 2). Readmission rates varied based on diagnosis and hospital type (Table 2). Overall, mean readmission rates were low, especially in acute respiratory tract related diseases. For chronic diseases, asthma readmissions were consistently low in all three hospital types, whereas sickle cell disease had the highest readmission rate in all three hospital types.

Achievable Benchmarks of Care by Hospital Type

The diagnoses for which ABC could be calculated across all three hospital types included appendectomy and four acute conditions (bronchiolitis, pneumonia, nonbacterial gastroenteritis, and kidney/urinary tract infections). For these conditions, metropolitan teaching hospitals had a more significant percentage of outlier hospitals compared to metropolitan nonteaching and nonmetropolitan hospitals. The percent of outlier hospitals varied by diagnosis and hospital type (Figure).

Metropolitan Teaching

The readmission ABC was calculated for all 17 diagnoses (Table 2). The ABC ranged from 0.4% in acute kidney and urinary tract infection to 7.0% in sickle cell anemia crisis. Bipolar disorder, major depressive disorders and other psychoses, and sickle cell disease (SCD) had the highest percent of outlier hospitals whose mean readmission rates confidence interval did not contain the ABC; tonsil and adenoid procedures and viral illness had the lowest.¹

Metropolitan Nonteaching

The ABC was calculated for 13 of the 17 diagnoses because ABCs were not calculated when there were fewer than three best practicing hospitals. This was the case for tonsil and adenoid procedures, diabetes, seizures, and depression except for major depressive disorder (Table 2). Seven of the 13 diagnoses had an ABC of 0.0%: viral illness, infections of the upper respiratory tract, bronchiolitis, gastroenteritis, hypovolemia and electrolyte disorders, asthma, and childhood behavioral disorders. Like the findings at the metropolitan teaching hospitals, ABCs were lowest for surgical and acute conditions while bipolar disorder, major depressive disorders and other psychoses, and SCD had the highest percent of outlier hospitals with readmission rates beyond the 95% confidence interval of their hospital type’s ABC.

Nonmetropolitan

There was a sufficient number of best practicing hospitals to calculate the ABC for six of the 17 diagnoses (Table 2). For conditions where readmission ABCs could be calculated, they were low: 0.0% for appendectomy, bronchiolitis, gastroenteritis, and seizure; 0.3% for pneumonia; and 1.3% in kidney and urinary tract disorders. None of the conditions with the highest ABCs in other hospital settings (bipolar disease, sickle cell anemia crisis, and major depressive disorders and other psychoses) could be calculated in this setting. Seizure-related readmissions exhibited the most outlier hospitals yet were less than 5%.¹

Among a nationally representative sample of different hospital types that deliver care to children, we report the mean readmission rates and ABCs for 30-day all-cause, same-hospital readmissions for the most commonly readmitted pediatric diagnoses based on hospital type. Previous studies have shown patient variables such as race, ethnicity, and insurance type influencing readmission rates.^19,20 However, hospital type has also been associated with a higher risk of readmission due to the varying complexity of patients at different hospital types.^21,22 Our analyses provide hospital-type specific national estimates of pediatric readmission ABCs for medical and surgical conditions, many less than 1%. While commonly encountered pediatric conditions like asthma and bronchiolitis had low mean readmission rates and ABCs across all hospital types, the mean rates and ABCs for SCD and mental health disorders were much higher with more hospitals performing far from the ABCs.

Diagnoses with a larger percentage of outlier hospitals may represent a national opportunity to improve care for children. Conditions such as SCD and mental illnesses have the highest percentage of hospitals whose readmission rates fall outside of the ABCs in both metropolitan teaching and metropolitan nonteaching hospitals. Hospital performance on SCD and mental health disorders may not reflect deficits in hospital quality or poor adherence to evidence-based best practices, but rather the complex interplay of factors on various levels from government policy and insurance plans, to patient and family resources, to access and availability of medical and mental health specific care. Most importantly, these diseases may represent a significant opportunity for quality improvementin hospitals across the United States.

Sickle cell disease is predominantly a disease among African-Americans, a demographic risk factor for decreased access to care and limited patient and family resources.^23-26 In previous studies evaluating the disparity in readmission rates for Black children with asthma, socioeconomic variables explained 53% of the observed disparity and readmission rates were inversely related to the childhood opportunity index of the patient’s census tract and positively related with geographic social risk.^27,28 Likewise, with SCD affecting a specific demographic and being a chronic disease, best practice policies need to account for the child’s medical needs and include the patient and family resources to ensure access to care and enhanced case management for chronic disease if we aim to improve performance among the outlier hospitals.

Similarly, barriers to care for children with mental illnesses in the United States need attention.^29,30 While there is a paucity of data on the prevalence of mental health disorders in children, one national report estimates that one in 10 American adolescents have depression.^29,31 The American Academy of Pediatrics has developed a policy statement on mental health competencies and a mental health tool-kit for primary care pediatricians; however, no such guidelines or policy statements exist for hospitalized patients with acute or chronic psychiatric conditions.^32,33 Moreover, hospitals are increasingly facing “boarding” of children with acute psychiatric illness in inpatient units and emergency departments.³⁴ The American Medical Association and the American College of Emergency Physicians have expressed concerns regarding the boarding of children with acute psychiatric illness because nonpsychiatric hospitals do not have adequate resources to evaluate, manage, and place these children who deserve appropriate facilities for further management. Coordinated case management and “bundled” discharge planning in other chronic illnesses have shown benefit in cost reduction and readmission.^35-37 Evidence-based practices around pediatric readmissions in other diagnoses should be explored as possible interventions in these conditions.³⁸

There are several limitations to this study. Our data is limited to one calendar year; therefore, admissions in January do not account for potential readmissions from December of the previous year, as patient identifiers do not link across years in the NRD. We also limited our evaluation to the conventional 30-day readmission window, but recent publications may indicate that readmission windows with different timelines could be a more accurate reflection of medically preventable readmissions versus a reflection of social determinants of health leading to readmissions.²⁴ Newborn index admissions were not an allowable index admission; therefore, we may be underreporting readmissions in the neonatal age group. We also chose to include all-cause readmissions, a conventional method to evaluate readmission within an institution, but which may not reflect the quality of care delivered in the index admission. For example, an asthmatic discharged after an acute exacerbation readmitted for dehydration secondary to gastroenteritis may not reflect a lack of quality in asthma inpatient care. Readmissions were limited to the same hospital; therefore, this study cannot account for readmissions at other institutions, which may cause us to underestimate readmission rates. However, end-users of our findings most likely have access only to their own institution’s data. The inclusion of observation status admissions in the database varies from state to state; therefore, this percent of admissions in the database is unknown.

The use of the ABC methodology has some inherent limitations. One hospital with a significant volume diagnosis and low readmission rate within a hospital type may prohibit the reporting of an ABC if less than three hospitals composed the total of the ‘best performing’ hospitals. This was a significant limitation leading to the exclusion of many ABCs in nonmetropolitan institutions. The limitation of calculating and reporting an ABC then prohibits the calculation of outlier hospitals within a hospital type for a given diagnosis. However, when the ABCs are not available, we do provide the mean readmission rate for the diagnosis within the hospital type. While the hospital groupings by population and teaching status for ABCs provide meaningful comparisons for within each hospital setting, it should be noted that there may be vast differences among hospitals within each type (eg, tertiary children’s hospitals compared to teaching hospitals with a pediatric floor in the metropolitan teaching hospital category).^39,40

As healthcare moves from a fee-for-service model to a population-health centered, value-based model, reduction in readmission rates will be more than a quality measure and will have potential financial implications.⁴¹ In the Medicare fee-for-service patients, the Hospital Readmission Reduction Program (HRRP) penalize hospitals with excess readmissions for acute myocardial infarction, heart failure, and pneumonia. The hospitals subject to penalties in the HRRP had greater reduction in readmission rates in the targeted, and even nontargeted conditions, compared with hospitals not subject to penalties.⁴² Similarly, we believe that our data on low readmission rates and ABCs for conditions such as asthma, bronchiolitis, and appendicitis could represent decades of quality improvement work for the most common pediatric conditions among hospitalized children. Sickle cell disease and mental health problems remain as outliers and merit further attention. To move to a true population-health model, hospitals will need to explore outlier conditions including evaluating patient-level readmission patterns across institutions. This moves readmission from a hospital quality measure to a patient-centric quality measure, and perhaps will provide value to the patient and the healthcare system alike.

The readmission ABCs for the most commonly readmitted pediatric diagnoses are low, regardless of the hospital setting. The highest pediatric readmission rates in SCD, bipolar disorders, and major depressive disorder were lower than the most common adult readmission diagnoses. However, mental health conditions and SCD remain as outliers for pediatric readmissions, burden hospital systems, and perhaps warrant national-level attention. The ABCs stratified by hospital type in this study facilitate comparisons and identify opportunities for population-level interventions to meaningfully improve patient care.

Disclosures

The authors have nothing to disclose.

In 2016, 15.2% of older Americans were hospitalized compared with 7% of the overall population and their length of stay (LOS) was 0.7 days greater.¹ Geriatric hospitalizations frequently result in complications, functional decline, nursing home transfers, and increased cost.^2-4 This pattern of decline has been termed “hospitalitis” or dysfunctional syndrome.^5,6 Hospitals need data-driven approaches to improve outcomes for elders. The Acute Care for Elders (ACE) program, which has been in existence for roughly 25 years, is one such model. ACE features include an environment prepared for older adults, patient-centered care to prevent functional and cognitive decline, frequent medical review to prevent iatrogenic injury or new geriatric syndromes, and early discharge and rehabilitation planning to maximize the likelihood of return to the community.⁷ Although published data vary somewhat, ACE programs have robust evidence documenting improved safety, quality, and value.^8-15 A recent meta-analysis found that ACE programs decrease LOS, costs, new nursing home discharges, falls, delirium, and functional decline.¹⁶ However, of the 13 ACE trials reported to date, only five were published in the last decade. Recent rising pressure to decrease hospitalizations and reduce LOS has shifted some care to other settings and it is unclear whether the same results would persist in today’s rapid-paced hospitals.

ACE programs require enhanced resources and restructured care processes but there is a notable lack of data to guide patient selection. Admission criteria vary among the published reports, and information on whether comorbidity burden impacts the magnitude of benefit is scarce. One ACE investigator commented, “We were not able to identify a subgroup of patients who were most likely to benefit.”¹⁷ Not all hospitalized older adults can receive ACE care, and some units have closed due to financial and logistic pressures; thus, criteria to target this scarce resource are urgently needed. Our hospital implemented an ACE program in 2014 and we have measured and internally benchmarked important quality improvement metrics. Using this data, we conducted an exploratory analysis to generate hypotheses on the differential impact across the spectrum of cost, LOS, 30-day readmissions, and variations across quartiles of comorbidity severity.

Setting and Patients

In September 2014, our 716-bed teaching hospital in Springfield, Massachusetts launched an ACE program to improve care for older adults on a single medical unit. The program succeeded in engaging the senior leadership, and geriatrics was identified as a priority in Baystate’s 5-year strategic plan. ACE patients ≥70 years were admitted from the emergency department with inpatient status. Patients transferred from other units or with advanced dementia or nearing death were excluded. Core components of the ACE program were derived from published summaries (see supplementary material).^7,16

Interprofessional ”ACE Rounds”

Interprofessional ACE Rounds occurred every weekday. As one ACE analyst has noted, “the interdisciplinary team…ensures that the multifactorial nature of functional decline is met with a multicomponent plan to prevent it.”¹⁸ Rounds participants shifted over time but always included a geriatrics physician assistant (PA) or geriatrician (team leader), a pharmacist, staff nurses, and a chaplain. The nurse educator, dietician, research assistant, and patient advocate/volunteers attended intermittently. Before rounds, the PA reviewed the admission notes for new ACE patients. Initially, rounds were lengthy and included nurse coaching. Later, nurses’ presentations were structured by the SPICES tool (Sleep, Problems with eating/feeding, Incontinence, Confusion, Evidence of falls, Skin Breakdown)¹⁹ and tracking and reporting templates. Coaching and education, along with conversations that did not require the full team, were removed from rounds. Thus, the time required for rounds declined from about 75 minutes to 35 minutes, which allowed more patients to be discussed efficiently. This change was critical as the number of ACE patients rose following the shift to the larger unit. The pharmacist reviewed medications focusing on potentially inappropriate drugs. Following rounds, the nurses and pharmacist conveyed recommendations to the hospitalists.

Patient-Centered Activities to Prevent Functional and Cognitive Decline

Project leaders coached staff about the importance of mobility, sleep, and delirium prevention and identification. The nurses screened patients using the Confusion Assessment Method (CAM) and reported delirium promptly. Specific care sets for ACE patients were implemented (see supplementary material).

The project was enhanced by several palliative care components, ie tracking pain, noting psychiatric symptoms, and considering prognosis by posing the “Surprise Question” during rounds.²⁰ (“Would you be surprised if this patient died in the next year?”). As far as staffing and logistics allowed, the goals of care conversation were held by a geriatrics PA with patients/families who “screened in.”

Prepared Environment

The ACE program’s unit was remodeled to facilitate physical and cognitive functioning and promote sleep at night (quiet hours: 10 PM-6 AM).

In accordance with quality improvement processes, iterative shifts were implemented over time in terms of checklist, presentation format, timing, and team participation. In December 2016, the program relocated to a unit with 34 ACE beds and 5 end-of-life beds; this move markedly increased the number of eligible ACE patients.

Study Design, Data Source, and Patients

Since we were implementing and measuring our ACE program with a quality improvement lens, we chose a descriptive cross-sectional study design to generate hypotheses regarding our program’s impact compared to usual care. Using a hospital-wide billing database (McKesson Performance Analytics, v19, Alpharetta, Georgia) we sampled inpatients aged >70 years with a medical Diagnosis Related Group (DRGs) admitted through the emergency department and discharged from a medical unit from September 22, 2014 to August 31, 2017. These criteria mirrored those in the ACE unit. Older adults requiring specialized care (eg, those with myocardial infarct) were excluded, as were those with billing codes for mechanical ventilation, admission to critical care units, or discharge to hospice. Because one of our outcomes was readmission, we excluded patients who died during hospitalization. Patient characteristics collected included demographics and insurance category. To evaluate comorbidity burden, we collected ICD-9/ICD-10 diagnostic codes and generated a combined comorbidity score as described by Gagne, et al.²¹ This score was devised to predict mortality and 30-day readmissions and had better predictive ability in elders than the Elixhauser or Charlson scores. Scores ranged from −2 to 26, although values >20 are rare.

Exposure

Subjects were categorized as either discharged from the ACE or discharged from usual care. ACE discharges were tracked daily on a spreadsheet that was linked into our sample of eligible subjects.

Outcomes

Total cost of hospitalization (direct plus indirect costs), LOS, and all-cause 30-day readmissions were queried from the same billing database.

Statistical Analysis

As this study was a quality improvement project, analyses were descriptive and exploratory; no statistical hypothesis testing was conducted. We initially evaluated subject characteristics and comorbidities across study groups to determine group balance and comparability using means and standard deviations for continuous data and frequencies and percentages for categorical data. To analyze total cost and LOS, we utilized quantile regression with clustered standard errors to account for clustering by patient. We calculated the median difference between hospitalization cost and LOS for usual care versus ACE patients (with ACE as the referent group). To explore variations across the distributions of outcomes, we determined differences in cost and LOS and their 95% confidence intervals at the 25th, 50th, 75th, and 90th percentiles. Thirty-day readmission risk was estimated using a generalized estimating equation model with a logit link and binomial family. Readmission risk is presented along with 95% confidence intervals. For all models, we initially evaluated change over time (by quarter). After establishing the absence of time trends, we collapsed results into a comparison of usual care versus ACE care. Model estimates are presented both unadjusted and adjusted for age and comorbidity score. Following our initial analyses of cost, LOS, and 30-day readmission risk; we explored differences across quartiles of combined comorbidity scores. We used the same unadjusted models described above but incorporated an interaction term to generate estimates stratified by quartile of comorbidity score. We performed two additional analyses to evaluate the robustness of our findings. First, because hemiplegia prevalence was higher in the usual-care group than in the ACE group and can result in higher cost of care, we repeated the analysis after excluding those patients with hemiplegia. Second, because we were unable to control for functional capacity in the entire sample, we evaluated group differences in mobility for a subsample obtained prior to October 2015 using ICD-9 diagnostic codes, which can be considered surrogate markers for mobility.²² The results of our first analysis did not substantively change our main findings; in our second analysis, groups were balanced by mobility factors which suggested that confounding by functional capacity would be limited in our full sample. The results of these analyses are reported in the supplemental material.

Analysis was completed using Stata v15.1 (StataCorp, LP College Station, Texas). The Baystate Medical Center Institutional Review Board determined that the initiative was quality improvement and “not research.”

A total of 13,209 patients met the initial inclusion criteria; 1,621 were excluded, resulting in a sample of 11,588 patients. Over the 3-year study period, 1,429 (12.3%) were discharged from ACE and 10,159 (87.7%) were discharged from usual care. The groups were similar in age, sex, race and insurance status. Compared with the usual-care group, ACE patients had a higher median comorbidity score (3 vs 2 for usual care) and higher rates for anemia, dementia, fluid and electrolyte disorders, hypertension, and chronic obstructive pulmonary disease (COPD). However, ACE patients had lower rates of hemiplegia (0.9% vs 3%), arrhythmias, and pulmonary circulation disorders than those with usual care (Table 1).

The median cost per ACE patient was slightly lower at $6,258 (interquartile range [IQR] = $4,683-$8,547) versus $6,858 (IQR = $4,855-$10,478) in usual care. Across the cost distribution, the ACE program had lower costs than usual care; however, these differences became more pronounced at the higher end of the distribution. For example, compared with the ACE group, the usual-care group’s unadjusted cost difference was $171 higher at the 25th percentile, $600 higher at the median, $1,932 higher at the 75^th percentile, and $3,687 higher at the 90th percentile. The ACE median LOS was 3.7 days (IQR = 2.7-5.0) compared with 3.8 days (IQR = 2.7-6.0) for non-ACE patients. Similar to cost, LOS differences rose at higher percentiles of the distribution, with shorter stays for the ACE patients within each grouping. Compared with the ACE group, the unadjusted LOS difference for usual-care patients ranged from 0 days at the 25th percentile to 0.2 day longer at the median, 1.0 day longer at the 75th percentile, and 1.9 days longer at the 90th percentile. For both cost and LOS models, estimates remained stable after adjusting for age and combined comorbidity score (Table 2).

This quality improvement initiative evaluated which ACE admissions yielded the greatest value and found the largest reductions in LOS and cost in patients with the greatest comorbidity scores (frequently referred to as “high need, high cost”).^23,24 Based on prior literature, we had anticipated that moderate risk patients would show the maximum benefit.^15,25 In contrast to our findings, a University of Alabama (UAB) ACE program subgroup analysis using the CMS Case Mix Index (CMI) found a cost reduction for patients with low or intermediate CMI scores but not for those with high scores.¹⁵ The Hospital Elder Life Program (HELP) has yielded maximal impact for patients at moderate risk for delirium.²⁶ Our results are supported by a University of Texas, Houston, study revealing lower LOS and cost for ACE patients, despite high CMI scores and endemic frailty, although it did not report outcomes across a range of comorbidities or costs.²⁷ Our results may be determined by the specific characteristics of the Baystate ACE initiative. Our emphasis on considering prognosis and encouraging advance care planning could have contributed to the improved metrics for more complicated patients. It is possible that patients with high comorbidity burden were more likely to screen in with the surprise question, leading to more frequent goals of care discussions by the hospitalists or geriatrics team, which, in turn, may have resulted in less aggressive care and consequently lower costs. The emphasis on prognosis and palliative care was not a feature of the UAB or Texas studies. Additional components, such as the delirium screening and the presence of volunteer advocates, could also have impacted the results. Our tiered approach during rounds with rapid reviews for most patients and longer discussions for those at highest risk may have further contributed to the findings. Finally, although we did not track the recommendation acceptance rate for the entire study period, in the first 9 months of the project, 9,325 recommendations were made with an acceptance rate of >85%. We previously reported a similar acceptance rate for medication recommendations.²⁸ Another factor contributing to our results may be the ways in which we categorized patients and calculated costs. We used the Gagne combined comorbidity score, which includes only prior conditions;²¹ the UAB study used CMI, which includes severity of presenting illness and complications, as well as baseline comorbidities. We also compared total cost, while UAB reported variable direct cost.

This study has a number of limitations. First, it was conducted at a single site and may not apply to other hospitals. Second, as a quality improvement program, its design, processes, and personnel evolved over time, and, as in any multicomponent initiative, the effect of individual factors on the outcomes is unknown. Third, this is an observational study with the aim of generating hypotheses for more rigorous studies in the future and residual confounding factors may exist despite efforts to adjust for variables present in an administrative database. Thus, we were unable to completely adjust for potentially important social factors, presence of delirium, or baseline functional status.

To our knowledge, this study is the first report on the differential impact of comorbidity scores and cost distribution on ACE total cost and LOS reductions. Despite its limitations, it contributes to the existing literature by suggesting that the Gagne comorbidity score can help identify which admissions will yield the greatest value. The Gagne score could be calculated at admission using the ePrognosis risk calculator or incorporated and automated in the EMR.²⁹ Many health systems are reluctant to designate beds for specific subpopulations since doing so decreases flexibility and complicates the admission process. A dynamic tension exists between increasing income streams now and generating future savings by supporting initiatives with upfront costs. Other successful acute care geriatrics programs, such as NICHE,³⁰ HELP,³¹ MACE,³² and consultation teams, exist.³³ Studies reporting the outcomes of combining ACE units with these other approaches in a “portfolio approach” will inform the design of the most efficient and impactful programs.³⁴ Scrupulous attention to symptom control and advance care planning are key features of our program, and, given the high prevalence of advanced serious illness in hospitalized older adults, this consideration may be critical for success.

As ACE units can only care for a small fraction of hospitalized older adults, determining which patients will maximally benefit from the structured, team-based care on ACE units is crucial. We found that the greatest impact on LOS and costs occurred in the subgroup with the highest comorbidity scores and overall cost. ACE care for the most vulnerable patients appeared to yield the greatest value for the system; thus, these older adults may need to be prioritized for admission. This improvement may enhance quality and value outcomes, maximize a scarce resource, and secure results needed to sustain the “clinician-led and data-driven” ACE model in the face of changing clinical and financial landscapes.³⁵

Acknowledgments

All those with significant contributions to this work are included as authors.

The authors express their deep appreciation to all their Baystate collaborators, particularly to Rebecca Starr, MD, the first geriatrics medical director of the program, Ms. Virginia Chipps, RN, the program’s first nurse manager, and Tasmiah Chowdhury, PharmD, the program’s first pharmacist. We are also deeply grateful to those persons who provided programmatic advice and input on model ACE programs elsewhere, including Kyle Allen, MD, Michael Malone MD, Robert Palmer MD, and, especially, Kellie Flood, MD.

Disclosures

None of the authors have any existing or potential personal or financial conflicts relevant to this paper to report.

Funding

This work was supported in part by a Geriatric Workforce Enhancement Program award (grant # U1QHP28702) from the Health Resources and Services Administration and by internal support from Baystate Health

In 2016, 15.2% of older Americans were hospitalized compared with 7% of the overall population and their length of stay (LOS) was 0.7 days greater.¹ Geriatric hospitalizations frequently result in complications, functional decline, nursing home transfers, and increased cost.^2-4 This pattern of decline has been termed “hospitalitis” or dysfunctional syndrome.^5,6 Hospitals need data-driven approaches to improve outcomes for elders. The Acute Care for Elders (ACE) program, which has been in existence for roughly 25 years, is one such model. ACE features include an environment prepared for older adults, patient-centered care to prevent functional and cognitive decline, frequent medical review to prevent iatrogenic injury or new geriatric syndromes, and early discharge and rehabilitation planning to maximize the likelihood of return to the community.⁷ Although published data vary somewhat, ACE programs have robust evidence documenting improved safety, quality, and value.^8-15 A recent meta-analysis found that ACE programs decrease LOS, costs, new nursing home discharges, falls, delirium, and functional decline.¹⁶ However, of the 13 ACE trials reported to date, only five were published in the last decade. Recent rising pressure to decrease hospitalizations and reduce LOS has shifted some care to other settings and it is unclear whether the same results would persist in today’s rapid-paced hospitals.

ACE programs require enhanced resources and restructured care processes but there is a notable lack of data to guide patient selection. Admission criteria vary among the published reports, and information on whether comorbidity burden impacts the magnitude of benefit is scarce. One ACE investigator commented, “We were not able to identify a subgroup of patients who were most likely to benefit.”¹⁷ Not all hospitalized older adults can receive ACE care, and some units have closed due to financial and logistic pressures; thus, criteria to target this scarce resource are urgently needed. Our hospital implemented an ACE program in 2014 and we have measured and internally benchmarked important quality improvement metrics. Using this data, we conducted an exploratory analysis to generate hypotheses on the differential impact across the spectrum of cost, LOS, 30-day readmissions, and variations across quartiles of comorbidity severity.

Setting and Patients

In September 2014, our 716-bed teaching hospital in Springfield, Massachusetts launched an ACE program to improve care for older adults on a single medical unit. The program succeeded in engaging the senior leadership, and geriatrics was identified as a priority in Baystate’s 5-year strategic plan. ACE patients ≥70 years were admitted from the emergency department with inpatient status. Patients transferred from other units or with advanced dementia or nearing death were excluded. Core components of the ACE program were derived from published summaries (see supplementary material).^7,16

Interprofessional ”ACE Rounds”

Interprofessional ACE Rounds occurred every weekday. As one ACE analyst has noted, “the interdisciplinary team…ensures that the multifactorial nature of functional decline is met with a multicomponent plan to prevent it.”¹⁸ Rounds participants shifted over time but always included a geriatrics physician assistant (PA) or geriatrician (team leader), a pharmacist, staff nurses, and a chaplain. The nurse educator, dietician, research assistant, and patient advocate/volunteers attended intermittently. Before rounds, the PA reviewed the admission notes for new ACE patients. Initially, rounds were lengthy and included nurse coaching. Later, nurses’ presentations were structured by the SPICES tool (Sleep, Problems with eating/feeding, Incontinence, Confusion, Evidence of falls, Skin Breakdown)¹⁹ and tracking and reporting templates. Coaching and education, along with conversations that did not require the full team, were removed from rounds. Thus, the time required for rounds declined from about 75 minutes to 35 minutes, which allowed more patients to be discussed efficiently. This change was critical as the number of ACE patients rose following the shift to the larger unit. The pharmacist reviewed medications focusing on potentially inappropriate drugs. Following rounds, the nurses and pharmacist conveyed recommendations to the hospitalists.

Patient-Centered Activities to Prevent Functional and Cognitive Decline

Project leaders coached staff about the importance of mobility, sleep, and delirium prevention and identification. The nurses screened patients using the Confusion Assessment Method (CAM) and reported delirium promptly. Specific care sets for ACE patients were implemented (see supplementary material).

The project was enhanced by several palliative care components, ie tracking pain, noting psychiatric symptoms, and considering prognosis by posing the “Surprise Question” during rounds.²⁰ (“Would you be surprised if this patient died in the next year?”). As far as staffing and logistics allowed, the goals of care conversation were held by a geriatrics PA with patients/families who “screened in.”

Prepared Environment

The ACE program’s unit was remodeled to facilitate physical and cognitive functioning and promote sleep at night (quiet hours: 10 PM-6 AM).

In accordance with quality improvement processes, iterative shifts were implemented over time in terms of checklist, presentation format, timing, and team participation. In December 2016, the program relocated to a unit with 34 ACE beds and 5 end-of-life beds; this move markedly increased the number of eligible ACE patients.

Study Design, Data Source, and Patients

Since we were implementing and measuring our ACE program with a quality improvement lens, we chose a descriptive cross-sectional study design to generate hypotheses regarding our program’s impact compared to usual care. Using a hospital-wide billing database (McKesson Performance Analytics, v19, Alpharetta, Georgia) we sampled inpatients aged >70 years with a medical Diagnosis Related Group (DRGs) admitted through the emergency department and discharged from a medical unit from September 22, 2014 to August 31, 2017. These criteria mirrored those in the ACE unit. Older adults requiring specialized care (eg, those with myocardial infarct) were excluded, as were those with billing codes for mechanical ventilation, admission to critical care units, or discharge to hospice. Because one of our outcomes was readmission, we excluded patients who died during hospitalization. Patient characteristics collected included demographics and insurance category. To evaluate comorbidity burden, we collected ICD-9/ICD-10 diagnostic codes and generated a combined comorbidity score as described by Gagne, et al.²¹ This score was devised to predict mortality and 30-day readmissions and had better predictive ability in elders than the Elixhauser or Charlson scores. Scores ranged from −2 to 26, although values >20 are rare.

Exposure

Subjects were categorized as either discharged from the ACE or discharged from usual care. ACE discharges were tracked daily on a spreadsheet that was linked into our sample of eligible subjects.

Outcomes

Total cost of hospitalization (direct plus indirect costs), LOS, and all-cause 30-day readmissions were queried from the same billing database.

Statistical Analysis

As this study was a quality improvement project, analyses were descriptive and exploratory; no statistical hypothesis testing was conducted. We initially evaluated subject characteristics and comorbidities across study groups to determine group balance and comparability using means and standard deviations for continuous data and frequencies and percentages for categorical data. To analyze total cost and LOS, we utilized quantile regression with clustered standard errors to account for clustering by patient. We calculated the median difference between hospitalization cost and LOS for usual care versus ACE patients (with ACE as the referent group). To explore variations across the distributions of outcomes, we determined differences in cost and LOS and their 95% confidence intervals at the 25th, 50th, 75th, and 90th percentiles. Thirty-day readmission risk was estimated using a generalized estimating equation model with a logit link and binomial family. Readmission risk is presented along with 95% confidence intervals. For all models, we initially evaluated change over time (by quarter). After establishing the absence of time trends, we collapsed results into a comparison of usual care versus ACE care. Model estimates are presented both unadjusted and adjusted for age and comorbidity score. Following our initial analyses of cost, LOS, and 30-day readmission risk; we explored differences across quartiles of combined comorbidity scores. We used the same unadjusted models described above but incorporated an interaction term to generate estimates stratified by quartile of comorbidity score. We performed two additional analyses to evaluate the robustness of our findings. First, because hemiplegia prevalence was higher in the usual-care group than in the ACE group and can result in higher cost of care, we repeated the analysis after excluding those patients with hemiplegia. Second, because we were unable to control for functional capacity in the entire sample, we evaluated group differences in mobility for a subsample obtained prior to October 2015 using ICD-9 diagnostic codes, which can be considered surrogate markers for mobility.²² The results of our first analysis did not substantively change our main findings; in our second analysis, groups were balanced by mobility factors which suggested that confounding by functional capacity would be limited in our full sample. The results of these analyses are reported in the supplemental material.

Analysis was completed using Stata v15.1 (StataCorp, LP College Station, Texas). The Baystate Medical Center Institutional Review Board determined that the initiative was quality improvement and “not research.”

A total of 13,209 patients met the initial inclusion criteria; 1,621 were excluded, resulting in a sample of 11,588 patients. Over the 3-year study period, 1,429 (12.3%) were discharged from ACE and 10,159 (87.7%) were discharged from usual care. The groups were similar in age, sex, race and insurance status. Compared with the usual-care group, ACE patients had a higher median comorbidity score (3 vs 2 for usual care) and higher rates for anemia, dementia, fluid and electrolyte disorders, hypertension, and chronic obstructive pulmonary disease (COPD). However, ACE patients had lower rates of hemiplegia (0.9% vs 3%), arrhythmias, and pulmonary circulation disorders than those with usual care (Table 1).

The median cost per ACE patient was slightly lower at $6,258 (interquartile range [IQR] = $4,683-$8,547) versus $6,858 (IQR = $4,855-$10,478) in usual care. Across the cost distribution, the ACE program had lower costs than usual care; however, these differences became more pronounced at the higher end of the distribution. For example, compared with the ACE group, the usual-care group’s unadjusted cost difference was $171 higher at the 25th percentile, $600 higher at the median, $1,932 higher at the 75^th percentile, and $3,687 higher at the 90th percentile. The ACE median LOS was 3.7 days (IQR = 2.7-5.0) compared with 3.8 days (IQR = 2.7-6.0) for non-ACE patients. Similar to cost, LOS differences rose at higher percentiles of the distribution, with shorter stays for the ACE patients within each grouping. Compared with the ACE group, the unadjusted LOS difference for usual-care patients ranged from 0 days at the 25th percentile to 0.2 day longer at the median, 1.0 day longer at the 75th percentile, and 1.9 days longer at the 90th percentile. For both cost and LOS models, estimates remained stable after adjusting for age and combined comorbidity score (Table 2).

This quality improvement initiative evaluated which ACE admissions yielded the greatest value and found the largest reductions in LOS and cost in patients with the greatest comorbidity scores (frequently referred to as “high need, high cost”).^23,24 Based on prior literature, we had anticipated that moderate risk patients would show the maximum benefit.^15,25 In contrast to our findings, a University of Alabama (UAB) ACE program subgroup analysis using the CMS Case Mix Index (CMI) found a cost reduction for patients with low or intermediate CMI scores but not for those with high scores.¹⁵ The Hospital Elder Life Program (HELP) has yielded maximal impact for patients at moderate risk for delirium.²⁶ Our results are supported by a University of Texas, Houston, study revealing lower LOS and cost for ACE patients, despite high CMI scores and endemic frailty, although it did not report outcomes across a range of comorbidities or costs.²⁷ Our results may be determined by the specific characteristics of the Baystate ACE initiative. Our emphasis on considering prognosis and encouraging advance care planning could have contributed to the improved metrics for more complicated patients. It is possible that patients with high comorbidity burden were more likely to screen in with the surprise question, leading to more frequent goals of care discussions by the hospitalists or geriatrics team, which, in turn, may have resulted in less aggressive care and consequently lower costs. The emphasis on prognosis and palliative care was not a feature of the UAB or Texas studies. Additional components, such as the delirium screening and the presence of volunteer advocates, could also have impacted the results. Our tiered approach during rounds with rapid reviews for most patients and longer discussions for those at highest risk may have further contributed to the findings. Finally, although we did not track the recommendation acceptance rate for the entire study period, in the first 9 months of the project, 9,325 recommendations were made with an acceptance rate of >85%. We previously reported a similar acceptance rate for medication recommendations.²⁸ Another factor contributing to our results may be the ways in which we categorized patients and calculated costs. We used the Gagne combined comorbidity score, which includes only prior conditions;²¹ the UAB study used CMI, which includes severity of presenting illness and complications, as well as baseline comorbidities. We also compared total cost, while UAB reported variable direct cost.

This study has a number of limitations. First, it was conducted at a single site and may not apply to other hospitals. Second, as a quality improvement program, its design, processes, and personnel evolved over time, and, as in any multicomponent initiative, the effect of individual factors on the outcomes is unknown. Third, this is an observational study with the aim of generating hypotheses for more rigorous studies in the future and residual confounding factors may exist despite efforts to adjust for variables present in an administrative database. Thus, we were unable to completely adjust for potentially important social factors, presence of delirium, or baseline functional status.

To our knowledge, this study is the first report on the differential impact of comorbidity scores and cost distribution on ACE total cost and LOS reductions. Despite its limitations, it contributes to the existing literature by suggesting that the Gagne comorbidity score can help identify which admissions will yield the greatest value. The Gagne score could be calculated at admission using the ePrognosis risk calculator or incorporated and automated in the EMR.²⁹ Many health systems are reluctant to designate beds for specific subpopulations since doing so decreases flexibility and complicates the admission process. A dynamic tension exists between increasing income streams now and generating future savings by supporting initiatives with upfront costs. Other successful acute care geriatrics programs, such as NICHE,³⁰ HELP,³¹ MACE,³² and consultation teams, exist.³³ Studies reporting the outcomes of combining ACE units with these other approaches in a “portfolio approach” will inform the design of the most efficient and impactful programs.³⁴ Scrupulous attention to symptom control and advance care planning are key features of our program, and, given the high prevalence of advanced serious illness in hospitalized older adults, this consideration may be critical for success.

As ACE units can only care for a small fraction of hospitalized older adults, determining which patients will maximally benefit from the structured, team-based care on ACE units is crucial. We found that the greatest impact on LOS and costs occurred in the subgroup with the highest comorbidity scores and overall cost. ACE care for the most vulnerable patients appeared to yield the greatest value for the system; thus, these older adults may need to be prioritized for admission. This improvement may enhance quality and value outcomes, maximize a scarce resource, and secure results needed to sustain the “clinician-led and data-driven” ACE model in the face of changing clinical and financial landscapes.³⁵

Acknowledgments

All those with significant contributions to this work are included as authors.

The authors express their deep appreciation to all their Baystate collaborators, particularly to Rebecca Starr, MD, the first geriatrics medical director of the program, Ms. Virginia Chipps, RN, the program’s first nurse manager, and Tasmiah Chowdhury, PharmD, the program’s first pharmacist. We are also deeply grateful to those persons who provided programmatic advice and input on model ACE programs elsewhere, including Kyle Allen, MD, Michael Malone MD, Robert Palmer MD, and, especially, Kellie Flood, MD.

Disclosures

None of the authors have any existing or potential personal or financial conflicts relevant to this paper to report.

Funding

This work was supported in part by a Geriatric Workforce Enhancement Program award (grant # U1QHP28702) from the Health Resources and Services Administration and by internal support from Baystate Health

In 2016, 15.2% of older Americans were hospitalized compared with 7% of the overall population and their length of stay (LOS) was 0.7 days greater.¹ Geriatric hospitalizations frequently result in complications, functional decline, nursing home transfers, and increased cost.^2-4 This pattern of decline has been termed “hospitalitis” or dysfunctional syndrome.^5,6 Hospitals need data-driven approaches to improve outcomes for elders. The Acute Care for Elders (ACE) program, which has been in existence for roughly 25 years, is one such model. ACE features include an environment prepared for older adults, patient-centered care to prevent functional and cognitive decline, frequent medical review to prevent iatrogenic injury or new geriatric syndromes, and early discharge and rehabilitation planning to maximize the likelihood of return to the community.⁷ Although published data vary somewhat, ACE programs have robust evidence documenting improved safety, quality, and value.^8-15 A recent meta-analysis found that ACE programs decrease LOS, costs, new nursing home discharges, falls, delirium, and functional decline.¹⁶ However, of the 13 ACE trials reported to date, only five were published in the last decade. Recent rising pressure to decrease hospitalizations and reduce LOS has shifted some care to other settings and it is unclear whether the same results would persist in today’s rapid-paced hospitals.

ACE programs require enhanced resources and restructured care processes but there is a notable lack of data to guide patient selection. Admission criteria vary among the published reports, and information on whether comorbidity burden impacts the magnitude of benefit is scarce. One ACE investigator commented, “We were not able to identify a subgroup of patients who were most likely to benefit.”¹⁷ Not all hospitalized older adults can receive ACE care, and some units have closed due to financial and logistic pressures; thus, criteria to target this scarce resource are urgently needed. Our hospital implemented an ACE program in 2014 and we have measured and internally benchmarked important quality improvement metrics. Using this data, we conducted an exploratory analysis to generate hypotheses on the differential impact across the spectrum of cost, LOS, 30-day readmissions, and variations across quartiles of comorbidity severity.

Setting and Patients

In September 2014, our 716-bed teaching hospital in Springfield, Massachusetts launched an ACE program to improve care for older adults on a single medical unit. The program succeeded in engaging the senior leadership, and geriatrics was identified as a priority in Baystate’s 5-year strategic plan. ACE patients ≥70 years were admitted from the emergency department with inpatient status. Patients transferred from other units or with advanced dementia or nearing death were excluded. Core components of the ACE program were derived from published summaries (see supplementary material).^7,16

Interprofessional ”ACE Rounds”

Interprofessional ACE Rounds occurred every weekday. As one ACE analyst has noted, “the interdisciplinary team…ensures that the multifactorial nature of functional decline is met with a multicomponent plan to prevent it.”¹⁸ Rounds participants shifted over time but always included a geriatrics physician assistant (PA) or geriatrician (team leader), a pharmacist, staff nurses, and a chaplain. The nurse educator, dietician, research assistant, and patient advocate/volunteers attended intermittently. Before rounds, the PA reviewed the admission notes for new ACE patients. Initially, rounds were lengthy and included nurse coaching. Later, nurses’ presentations were structured by the SPICES tool (Sleep, Problems with eating/feeding, Incontinence, Confusion, Evidence of falls, Skin Breakdown)¹⁹ and tracking and reporting templates. Coaching and education, along with conversations that did not require the full team, were removed from rounds. Thus, the time required for rounds declined from about 75 minutes to 35 minutes, which allowed more patients to be discussed efficiently. This change was critical as the number of ACE patients rose following the shift to the larger unit. The pharmacist reviewed medications focusing on potentially inappropriate drugs. Following rounds, the nurses and pharmacist conveyed recommendations to the hospitalists.

Patient-Centered Activities to Prevent Functional and Cognitive Decline

Project leaders coached staff about the importance of mobility, sleep, and delirium prevention and identification. The nurses screened patients using the Confusion Assessment Method (CAM) and reported delirium promptly. Specific care sets for ACE patients were implemented (see supplementary material).

The project was enhanced by several palliative care components, ie tracking pain, noting psychiatric symptoms, and considering prognosis by posing the “Surprise Question” during rounds.²⁰ (“Would you be surprised if this patient died in the next year?”). As far as staffing and logistics allowed, the goals of care conversation were held by a geriatrics PA with patients/families who “screened in.”

Prepared Environment

The ACE program’s unit was remodeled to facilitate physical and cognitive functioning and promote sleep at night (quiet hours: 10 PM-6 AM).

In accordance with quality improvement processes, iterative shifts were implemented over time in terms of checklist, presentation format, timing, and team participation. In December 2016, the program relocated to a unit with 34 ACE beds and 5 end-of-life beds; this move markedly increased the number of eligible ACE patients.

Study Design, Data Source, and Patients

Since we were implementing and measuring our ACE program with a quality improvement lens, we chose a descriptive cross-sectional study design to generate hypotheses regarding our program’s impact compared to usual care. Using a hospital-wide billing database (McKesson Performance Analytics, v19, Alpharetta, Georgia) we sampled inpatients aged >70 years with a medical Diagnosis Related Group (DRGs) admitted through the emergency department and discharged from a medical unit from September 22, 2014 to August 31, 2017. These criteria mirrored those in the ACE unit. Older adults requiring specialized care (eg, those with myocardial infarct) were excluded, as were those with billing codes for mechanical ventilation, admission to critical care units, or discharge to hospice. Because one of our outcomes was readmission, we excluded patients who died during hospitalization. Patient characteristics collected included demographics and insurance category. To evaluate comorbidity burden, we collected ICD-9/ICD-10 diagnostic codes and generated a combined comorbidity score as described by Gagne, et al.²¹ This score was devised to predict mortality and 30-day readmissions and had better predictive ability in elders than the Elixhauser or Charlson scores. Scores ranged from −2 to 26, although values >20 are rare.

Exposure

Subjects were categorized as either discharged from the ACE or discharged from usual care. ACE discharges were tracked daily on a spreadsheet that was linked into our sample of eligible subjects.

Outcomes

Total cost of hospitalization (direct plus indirect costs), LOS, and all-cause 30-day readmissions were queried from the same billing database.

Statistical Analysis

As this study was a quality improvement project, analyses were descriptive and exploratory; no statistical hypothesis testing was conducted. We initially evaluated subject characteristics and comorbidities across study groups to determine group balance and comparability using means and standard deviations for continuous data and frequencies and percentages for categorical data. To analyze total cost and LOS, we utilized quantile regression with clustered standard errors to account for clustering by patient. We calculated the median difference between hospitalization cost and LOS for usual care versus ACE patients (with ACE as the referent group). To explore variations across the distributions of outcomes, we determined differences in cost and LOS and their 95% confidence intervals at the 25th, 50th, 75th, and 90th percentiles. Thirty-day readmission risk was estimated using a generalized estimating equation model with a logit link and binomial family. Readmission risk is presented along with 95% confidence intervals. For all models, we initially evaluated change over time (by quarter). After establishing the absence of time trends, we collapsed results into a comparison of usual care versus ACE care. Model estimates are presented both unadjusted and adjusted for age and comorbidity score. Following our initial analyses of cost, LOS, and 30-day readmission risk; we explored differences across quartiles of combined comorbidity scores. We used the same unadjusted models described above but incorporated an interaction term to generate estimates stratified by quartile of comorbidity score. We performed two additional analyses to evaluate the robustness of our findings. First, because hemiplegia prevalence was higher in the usual-care group than in the ACE group and can result in higher cost of care, we repeated the analysis after excluding those patients with hemiplegia. Second, because we were unable to control for functional capacity in the entire sample, we evaluated group differences in mobility for a subsample obtained prior to October 2015 using ICD-9 diagnostic codes, which can be considered surrogate markers for mobility.²² The results of our first analysis did not substantively change our main findings; in our second analysis, groups were balanced by mobility factors which suggested that confounding by functional capacity would be limited in our full sample. The results of these analyses are reported in the supplemental material.

Analysis was completed using Stata v15.1 (StataCorp, LP College Station, Texas). The Baystate Medical Center Institutional Review Board determined that the initiative was quality improvement and “not research.”

A total of 13,209 patients met the initial inclusion criteria; 1,621 were excluded, resulting in a sample of 11,588 patients. Over the 3-year study period, 1,429 (12.3%) were discharged from ACE and 10,159 (87.7%) were discharged from usual care. The groups were similar in age, sex, race and insurance status. Compared with the usual-care group, ACE patients had a higher median comorbidity score (3 vs 2 for usual care) and higher rates for anemia, dementia, fluid and electrolyte disorders, hypertension, and chronic obstructive pulmonary disease (COPD). However, ACE patients had lower rates of hemiplegia (0.9% vs 3%), arrhythmias, and pulmonary circulation disorders than those with usual care (Table 1).

The median cost per ACE patient was slightly lower at $6,258 (interquartile range [IQR] = $4,683-$8,547) versus $6,858 (IQR = $4,855-$10,478) in usual care. Across the cost distribution, the ACE program had lower costs than usual care; however, these differences became more pronounced at the higher end of the distribution. For example, compared with the ACE group, the usual-care group’s unadjusted cost difference was $171 higher at the 25th percentile, $600 higher at the median, $1,932 higher at the 75^th percentile, and $3,687 higher at the 90th percentile. The ACE median LOS was 3.7 days (IQR = 2.7-5.0) compared with 3.8 days (IQR = 2.7-6.0) for non-ACE patients. Similar to cost, LOS differences rose at higher percentiles of the distribution, with shorter stays for the ACE patients within each grouping. Compared with the ACE group, the unadjusted LOS difference for usual-care patients ranged from 0 days at the 25th percentile to 0.2 day longer at the median, 1.0 day longer at the 75th percentile, and 1.9 days longer at the 90th percentile. For both cost and LOS models, estimates remained stable after adjusting for age and combined comorbidity score (Table 2).

This quality improvement initiative evaluated which ACE admissions yielded the greatest value and found the largest reductions in LOS and cost in patients with the greatest comorbidity scores (frequently referred to as “high need, high cost”).^23,24 Based on prior literature, we had anticipated that moderate risk patients would show the maximum benefit.^15,25 In contrast to our findings, a University of Alabama (UAB) ACE program subgroup analysis using the CMS Case Mix Index (CMI) found a cost reduction for patients with low or intermediate CMI scores but not for those with high scores.¹⁵ The Hospital Elder Life Program (HELP) has yielded maximal impact for patients at moderate risk for delirium.²⁶ Our results are supported by a University of Texas, Houston, study revealing lower LOS and cost for ACE patients, despite high CMI scores and endemic frailty, although it did not report outcomes across a range of comorbidities or costs.²⁷ Our results may be determined by the specific characteristics of the Baystate ACE initiative. Our emphasis on considering prognosis and encouraging advance care planning could have contributed to the improved metrics for more complicated patients. It is possible that patients with high comorbidity burden were more likely to screen in with the surprise question, leading to more frequent goals of care discussions by the hospitalists or geriatrics team, which, in turn, may have resulted in less aggressive care and consequently lower costs. The emphasis on prognosis and palliative care was not a feature of the UAB or Texas studies. Additional components, such as the delirium screening and the presence of volunteer advocates, could also have impacted the results. Our tiered approach during rounds with rapid reviews for most patients and longer discussions for those at highest risk may have further contributed to the findings. Finally, although we did not track the recommendation acceptance rate for the entire study period, in the first 9 months of the project, 9,325 recommendations were made with an acceptance rate of >85%. We previously reported a similar acceptance rate for medication recommendations.²⁸ Another factor contributing to our results may be the ways in which we categorized patients and calculated costs. We used the Gagne combined comorbidity score, which includes only prior conditions;²¹ the UAB study used CMI, which includes severity of presenting illness and complications, as well as baseline comorbidities. We also compared total cost, while UAB reported variable direct cost.

This study has a number of limitations. First, it was conducted at a single site and may not apply to other hospitals. Second, as a quality improvement program, its design, processes, and personnel evolved over time, and, as in any multicomponent initiative, the effect of individual factors on the outcomes is unknown. Third, this is an observational study with the aim of generating hypotheses for more rigorous studies in the future and residual confounding factors may exist despite efforts to adjust for variables present in an administrative database. Thus, we were unable to completely adjust for potentially important social factors, presence of delirium, or baseline functional status.

To our knowledge, this study is the first report on the differential impact of comorbidity scores and cost distribution on ACE total cost and LOS reductions. Despite its limitations, it contributes to the existing literature by suggesting that the Gagne comorbidity score can help identify which admissions will yield the greatest value. The Gagne score could be calculated at admission using the ePrognosis risk calculator or incorporated and automated in the EMR.²⁹ Many health systems are reluctant to designate beds for specific subpopulations since doing so decreases flexibility and complicates the admission process. A dynamic tension exists between increasing income streams now and generating future savings by supporting initiatives with upfront costs. Other successful acute care geriatrics programs, such as NICHE,³⁰ HELP,³¹ MACE,³² and consultation teams, exist.³³ Studies reporting the outcomes of combining ACE units with these other approaches in a “portfolio approach” will inform the design of the most efficient and impactful programs.³⁴ Scrupulous attention to symptom control and advance care planning are key features of our program, and, given the high prevalence of advanced serious illness in hospitalized older adults, this consideration may be critical for success.

As ACE units can only care for a small fraction of hospitalized older adults, determining which patients will maximally benefit from the structured, team-based care on ACE units is crucial. We found that the greatest impact on LOS and costs occurred in the subgroup with the highest comorbidity scores and overall cost. ACE care for the most vulnerable patients appeared to yield the greatest value for the system; thus, these older adults may need to be prioritized for admission. This improvement may enhance quality and value outcomes, maximize a scarce resource, and secure results needed to sustain the “clinician-led and data-driven” ACE model in the face of changing clinical and financial landscapes.³⁵

Acknowledgments

All those with significant contributions to this work are included as authors.

The authors express their deep appreciation to all their Baystate collaborators, particularly to Rebecca Starr, MD, the first geriatrics medical director of the program, Ms. Virginia Chipps, RN, the program’s first nurse manager, and Tasmiah Chowdhury, PharmD, the program’s first pharmacist. We are also deeply grateful to those persons who provided programmatic advice and input on model ACE programs elsewhere, including Kyle Allen, MD, Michael Malone MD, Robert Palmer MD, and, especially, Kellie Flood, MD.

Disclosures

None of the authors have any existing or potential personal or financial conflicts relevant to this paper to report.

Funding

This work was supported in part by a Geriatric Workforce Enhancement Program award (grant # U1QHP28702) from the Health Resources and Services Administration and by internal support from Baystate Health

In the1990s, gabapentin was licensed in the United States as an anticonvulsant and it became widely successful in the mid-2000s when marketed for the treatment of pain. Since then, prescriptions for gabapentinoids have accelerated dramatically.^1,2 Between 2012 and 2016, the total spending on pregabalin in the United States increased from $1.9 to $4.4 billion, with pregabalin ranking eighth overall for specific drug spending.³

Despite a finite number of indications, there has been a steady rise in off-label use, with an increased risk of adverse drug events (ADEs).^4,5 Several meta-analyses suggest either low-quality or no evidence of benefit for gabapentinoid use in settings including neuropathic pain in cancer, sciatica, and chronic low back pain.^6-8 Lack of efficacy is compounded by adverse effects such as altered mental status, fluid retention, sedation, and increased risk of traumatic falls in older adults.^6,9,10 Finally, dependency is a concern; opioids are coprescribed in up to 50% of patients,¹¹ increasing the odds of opioid-related death by up to 60%.¹²

To better characterize gabapentinoid use in hospitalized patients, we analyzed a retrospective cohort of patients admitted to our tertiary care medical teaching unit, examining preadmission and in-hospital prescribing trends, off-label use, and deprescribing.

Patient data were collected from a retrospective cohort, including all consecutive admissions to our 52-bed medical clinical teaching unit in Montréal, Canada, since December 2013.¹³ We reviewed admissions between December 17, 2013 and June 30, 2017 and identified three populations of gabapentinoid users from medication reconciliation documents: preadmission users continued at discharge, preadmission users deprescribed in hospital, and new in-hospital users continued at discharge. Deprescribing was defined as having the drug stopped at discharge or a prescribed taper that included stopping. The term “gabapentinoid users” refers to preadmission gabapentinoid use.

Gabapentinoid users were compared with nonusers with regard to demographic characteristics; select comorbidities; coprescription of opioids, benzodiazepines, and Z-drugs; length of stay (LOS); and inpatient mortality. Only the first eligible admission per patient was considered. Patients who had multiple admissions over the period of interest were classified as “users” in the patient-level analyses if they were taking a gabapentinoid at home or at discharge on at least one admission.

Doses and indications were collected from medication reconciliation performed by a clinical pharmacist, which included an interview with the patient or a proxy and a review of the indications for all drugs. These data were merged with any additional potential indications found in the admission notes (listing all chronic conditions from a detailed medical history) and review of the electronic medical record. The US Food and Drug Administration (FDA) approved the indications and the recommended doses were taken from product monographs and compared with doses prescribed to patients. When documented, the reason for new prescriptions and justification for deprescribing at discharge were manually abstracted from discharge summaries and medication reconciliation documents.

Continuous variables were expressed as median and interquartile range (IQR) and compared using the Wilcoxon rank-sum test. Categorical variables were compared using the χ² test. Proportions of gabapentinoid use and deprescribing, including 95% confidence intervals around each proportion, were plotted and linear regression was performed versus fiscal quarter to evaluate for temporal trends. A two-sided α value of 0.05 was considered to be statistically significant. Statistical analyses were performed using Stata version 15 (StataCorp LLC, College Station, Texas). The McGill University Health Centre Research Ethics Board approved this study.

A total of 4,103 unique patients were admitted from December 2013 to July 2017, of whom 550 (13.4%) were receiving a gabapentinoid before admission. Two preadmission users were coprescribed gabapentin and pregabalin for a total of 552 prescriptions. The prevalence of preadmission gabapentinoid use remained steady during the period of interest (Appendix 1; P = .29 for temporal trend). There were no significant differences between gabapentinoid users and nonusers with regard to age or sex, but users had a higher prevalence of chronic disease (Table 1). In addition, compared with nonusers, gabapentinoid users were more likely to be coprescribed opioids (28.2% vs 12%; P < .01), benzodiazepines (24.5% vs 14.3%; P < .01), and nonbenzodiazepine sedative hypnotics (7.5% vs 3.6%; P < .01). Of note, 10.2% of gabapentinoid users were simultaneously coprescribed both opioids and benzodiazepines versus. 3.6% of nonusers (P < .01; Table 1). Nonetheless, there was no statistically significant difference between users and nonusers with regard to inpatient mortality (10% vs 12%; P = .17).

The indications for gabapentinoid use are presented in Table 2. Only a minority (17% or 94/552) had an approved indication. Among these 94 patients, 38 (40%) received FDA-recommended doses, 47 (50%) received doses below those demonstrated to be effective, and 9 (10%) received higher-than-recommended doses. New prescriptions at discharge were observed in 1.5% of patients, with the majority given for off-label indications (Appendix 2).

In this large cohort study of hospitalized medical patients, preadmission gabapentinoid use was present in one in every eight admitted patients. Most patients had off-label indications, including the small number of patients who had the drug started in hospital. Even for approved indications, the doses were often lower than what trials have suggested to be effective. Finally, although we have demonstrated that deprescribing occurred, it was uncommon and either precipitated by an adverse event or the justification was poorly documented.

To our knowledge, our study is one of the first to examine what happens to gabapentinoids in hospitalized patients and we present important new data with respect to dosing and prescribing patterns. The low rates of discontinuation, intent to taper, or dose decreases in our cohort represent a potential area of improvement in deprescribing.

Deprescribing should be considered for patients with serious adverse events, for whom less serious adverse effects preclude achieving clinically effective doses, and for those who do not perceive benefit. Given the magnitude of the problems presented by polypharmacy, we propose that stopping priority be given to off-label use (especially when clinically ineffective) and for patients coprescribed opioids or sedatives. Up to a third of users in our cohort were coprescribed opioids or benzodiazepines, which is particularly concerning given the association with increased opioid-related mortality.^12,15 Although we did not observe a difference in inpatient mortality, such a study is underpowered for this outcome especially when considering the competing risks of death in hospital. Importantly, when deprescribing, the drug should be tapered over several weeks to limit symptoms of withdrawal and to prevent seizure.¹¹

Presumed off-label use and subtherapeutic doses were common in our cohort, with only 17% of users having a clearly documented FDA-approved indication, in agreement with a previous study that reported only 5% on-label use.⁴ High doses of gabapentinoids required for efficacy in clinical trials may be difficult to achieve because of dose-limiting side effects, which may explain the relatively low median doses recorded in our real-world cohort. Another possibility is that frail, older patients with renal dysfunction experience effectiveness at lower median doses than those quoted from study populations. In our study, patients on lower doses of gabapentinoids had a higher prevalence of stage IV or V chronic kidney disease (CKD). Stage IV/V CKD was identified in 16/47 (34.0%) patients on lower doses of gabapentinoids, compared to 4/38 (10.5%) on doses within the FDA-recommended range.

Our study has limitations; findings from a single Canadian tertiary care hospital may not be generalizable to other hospitals or countries, particularly given the differences between the Canadian and US health systems. Indications were extracted from the patient chart and even with the best possible medication history and thorough review, sometimes they had to be inferred. Caution should also be exercised when interpreting the omission of an indication as equating to a lack of justifiable medication use; however, the rate of off-label use in our cohort is in agreement with prior research.⁴ Moreover, with a retrospective design, the effectiveness of the drug on an individual basis could not be assessed, which would have allowed a more precise estimate of the proportion of patients for whom deprescribing might have been appropriate. The strengths of this study include a large sample of real-world, heterogeneous, general medical patients spanning several years and our use of trained pharmacists and physicians to determine the drug indication as opposed to reliance on administrative data.

Gabapentinoid use was frequent in our cohort of hospitalized medical patients, with a high prevalence of off-label use, subtherapeutic doses, and coadministration with opioids and benzodiazepines. Deprescribing at discharge was uncommon and often triggered by an adverse event. The identification of gabapentinoids during hospitalization is an opportunity to reevaluate the indication for the drug, assess for effectiveness, and consider deprescribing to help reduce polypharmacy and ideally ADEs.

Acknowledgment

For the purposes of authorship, Dr. McDonald and Dr. Lee contributed equally.

Disclosures

Dr. Emily McDonald and Dr. Todd Lee have a patent pending for MedSafer, a deprescribing software, and both receive research salary support from the Fonds de Recherche Santé du Québec. Dr. Gingras, Dr. Lieu, and Dr. Papillon-Ferland have nothing to disclose.

In the1990s, gabapentin was licensed in the United States as an anticonvulsant and it became widely successful in the mid-2000s when marketed for the treatment of pain. Since then, prescriptions for gabapentinoids have accelerated dramatically.^1,2 Between 2012 and 2016, the total spending on pregabalin in the United States increased from $1.9 to $4.4 billion, with pregabalin ranking eighth overall for specific drug spending.³

Despite a finite number of indications, there has been a steady rise in off-label use, with an increased risk of adverse drug events (ADEs).^4,5 Several meta-analyses suggest either low-quality or no evidence of benefit for gabapentinoid use in settings including neuropathic pain in cancer, sciatica, and chronic low back pain.^6-8 Lack of efficacy is compounded by adverse effects such as altered mental status, fluid retention, sedation, and increased risk of traumatic falls in older adults.^6,9,10 Finally, dependency is a concern; opioids are coprescribed in up to 50% of patients,¹¹ increasing the odds of opioid-related death by up to 60%.¹²

To better characterize gabapentinoid use in hospitalized patients, we analyzed a retrospective cohort of patients admitted to our tertiary care medical teaching unit, examining preadmission and in-hospital prescribing trends, off-label use, and deprescribing.

Patient data were collected from a retrospective cohort, including all consecutive admissions to our 52-bed medical clinical teaching unit in Montréal, Canada, since December 2013.¹³ We reviewed admissions between December 17, 2013 and June 30, 2017 and identified three populations of gabapentinoid users from medication reconciliation documents: preadmission users continued at discharge, preadmission users deprescribed in hospital, and new in-hospital users continued at discharge. Deprescribing was defined as having the drug stopped at discharge or a prescribed taper that included stopping. The term “gabapentinoid users” refers to preadmission gabapentinoid use.

Gabapentinoid users were compared with nonusers with regard to demographic characteristics; select comorbidities; coprescription of opioids, benzodiazepines, and Z-drugs; length of stay (LOS); and inpatient mortality. Only the first eligible admission per patient was considered. Patients who had multiple admissions over the period of interest were classified as “users” in the patient-level analyses if they were taking a gabapentinoid at home or at discharge on at least one admission.

Doses and indications were collected from medication reconciliation performed by a clinical pharmacist, which included an interview with the patient or a proxy and a review of the indications for all drugs. These data were merged with any additional potential indications found in the admission notes (listing all chronic conditions from a detailed medical history) and review of the electronic medical record. The US Food and Drug Administration (FDA) approved the indications and the recommended doses were taken from product monographs and compared with doses prescribed to patients. When documented, the reason for new prescriptions and justification for deprescribing at discharge were manually abstracted from discharge summaries and medication reconciliation documents.

Continuous variables were expressed as median and interquartile range (IQR) and compared using the Wilcoxon rank-sum test. Categorical variables were compared using the χ² test. Proportions of gabapentinoid use and deprescribing, including 95% confidence intervals around each proportion, were plotted and linear regression was performed versus fiscal quarter to evaluate for temporal trends. A two-sided α value of 0.05 was considered to be statistically significant. Statistical analyses were performed using Stata version 15 (StataCorp LLC, College Station, Texas). The McGill University Health Centre Research Ethics Board approved this study.

A total of 4,103 unique patients were admitted from December 2013 to July 2017, of whom 550 (13.4%) were receiving a gabapentinoid before admission. Two preadmission users were coprescribed gabapentin and pregabalin for a total of 552 prescriptions. The prevalence of preadmission gabapentinoid use remained steady during the period of interest (Appendix 1; P = .29 for temporal trend). There were no significant differences between gabapentinoid users and nonusers with regard to age or sex, but users had a higher prevalence of chronic disease (Table 1). In addition, compared with nonusers, gabapentinoid users were more likely to be coprescribed opioids (28.2% vs 12%; P < .01), benzodiazepines (24.5% vs 14.3%; P < .01), and nonbenzodiazepine sedative hypnotics (7.5% vs 3.6%; P < .01). Of note, 10.2% of gabapentinoid users were simultaneously coprescribed both opioids and benzodiazepines versus. 3.6% of nonusers (P < .01; Table 1). Nonetheless, there was no statistically significant difference between users and nonusers with regard to inpatient mortality (10% vs 12%; P = .17).

The indications for gabapentinoid use are presented in Table 2. Only a minority (17% or 94/552) had an approved indication. Among these 94 patients, 38 (40%) received FDA-recommended doses, 47 (50%) received doses below those demonstrated to be effective, and 9 (10%) received higher-than-recommended doses. New prescriptions at discharge were observed in 1.5% of patients, with the majority given for off-label indications (Appendix 2).

In this large cohort study of hospitalized medical patients, preadmission gabapentinoid use was present in one in every eight admitted patients. Most patients had off-label indications, including the small number of patients who had the drug started in hospital. Even for approved indications, the doses were often lower than what trials have suggested to be effective. Finally, although we have demonstrated that deprescribing occurred, it was uncommon and either precipitated by an adverse event or the justification was poorly documented.

To our knowledge, our study is one of the first to examine what happens to gabapentinoids in hospitalized patients and we present important new data with respect to dosing and prescribing patterns. The low rates of discontinuation, intent to taper, or dose decreases in our cohort represent a potential area of improvement in deprescribing.

Deprescribing should be considered for patients with serious adverse events, for whom less serious adverse effects preclude achieving clinically effective doses, and for those who do not perceive benefit. Given the magnitude of the problems presented by polypharmacy, we propose that stopping priority be given to off-label use (especially when clinically ineffective) and for patients coprescribed opioids or sedatives. Up to a third of users in our cohort were coprescribed opioids or benzodiazepines, which is particularly concerning given the association with increased opioid-related mortality.^12,15 Although we did not observe a difference in inpatient mortality, such a study is underpowered for this outcome especially when considering the competing risks of death in hospital. Importantly, when deprescribing, the drug should be tapered over several weeks to limit symptoms of withdrawal and to prevent seizure.¹¹

Presumed off-label use and subtherapeutic doses were common in our cohort, with only 17% of users having a clearly documented FDA-approved indication, in agreement with a previous study that reported only 5% on-label use.⁴ High doses of gabapentinoids required for efficacy in clinical trials may be difficult to achieve because of dose-limiting side effects, which may explain the relatively low median doses recorded in our real-world cohort. Another possibility is that frail, older patients with renal dysfunction experience effectiveness at lower median doses than those quoted from study populations. In our study, patients on lower doses of gabapentinoids had a higher prevalence of stage IV or V chronic kidney disease (CKD). Stage IV/V CKD was identified in 16/47 (34.0%) patients on lower doses of gabapentinoids, compared to 4/38 (10.5%) on doses within the FDA-recommended range.

Our study has limitations; findings from a single Canadian tertiary care hospital may not be generalizable to other hospitals or countries, particularly given the differences between the Canadian and US health systems. Indications were extracted from the patient chart and even with the best possible medication history and thorough review, sometimes they had to be inferred. Caution should also be exercised when interpreting the omission of an indication as equating to a lack of justifiable medication use; however, the rate of off-label use in our cohort is in agreement with prior research.⁴ Moreover, with a retrospective design, the effectiveness of the drug on an individual basis could not be assessed, which would have allowed a more precise estimate of the proportion of patients for whom deprescribing might have been appropriate. The strengths of this study include a large sample of real-world, heterogeneous, general medical patients spanning several years and our use of trained pharmacists and physicians to determine the drug indication as opposed to reliance on administrative data.

Gabapentinoid use was frequent in our cohort of hospitalized medical patients, with a high prevalence of off-label use, subtherapeutic doses, and coadministration with opioids and benzodiazepines. Deprescribing at discharge was uncommon and often triggered by an adverse event. The identification of gabapentinoids during hospitalization is an opportunity to reevaluate the indication for the drug, assess for effectiveness, and consider deprescribing to help reduce polypharmacy and ideally ADEs.

Acknowledgment

For the purposes of authorship, Dr. McDonald and Dr. Lee contributed equally.

Disclosures

Dr. Emily McDonald and Dr. Todd Lee have a patent pending for MedSafer, a deprescribing software, and both receive research salary support from the Fonds de Recherche Santé du Québec. Dr. Gingras, Dr. Lieu, and Dr. Papillon-Ferland have nothing to disclose.

In the1990s, gabapentin was licensed in the United States as an anticonvulsant and it became widely successful in the mid-2000s when marketed for the treatment of pain. Since then, prescriptions for gabapentinoids have accelerated dramatically.^1,2 Between 2012 and 2016, the total spending on pregabalin in the United States increased from $1.9 to $4.4 billion, with pregabalin ranking eighth overall for specific drug spending.³

Despite a finite number of indications, there has been a steady rise in off-label use, with an increased risk of adverse drug events (ADEs).^4,5 Several meta-analyses suggest either low-quality or no evidence of benefit for gabapentinoid use in settings including neuropathic pain in cancer, sciatica, and chronic low back pain.^6-8 Lack of efficacy is compounded by adverse effects such as altered mental status, fluid retention, sedation, and increased risk of traumatic falls in older adults.^6,9,10 Finally, dependency is a concern; opioids are coprescribed in up to 50% of patients,¹¹ increasing the odds of opioid-related death by up to 60%.¹²

To better characterize gabapentinoid use in hospitalized patients, we analyzed a retrospective cohort of patients admitted to our tertiary care medical teaching unit, examining preadmission and in-hospital prescribing trends, off-label use, and deprescribing.

Patient data were collected from a retrospective cohort, including all consecutive admissions to our 52-bed medical clinical teaching unit in Montréal, Canada, since December 2013.¹³ We reviewed admissions between December 17, 2013 and June 30, 2017 and identified three populations of gabapentinoid users from medication reconciliation documents: preadmission users continued at discharge, preadmission users deprescribed in hospital, and new in-hospital users continued at discharge. Deprescribing was defined as having the drug stopped at discharge or a prescribed taper that included stopping. The term “gabapentinoid users” refers to preadmission gabapentinoid use.

Gabapentinoid users were compared with nonusers with regard to demographic characteristics; select comorbidities; coprescription of opioids, benzodiazepines, and Z-drugs; length of stay (LOS); and inpatient mortality. Only the first eligible admission per patient was considered. Patients who had multiple admissions over the period of interest were classified as “users” in the patient-level analyses if they were taking a gabapentinoid at home or at discharge on at least one admission.

Doses and indications were collected from medication reconciliation performed by a clinical pharmacist, which included an interview with the patient or a proxy and a review of the indications for all drugs. These data were merged with any additional potential indications found in the admission notes (listing all chronic conditions from a detailed medical history) and review of the electronic medical record. The US Food and Drug Administration (FDA) approved the indications and the recommended doses were taken from product monographs and compared with doses prescribed to patients. When documented, the reason for new prescriptions and justification for deprescribing at discharge were manually abstracted from discharge summaries and medication reconciliation documents.

Continuous variables were expressed as median and interquartile range (IQR) and compared using the Wilcoxon rank-sum test. Categorical variables were compared using the χ² test. Proportions of gabapentinoid use and deprescribing, including 95% confidence intervals around each proportion, were plotted and linear regression was performed versus fiscal quarter to evaluate for temporal trends. A two-sided α value of 0.05 was considered to be statistically significant. Statistical analyses were performed using Stata version 15 (StataCorp LLC, College Station, Texas). The McGill University Health Centre Research Ethics Board approved this study.

A total of 4,103 unique patients were admitted from December 2013 to July 2017, of whom 550 (13.4%) were receiving a gabapentinoid before admission. Two preadmission users were coprescribed gabapentin and pregabalin for a total of 552 prescriptions. The prevalence of preadmission gabapentinoid use remained steady during the period of interest (Appendix 1; P = .29 for temporal trend). There were no significant differences between gabapentinoid users and nonusers with regard to age or sex, but users had a higher prevalence of chronic disease (Table 1). In addition, compared with nonusers, gabapentinoid users were more likely to be coprescribed opioids (28.2% vs 12%; P < .01), benzodiazepines (24.5% vs 14.3%; P < .01), and nonbenzodiazepine sedative hypnotics (7.5% vs 3.6%; P < .01). Of note, 10.2% of gabapentinoid users were simultaneously coprescribed both opioids and benzodiazepines versus. 3.6% of nonusers (P < .01; Table 1). Nonetheless, there was no statistically significant difference between users and nonusers with regard to inpatient mortality (10% vs 12%; P = .17).

The indications for gabapentinoid use are presented in Table 2. Only a minority (17% or 94/552) had an approved indication. Among these 94 patients, 38 (40%) received FDA-recommended doses, 47 (50%) received doses below those demonstrated to be effective, and 9 (10%) received higher-than-recommended doses. New prescriptions at discharge were observed in 1.5% of patients, with the majority given for off-label indications (Appendix 2).

In this large cohort study of hospitalized medical patients, preadmission gabapentinoid use was present in one in every eight admitted patients. Most patients had off-label indications, including the small number of patients who had the drug started in hospital. Even for approved indications, the doses were often lower than what trials have suggested to be effective. Finally, although we have demonstrated that deprescribing occurred, it was uncommon and either precipitated by an adverse event or the justification was poorly documented.

To our knowledge, our study is one of the first to examine what happens to gabapentinoids in hospitalized patients and we present important new data with respect to dosing and prescribing patterns. The low rates of discontinuation, intent to taper, or dose decreases in our cohort represent a potential area of improvement in deprescribing.

Deprescribing should be considered for patients with serious adverse events, for whom less serious adverse effects preclude achieving clinically effective doses, and for those who do not perceive benefit. Given the magnitude of the problems presented by polypharmacy, we propose that stopping priority be given to off-label use (especially when clinically ineffective) and for patients coprescribed opioids or sedatives. Up to a third of users in our cohort were coprescribed opioids or benzodiazepines, which is particularly concerning given the association with increased opioid-related mortality.^12,15 Although we did not observe a difference in inpatient mortality, such a study is underpowered for this outcome especially when considering the competing risks of death in hospital. Importantly, when deprescribing, the drug should be tapered over several weeks to limit symptoms of withdrawal and to prevent seizure.¹¹

Presumed off-label use and subtherapeutic doses were common in our cohort, with only 17% of users having a clearly documented FDA-approved indication, in agreement with a previous study that reported only 5% on-label use.⁴ High doses of gabapentinoids required for efficacy in clinical trials may be difficult to achieve because of dose-limiting side effects, which may explain the relatively low median doses recorded in our real-world cohort. Another possibility is that frail, older patients with renal dysfunction experience effectiveness at lower median doses than those quoted from study populations. In our study, patients on lower doses of gabapentinoids had a higher prevalence of stage IV or V chronic kidney disease (CKD). Stage IV/V CKD was identified in 16/47 (34.0%) patients on lower doses of gabapentinoids, compared to 4/38 (10.5%) on doses within the FDA-recommended range.

Our study has limitations; findings from a single Canadian tertiary care hospital may not be generalizable to other hospitals or countries, particularly given the differences between the Canadian and US health systems. Indications were extracted from the patient chart and even with the best possible medication history and thorough review, sometimes they had to be inferred. Caution should also be exercised when interpreting the omission of an indication as equating to a lack of justifiable medication use; however, the rate of off-label use in our cohort is in agreement with prior research.⁴ Moreover, with a retrospective design, the effectiveness of the drug on an individual basis could not be assessed, which would have allowed a more precise estimate of the proportion of patients for whom deprescribing might have been appropriate. The strengths of this study include a large sample of real-world, heterogeneous, general medical patients spanning several years and our use of trained pharmacists and physicians to determine the drug indication as opposed to reliance on administrative data.

Gabapentinoid use was frequent in our cohort of hospitalized medical patients, with a high prevalence of off-label use, subtherapeutic doses, and coadministration with opioids and benzodiazepines. Deprescribing at discharge was uncommon and often triggered by an adverse event. The identification of gabapentinoids during hospitalization is an opportunity to reevaluate the indication for the drug, assess for effectiveness, and consider deprescribing to help reduce polypharmacy and ideally ADEs.

Acknowledgment

For the purposes of authorship, Dr. McDonald and Dr. Lee contributed equally.

Disclosures

Dr. Emily McDonald and Dr. Todd Lee have a patent pending for MedSafer, a deprescribing software, and both receive research salary support from the Fonds de Recherche Santé du Québec. Dr. Gingras, Dr. Lieu, and Dr. Papillon-Ferland have nothing to disclose.

Neonatal herpes simplex virus (HSV) is associated with significant morbidity and mortality,¹ particularly when the diagnosis or treatment is delayed.² Therefore, many infants aged ≤60 days being evaluated for meningitis undergo cerebrospinal fluid (CSF) HSV polymerase chain reaction (PCR) testing even though the risk of HSV infection is low [estimated at 0.4% of those undergoing evaluation for central nervous system (CNS) infection].³ A single-center study demonstrated that CSF HSV PCR testing increases the hospital length of stay (LOS) for infants aged ≤56 days,⁴ although these single-center findings may not be generalizable. To this end, we measured the association between CSF HSV PCR testing and LOS in a multicenter cohort of hospitalized young infants.

Study Design

We conducted a planned secondary analysis of a retrospective cohort of infants aged ≤60 days who presented to the emergency department (ED) between January 1, 2005 and December 31, 2013, enrolled in the Pediatric Emergency Medicine Collaborative Research Committee (PEM CRC) HSV study.³ Our study was limited to the 20 hospitals that contributed hospital LOS data. The study protocol was approved by each site’s institutional review board with permission for data sharing.

Study Population

Eligible infants were identified at each site using a site-specific electronic search strategy. Infants were eligible for inclusion if a CSF culture was obtained in the ED or within 24 hours of ED arrival. We excluded infants who were discharged from the ED and those with missing hospital LOS data.

Data Collection

Site investigators extracted the following data elements either electronically or from medical records: patient demographics; ED arrival date and time; hospital discharge date and time; urinalysis results; peripheral and CSF cell counts; blood, urine, and CSF bacterial culture results; as well as the results of HSV PCR and viral cultures. Infants with growth of a pathogen in blood or CSF, or a catheterized urine culture with ≥50,000 colony-forming units (CFUs)/mL of a single pathogenic bacteria, or 10,000-50,000 CFUs/mL of a single pathogenic bacteria with an abnormal urinalysis (ie, positive nitrite or leukocyte esterase on urine dipstick or >5 white blood cells [WBCs] per high power field on urine microscopy) were classified as having a serious bacterial infection (SBI).^5,6 Infants with a positive HSV PCR or viral culture from any site were classified as having HSV infection.³ Hospitalized infants who did not have an HSV PCR test performed were assumed not to have HSV disease if not diagnosed during the hospital stay or repeat ED encounter.³

Outcome Measures

The primary outcome was hospital LOS, defined at all hospitals as the time from ED arrival to provider signature of the hospital discharge order, calculated in minutes and then converted into days.

Statistical Analysis

We described LOS using medians with interquartile ranges (IQR) and compared between infants with and without a CSF HSV PCR test performed using the Mann–Whitney U test. To evaluate the association between performance of CSF HSV PCR testing and hospital LOS, we used negative binomial regression given the count variable outcome (LOS) with an overdispersed distribution. For this analysis, we clustered by hospital after adjusting for the following factors determined a priori: age, gender, study year, and presence of serious bacterial or HSV infection. Using the relative marginal modeled estimates of LOS (tested vs not tested), we determined the percentage increase in LOS. We then repeated the analyses after stratifying by the location of testing (ie, in-house vs send-out), age (≤28 days vs 29-60 days), and presence or absence of CSF pleocytosis (defined as a CSF WBC of ≥16 cells/mm³for infants aged ≤28 days and ≥10 cells/mm³for infants aged 29-60 days),⁷ because infants aged 29-60 days and those without CSF pleocytosis are reported to be at very low risk for CNS HSV infection.^3,8 We utilized Stata Data Analysis and Statistical Software, version 15.0 (StataCorp, Inc.; College Station, Texas) for statistical analyses.

Of 24,103 infants with CSF cultures obtained at the 20 participating sites, we excluded 2,673 (11.1%) discharged from the ED or with missing disposition and 934 (3.9%) with missing LOS, leaving a study cohort of 20,496 infants (Figure). Overall, 1,780 infants (8.7%) had an SBI and 99 (0.5%) had an HSV infection, of which 46 (46.5%) had a CNS HSV infection.

Among the 20,496 study infants, 7,399 (36.1%) had a CSF HSV PCR test performed; 5,935 infants (80.2% of those tested) had in-house and 1,464 (19.8%) had send-out testing. Among infants with available CSF cell counts, a CSF HSV PCR test was more commonly performed in infants with CSF pleocytosis than in those without (1,865/4,439 [42.0%] with CSF pleocytosis vs 3,705/12,002 [30.9%] without CSF pleocytosis; odds ratio [OR] 1.6, 95% CI 1.5-1.7). Of the 7,399 infants who had a CSF HSV PCR test performed, 46 (0.6%) had a positive test. Of the tested infants, 5,570 (75.3%) had an available CSF WBC count; a positive CSF HSV PCR test was more common in infants with CSF pleocytosis than in those without (25 positive tests/1,865 infants with CSF pleocytosis [1.3%] vs 9/3,705 [0.2%] without CSF pleocytosis; OR 5.6, 95% CI 2.6-12.0). Among the 5,308 infants aged 29-60 days without CSF pleocytosis, 1,110 (20.9%) had a CSF HSV PCR test performed and only one infant (0.09% of those tested) had a positive test.

Without adjustment, infants with a CSF HSV PCR test had a longer median LOS than infants who were not tested (2.5 vs 2.3 days; P < .001). After adjustment, infants with a CSF HSV PCR test performed had a 23% longer duration of hospitalization. The association between testing and LOS was similar for older vs younger infants, infants with and without CSF pleocytosis, and in-house vs send-out testing (Table).

In a large, multicenter cohort of more than 20,000 hospitalized infants aged ≤60 days undergoing evaluation for meningitis, we examined the association of CSF HSV PCR testing with hospital LOS. Approximately one-third of study infants had a CSF HSV PCR test obtained. After adjustment for patient- and hospital-level factors, the treating clinician’s decision to obtain a CSF HSV PCR test was associated with a 23% longer hospital LOS (nearly one-half day).

Our findings are consistent with those of previous studies. First, our observed association of the decision to obtain a CSF HSV PCR test and LOS was similar in magnitude to that of a previous single-center investigation.⁴ Second, we also found that older infants and those without CSF pleocytosis were at very low risk of HSV infection.^3,8 For the otherwise low-risk infants, the longer LOS may be due to delays in obtaining CSF HSV PCR test results, which should be explored in future research. Our study has greater generalizability than previous single-center studies by substantially increasing the population size as well as the variety of clinical settings. Ensuring clinicians’ access to rapid HSV PCR testing platforms will further mitigate the impact of HSV testing on LOS.

When deciding to perform a CSF HSV PCR test for infants aged ≤60 days, clinicians must balance the low incidence of neonatal HSV³ with the risk of delayed diagnosis and treatment of HSV infection, which include neurologic sequelae or even death.^1,2 As infants with CNS HSV infection commonly present nonspecifically and only a minority of infected infants have skin vesicles,¹ controversy exists as to which infants should be evaluated for HSV infection, resulting in considerable variability in HSV testing.³ Some clinicians advocate for more conservative testing strategies that include the performance of CSF HSV PCR testing in all febrile infants aged ≤21 days.⁹ Others suggest limiting testing to infants who meet high-risk criteria (eg, seizures, ill-appearance, or CSF pleocytosis).^10,11 Further investigation will need to elucidate the clinical and laboratory predictors of HSV infection to identify those infants who would benefit most from HSV testing as well as the outcomes of infants not tested.

Our study has several limitations. First, we could not determine the reason why clinicians elected to obtain a CSF HSV PCR test, and we do not know the test turnaround time or the time when results became available to the clinical team. Second, we did not abstract clinical data such as ill-appearance or seizures. Although we adjusted for the presence of serious bacterial or HSV infection as proxy measures for illness severity, it is possible that other clinical factors were associated with HSV testing and LOS. Third, although we adjusted for patient- and hospital-level factors in our regression model, the potential for residual confounding persists. Fourth, we did not explore acyclovir administration as a factor associated with LOS as some sites did not provide data on acyclovir. Fifth, we did not evaluate the impact of HSV testing of other sample types (eg, blood or skin) on LOS. Sixth, our study was conducted primarily at children’s hospitals, and our findings may not be generalizable to general hospitals with hospitalized neonates.

Neonatal herpes simplex virus (HSV) is associated with significant morbidity and mortality,¹ particularly when the diagnosis or treatment is delayed.² Therefore, many infants aged ≤60 days being evaluated for meningitis undergo cerebrospinal fluid (CSF) HSV polymerase chain reaction (PCR) testing even though the risk of HSV infection is low [estimated at 0.4% of those undergoing evaluation for central nervous system (CNS) infection].³ A single-center study demonstrated that CSF HSV PCR testing increases the hospital length of stay (LOS) for infants aged ≤56 days,⁴ although these single-center findings may not be generalizable. To this end, we measured the association between CSF HSV PCR testing and LOS in a multicenter cohort of hospitalized young infants.

Study Design

We conducted a planned secondary analysis of a retrospective cohort of infants aged ≤60 days who presented to the emergency department (ED) between January 1, 2005 and December 31, 2013, enrolled in the Pediatric Emergency Medicine Collaborative Research Committee (PEM CRC) HSV study.³ Our study was limited to the 20 hospitals that contributed hospital LOS data. The study protocol was approved by each site’s institutional review board with permission for data sharing.

Study Population

Eligible infants were identified at each site using a site-specific electronic search strategy. Infants were eligible for inclusion if a CSF culture was obtained in the ED or within 24 hours of ED arrival. We excluded infants who were discharged from the ED and those with missing hospital LOS data.

Data Collection

Site investigators extracted the following data elements either electronically or from medical records: patient demographics; ED arrival date and time; hospital discharge date and time; urinalysis results; peripheral and CSF cell counts; blood, urine, and CSF bacterial culture results; as well as the results of HSV PCR and viral cultures. Infants with growth of a pathogen in blood or CSF, or a catheterized urine culture with ≥50,000 colony-forming units (CFUs)/mL of a single pathogenic bacteria, or 10,000-50,000 CFUs/mL of a single pathogenic bacteria with an abnormal urinalysis (ie, positive nitrite or leukocyte esterase on urine dipstick or >5 white blood cells [WBCs] per high power field on urine microscopy) were classified as having a serious bacterial infection (SBI).^5,6 Infants with a positive HSV PCR or viral culture from any site were classified as having HSV infection.³ Hospitalized infants who did not have an HSV PCR test performed were assumed not to have HSV disease if not diagnosed during the hospital stay or repeat ED encounter.³

Outcome Measures

The primary outcome was hospital LOS, defined at all hospitals as the time from ED arrival to provider signature of the hospital discharge order, calculated in minutes and then converted into days.

Statistical Analysis

We described LOS using medians with interquartile ranges (IQR) and compared between infants with and without a CSF HSV PCR test performed using the Mann–Whitney U test. To evaluate the association between performance of CSF HSV PCR testing and hospital LOS, we used negative binomial regression given the count variable outcome (LOS) with an overdispersed distribution. For this analysis, we clustered by hospital after adjusting for the following factors determined a priori: age, gender, study year, and presence of serious bacterial or HSV infection. Using the relative marginal modeled estimates of LOS (tested vs not tested), we determined the percentage increase in LOS. We then repeated the analyses after stratifying by the location of testing (ie, in-house vs send-out), age (≤28 days vs 29-60 days), and presence or absence of CSF pleocytosis (defined as a CSF WBC of ≥16 cells/mm³for infants aged ≤28 days and ≥10 cells/mm³for infants aged 29-60 days),⁷ because infants aged 29-60 days and those without CSF pleocytosis are reported to be at very low risk for CNS HSV infection.^3,8 We utilized Stata Data Analysis and Statistical Software, version 15.0 (StataCorp, Inc.; College Station, Texas) for statistical analyses.

Of 24,103 infants with CSF cultures obtained at the 20 participating sites, we excluded 2,673 (11.1%) discharged from the ED or with missing disposition and 934 (3.9%) with missing LOS, leaving a study cohort of 20,496 infants (Figure). Overall, 1,780 infants (8.7%) had an SBI and 99 (0.5%) had an HSV infection, of which 46 (46.5%) had a CNS HSV infection.

Among the 20,496 study infants, 7,399 (36.1%) had a CSF HSV PCR test performed; 5,935 infants (80.2% of those tested) had in-house and 1,464 (19.8%) had send-out testing. Among infants with available CSF cell counts, a CSF HSV PCR test was more commonly performed in infants with CSF pleocytosis than in those without (1,865/4,439 [42.0%] with CSF pleocytosis vs 3,705/12,002 [30.9%] without CSF pleocytosis; odds ratio [OR] 1.6, 95% CI 1.5-1.7). Of the 7,399 infants who had a CSF HSV PCR test performed, 46 (0.6%) had a positive test. Of the tested infants, 5,570 (75.3%) had an available CSF WBC count; a positive CSF HSV PCR test was more common in infants with CSF pleocytosis than in those without (25 positive tests/1,865 infants with CSF pleocytosis [1.3%] vs 9/3,705 [0.2%] without CSF pleocytosis; OR 5.6, 95% CI 2.6-12.0). Among the 5,308 infants aged 29-60 days without CSF pleocytosis, 1,110 (20.9%) had a CSF HSV PCR test performed and only one infant (0.09% of those tested) had a positive test.

Without adjustment, infants with a CSF HSV PCR test had a longer median LOS than infants who were not tested (2.5 vs 2.3 days; P < .001). After adjustment, infants with a CSF HSV PCR test performed had a 23% longer duration of hospitalization. The association between testing and LOS was similar for older vs younger infants, infants with and without CSF pleocytosis, and in-house vs send-out testing (Table).

In a large, multicenter cohort of more than 20,000 hospitalized infants aged ≤60 days undergoing evaluation for meningitis, we examined the association of CSF HSV PCR testing with hospital LOS. Approximately one-third of study infants had a CSF HSV PCR test obtained. After adjustment for patient- and hospital-level factors, the treating clinician’s decision to obtain a CSF HSV PCR test was associated with a 23% longer hospital LOS (nearly one-half day).

Our findings are consistent with those of previous studies. First, our observed association of the decision to obtain a CSF HSV PCR test and LOS was similar in magnitude to that of a previous single-center investigation.⁴ Second, we also found that older infants and those without CSF pleocytosis were at very low risk of HSV infection.^3,8 For the otherwise low-risk infants, the longer LOS may be due to delays in obtaining CSF HSV PCR test results, which should be explored in future research. Our study has greater generalizability than previous single-center studies by substantially increasing the population size as well as the variety of clinical settings. Ensuring clinicians’ access to rapid HSV PCR testing platforms will further mitigate the impact of HSV testing on LOS.

When deciding to perform a CSF HSV PCR test for infants aged ≤60 days, clinicians must balance the low incidence of neonatal HSV³ with the risk of delayed diagnosis and treatment of HSV infection, which include neurologic sequelae or even death.^1,2 As infants with CNS HSV infection commonly present nonspecifically and only a minority of infected infants have skin vesicles,¹ controversy exists as to which infants should be evaluated for HSV infection, resulting in considerable variability in HSV testing.³ Some clinicians advocate for more conservative testing strategies that include the performance of CSF HSV PCR testing in all febrile infants aged ≤21 days.⁹ Others suggest limiting testing to infants who meet high-risk criteria (eg, seizures, ill-appearance, or CSF pleocytosis).^10,11 Further investigation will need to elucidate the clinical and laboratory predictors of HSV infection to identify those infants who would benefit most from HSV testing as well as the outcomes of infants not tested.

Our study has several limitations. First, we could not determine the reason why clinicians elected to obtain a CSF HSV PCR test, and we do not know the test turnaround time or the time when results became available to the clinical team. Second, we did not abstract clinical data such as ill-appearance or seizures. Although we adjusted for the presence of serious bacterial or HSV infection as proxy measures for illness severity, it is possible that other clinical factors were associated with HSV testing and LOS. Third, although we adjusted for patient- and hospital-level factors in our regression model, the potential for residual confounding persists. Fourth, we did not explore acyclovir administration as a factor associated with LOS as some sites did not provide data on acyclovir. Fifth, we did not evaluate the impact of HSV testing of other sample types (eg, blood or skin) on LOS. Sixth, our study was conducted primarily at children’s hospitals, and our findings may not be generalizable to general hospitals with hospitalized neonates.

Neonatal herpes simplex virus (HSV) is associated with significant morbidity and mortality,¹ particularly when the diagnosis or treatment is delayed.² Therefore, many infants aged ≤60 days being evaluated for meningitis undergo cerebrospinal fluid (CSF) HSV polymerase chain reaction (PCR) testing even though the risk of HSV infection is low [estimated at 0.4% of those undergoing evaluation for central nervous system (CNS) infection].³ A single-center study demonstrated that CSF HSV PCR testing increases the hospital length of stay (LOS) for infants aged ≤56 days,⁴ although these single-center findings may not be generalizable. To this end, we measured the association between CSF HSV PCR testing and LOS in a multicenter cohort of hospitalized young infants.

Study Design

We conducted a planned secondary analysis of a retrospective cohort of infants aged ≤60 days who presented to the emergency department (ED) between January 1, 2005 and December 31, 2013, enrolled in the Pediatric Emergency Medicine Collaborative Research Committee (PEM CRC) HSV study.³ Our study was limited to the 20 hospitals that contributed hospital LOS data. The study protocol was approved by each site’s institutional review board with permission for data sharing.

Study Population

Eligible infants were identified at each site using a site-specific electronic search strategy. Infants were eligible for inclusion if a CSF culture was obtained in the ED or within 24 hours of ED arrival. We excluded infants who were discharged from the ED and those with missing hospital LOS data.

Data Collection

Site investigators extracted the following data elements either electronically or from medical records: patient demographics; ED arrival date and time; hospital discharge date and time; urinalysis results; peripheral and CSF cell counts; blood, urine, and CSF bacterial culture results; as well as the results of HSV PCR and viral cultures. Infants with growth of a pathogen in blood or CSF, or a catheterized urine culture with ≥50,000 colony-forming units (CFUs)/mL of a single pathogenic bacteria, or 10,000-50,000 CFUs/mL of a single pathogenic bacteria with an abnormal urinalysis (ie, positive nitrite or leukocyte esterase on urine dipstick or >5 white blood cells [WBCs] per high power field on urine microscopy) were classified as having a serious bacterial infection (SBI).^5,6 Infants with a positive HSV PCR or viral culture from any site were classified as having HSV infection.³ Hospitalized infants who did not have an HSV PCR test performed were assumed not to have HSV disease if not diagnosed during the hospital stay or repeat ED encounter.³

Outcome Measures

The primary outcome was hospital LOS, defined at all hospitals as the time from ED arrival to provider signature of the hospital discharge order, calculated in minutes and then converted into days.

Statistical Analysis

We described LOS using medians with interquartile ranges (IQR) and compared between infants with and without a CSF HSV PCR test performed using the Mann–Whitney U test. To evaluate the association between performance of CSF HSV PCR testing and hospital LOS, we used negative binomial regression given the count variable outcome (LOS) with an overdispersed distribution. For this analysis, we clustered by hospital after adjusting for the following factors determined a priori: age, gender, study year, and presence of serious bacterial or HSV infection. Using the relative marginal modeled estimates of LOS (tested vs not tested), we determined the percentage increase in LOS. We then repeated the analyses after stratifying by the location of testing (ie, in-house vs send-out), age (≤28 days vs 29-60 days), and presence or absence of CSF pleocytosis (defined as a CSF WBC of ≥16 cells/mm³for infants aged ≤28 days and ≥10 cells/mm³for infants aged 29-60 days),⁷ because infants aged 29-60 days and those without CSF pleocytosis are reported to be at very low risk for CNS HSV infection.^3,8 We utilized Stata Data Analysis and Statistical Software, version 15.0 (StataCorp, Inc.; College Station, Texas) for statistical analyses.

Of 24,103 infants with CSF cultures obtained at the 20 participating sites, we excluded 2,673 (11.1%) discharged from the ED or with missing disposition and 934 (3.9%) with missing LOS, leaving a study cohort of 20,496 infants (Figure). Overall, 1,780 infants (8.7%) had an SBI and 99 (0.5%) had an HSV infection, of which 46 (46.5%) had a CNS HSV infection.

Among the 20,496 study infants, 7,399 (36.1%) had a CSF HSV PCR test performed; 5,935 infants (80.2% of those tested) had in-house and 1,464 (19.8%) had send-out testing. Among infants with available CSF cell counts, a CSF HSV PCR test was more commonly performed in infants with CSF pleocytosis than in those without (1,865/4,439 [42.0%] with CSF pleocytosis vs 3,705/12,002 [30.9%] without CSF pleocytosis; odds ratio [OR] 1.6, 95% CI 1.5-1.7). Of the 7,399 infants who had a CSF HSV PCR test performed, 46 (0.6%) had a positive test. Of the tested infants, 5,570 (75.3%) had an available CSF WBC count; a positive CSF HSV PCR test was more common in infants with CSF pleocytosis than in those without (25 positive tests/1,865 infants with CSF pleocytosis [1.3%] vs 9/3,705 [0.2%] without CSF pleocytosis; OR 5.6, 95% CI 2.6-12.0). Among the 5,308 infants aged 29-60 days without CSF pleocytosis, 1,110 (20.9%) had a CSF HSV PCR test performed and only one infant (0.09% of those tested) had a positive test.

Without adjustment, infants with a CSF HSV PCR test had a longer median LOS than infants who were not tested (2.5 vs 2.3 days; P < .001). After adjustment, infants with a CSF HSV PCR test performed had a 23% longer duration of hospitalization. The association between testing and LOS was similar for older vs younger infants, infants with and without CSF pleocytosis, and in-house vs send-out testing (Table).

In a large, multicenter cohort of more than 20,000 hospitalized infants aged ≤60 days undergoing evaluation for meningitis, we examined the association of CSF HSV PCR testing with hospital LOS. Approximately one-third of study infants had a CSF HSV PCR test obtained. After adjustment for patient- and hospital-level factors, the treating clinician’s decision to obtain a CSF HSV PCR test was associated with a 23% longer hospital LOS (nearly one-half day).

Our findings are consistent with those of previous studies. First, our observed association of the decision to obtain a CSF HSV PCR test and LOS was similar in magnitude to that of a previous single-center investigation.⁴ Second, we also found that older infants and those without CSF pleocytosis were at very low risk of HSV infection.^3,8 For the otherwise low-risk infants, the longer LOS may be due to delays in obtaining CSF HSV PCR test results, which should be explored in future research. Our study has greater generalizability than previous single-center studies by substantially increasing the population size as well as the variety of clinical settings. Ensuring clinicians’ access to rapid HSV PCR testing platforms will further mitigate the impact of HSV testing on LOS.

When deciding to perform a CSF HSV PCR test for infants aged ≤60 days, clinicians must balance the low incidence of neonatal HSV³ with the risk of delayed diagnosis and treatment of HSV infection, which include neurologic sequelae or even death.^1,2 As infants with CNS HSV infection commonly present nonspecifically and only a minority of infected infants have skin vesicles,¹ controversy exists as to which infants should be evaluated for HSV infection, resulting in considerable variability in HSV testing.³ Some clinicians advocate for more conservative testing strategies that include the performance of CSF HSV PCR testing in all febrile infants aged ≤21 days.⁹ Others suggest limiting testing to infants who meet high-risk criteria (eg, seizures, ill-appearance, or CSF pleocytosis).^10,11 Further investigation will need to elucidate the clinical and laboratory predictors of HSV infection to identify those infants who would benefit most from HSV testing as well as the outcomes of infants not tested.

Our study has several limitations. First, we could not determine the reason why clinicians elected to obtain a CSF HSV PCR test, and we do not know the test turnaround time or the time when results became available to the clinical team. Second, we did not abstract clinical data such as ill-appearance or seizures. Although we adjusted for the presence of serious bacterial or HSV infection as proxy measures for illness severity, it is possible that other clinical factors were associated with HSV testing and LOS. Third, although we adjusted for patient- and hospital-level factors in our regression model, the potential for residual confounding persists. Fourth, we did not explore acyclovir administration as a factor associated with LOS as some sites did not provide data on acyclovir. Fifth, we did not evaluate the impact of HSV testing of other sample types (eg, blood or skin) on LOS. Sixth, our study was conducted primarily at children’s hospitals, and our findings may not be generalizable to general hospitals with hospitalized neonates.