Evidence Needing a Lift

In this issue of the Journal of Hospital Medicine, Hansen and colleagues provide a first, early look at the effectiveness of the BOOST intervention to reduce 30‐day readmissions among hospitalized patients.[1] BOOST[2] is 1 of a number of care transition improvement methodologies that have been applied to the problem of readmissions, each of which has evidence to support its effectiveness in its initial settings[3, 4] but has proven to be difficult to translate to other sites.[5, 6, 7]

BOOST stands in contrast with other, largely research protocol‐derived, programs in that it allows sites to tailor adoption of recommendations to local contexts and is therefore potentially more feasible to implement. Feasibility and practicality has led BOOST to be adopted in large national settings, even if it has had little evidence to support its effectiveness to date.

Given the nonstandardized and ad hoc nature of most multicenter collaboratives generally, and the flexibility of the BOOST model specifically, the BOOST authors are to be commended for undertaking any evaluation at all. Perhaps, not surprisingly, they encountered many of the problems associated with a multicenter studydropout of sites, problematic data, and limited evidence for adoption of the intervention at participating hospitals. Although these represent real‐world experiences of a quality‐improvement program, as a group they pose a number of problems that limit the study's robustness, and generate important caveats that readers should use to temper their interpretation of the authors' findings.

The first caveat relates to the substantial number of sites that either dropped out of BOOST or failed to submit data after enlisting in the collaborative. Although this may be common in quality improvement collaboratives, similar problems would not be permissible in a trial of a new drug or device. Dropout and selected ability to contribute data suggest that the ability to fully adopt BOOST may not be universal, and raises the possibility of bias, because the least successful sites may have had less interest in remaining engaged and submitting data.

The second caveat relates to how readmission rates were assessed. Because sites provided rates of readmissions at the unit level rather than the actual counts of admissions or readmissions, the authors were unable to conduct statistical analyses typically performed for these interventions, such as time series or difference‐in‐difference analyses. More importantly, one cannot discern whether their results are driven by a small absolute but large relative change in the number of readmissions at small sites. That is, large percentage changes of low statistical significance could have misleadingly affected the overall results. Conversely, we cannot identify large sites where a similar relative reduction could be statistically significant and more broadly interpreted as representing the real effectiveness of BOOST efforts.

The third caveat is in regard to the data describing the sites' performance. The effectiveness of BOOST in this analysis varied greatly among sites, with only 1 site showing a strong reduction in readmission rate, and nearly all others showing no statistical improvements. In fact, it appears that their overall results were almost entirely driven by the improvements at that 1 site.

Variable effectiveness of an intervention can be related to variable adoption or contextual factors (such as availability of personnel to implement the program). Although these authors have data on BOOST programmatic adoption, they do not have qualitative data on local barriers and facilitators to BOOST implementation, which at this stage of evaluation would be particularly valuable in understanding the results. Analyzing site‐level effectiveness is of growing relevance to multicenter quality improvement collaboratives,[8, 9] but this evaluation provides little insight into reasons for variable success across institutions.

Finally, their study design does not allow us to understand a number of key questions. How many patients were involved in the intervention? How many patients received all BOOST‐recommended interventions? Which of these interventions seemed most effective in which patients? To what degree did patient severity of illness, cognitive status, social supports, or access to primary care influence readmission risk? Such information would help frame cost‐effective deployment of BOOST or related tools.

In the end, it seems unlikely that this iteration of the BOOST program produced broad reductions in readmission rates. Having said this, the authors provide the necessary start down the road toward a fuller understanding of real‐world efforts to reduce readmissions. Stated alternately, the nuances and flaws of this study provide ample fodder for others working in the field. BOOST is in good stead with other care transition models that have not translated well from their initial research environment to real‐world practices. The question now is: Do any of these interventions actually work in clinical practice settings, and will we ever know? Even more fundamentally, how important and meaningful are these hospital‐based care transition interventions? Where is the engagement with primary care? Where are the primary care outcomes? Does BOOST truly impact outcomes other than readmission?[10]

Doing high‐quality research in the context of a rapidly evolving quality improvement program is hard. Doing it at more than 1 site is harder. BOOST's flexibility is both a great source of strength and a clear challenge to rigorous evaluation. However, when the costs of care transition programs are so high, and the potential consequences of high readmission rates are so great for patients and for hospitals, the need to address these issues with real data and better evidence is paramount. We look forward to the next phase of BOOST and to the growth and refinement of the evidence base for how to improve care coordination and transitions effectively.

In this issue of the Journal of Hospital Medicine, Hansen and colleagues provide a first, early look at the effectiveness of the BOOST intervention to reduce 30‐day readmissions among hospitalized patients.[1] BOOST[2] is 1 of a number of care transition improvement methodologies that have been applied to the problem of readmissions, each of which has evidence to support its effectiveness in its initial settings[3, 4] but has proven to be difficult to translate to other sites.[5, 6, 7]

BOOST stands in contrast with other, largely research protocol‐derived, programs in that it allows sites to tailor adoption of recommendations to local contexts and is therefore potentially more feasible to implement. Feasibility and practicality has led BOOST to be adopted in large national settings, even if it has had little evidence to support its effectiveness to date.

Given the nonstandardized and ad hoc nature of most multicenter collaboratives generally, and the flexibility of the BOOST model specifically, the BOOST authors are to be commended for undertaking any evaluation at all. Perhaps, not surprisingly, they encountered many of the problems associated with a multicenter studydropout of sites, problematic data, and limited evidence for adoption of the intervention at participating hospitals. Although these represent real‐world experiences of a quality‐improvement program, as a group they pose a number of problems that limit the study's robustness, and generate important caveats that readers should use to temper their interpretation of the authors' findings.

The first caveat relates to the substantial number of sites that either dropped out of BOOST or failed to submit data after enlisting in the collaborative. Although this may be common in quality improvement collaboratives, similar problems would not be permissible in a trial of a new drug or device. Dropout and selected ability to contribute data suggest that the ability to fully adopt BOOST may not be universal, and raises the possibility of bias, because the least successful sites may have had less interest in remaining engaged and submitting data.

The second caveat relates to how readmission rates were assessed. Because sites provided rates of readmissions at the unit level rather than the actual counts of admissions or readmissions, the authors were unable to conduct statistical analyses typically performed for these interventions, such as time series or difference‐in‐difference analyses. More importantly, one cannot discern whether their results are driven by a small absolute but large relative change in the number of readmissions at small sites. That is, large percentage changes of low statistical significance could have misleadingly affected the overall results. Conversely, we cannot identify large sites where a similar relative reduction could be statistically significant and more broadly interpreted as representing the real effectiveness of BOOST efforts.

The third caveat is in regard to the data describing the sites' performance. The effectiveness of BOOST in this analysis varied greatly among sites, with only 1 site showing a strong reduction in readmission rate, and nearly all others showing no statistical improvements. In fact, it appears that their overall results were almost entirely driven by the improvements at that 1 site.

Variable effectiveness of an intervention can be related to variable adoption or contextual factors (such as availability of personnel to implement the program). Although these authors have data on BOOST programmatic adoption, they do not have qualitative data on local barriers and facilitators to BOOST implementation, which at this stage of evaluation would be particularly valuable in understanding the results. Analyzing site‐level effectiveness is of growing relevance to multicenter quality improvement collaboratives,[8, 9] but this evaluation provides little insight into reasons for variable success across institutions.

Finally, their study design does not allow us to understand a number of key questions. How many patients were involved in the intervention? How many patients received all BOOST‐recommended interventions? Which of these interventions seemed most effective in which patients? To what degree did patient severity of illness, cognitive status, social supports, or access to primary care influence readmission risk? Such information would help frame cost‐effective deployment of BOOST or related tools.

In the end, it seems unlikely that this iteration of the BOOST program produced broad reductions in readmission rates. Having said this, the authors provide the necessary start down the road toward a fuller understanding of real‐world efforts to reduce readmissions. Stated alternately, the nuances and flaws of this study provide ample fodder for others working in the field. BOOST is in good stead with other care transition models that have not translated well from their initial research environment to real‐world practices. The question now is: Do any of these interventions actually work in clinical practice settings, and will we ever know? Even more fundamentally, how important and meaningful are these hospital‐based care transition interventions? Where is the engagement with primary care? Where are the primary care outcomes? Does BOOST truly impact outcomes other than readmission?[10]

Doing high‐quality research in the context of a rapidly evolving quality improvement program is hard. Doing it at more than 1 site is harder. BOOST's flexibility is both a great source of strength and a clear challenge to rigorous evaluation. However, when the costs of care transition programs are so high, and the potential consequences of high readmission rates are so great for patients and for hospitals, the need to address these issues with real data and better evidence is paramount. We look forward to the next phase of BOOST and to the growth and refinement of the evidence base for how to improve care coordination and transitions effectively.

In this issue of the Journal of Hospital Medicine, Hansen and colleagues provide a first, early look at the effectiveness of the BOOST intervention to reduce 30‐day readmissions among hospitalized patients.[1] BOOST[2] is 1 of a number of care transition improvement methodologies that have been applied to the problem of readmissions, each of which has evidence to support its effectiveness in its initial settings[3, 4] but has proven to be difficult to translate to other sites.[5, 6, 7]

BOOST stands in contrast with other, largely research protocol‐derived, programs in that it allows sites to tailor adoption of recommendations to local contexts and is therefore potentially more feasible to implement. Feasibility and practicality has led BOOST to be adopted in large national settings, even if it has had little evidence to support its effectiveness to date.

Given the nonstandardized and ad hoc nature of most multicenter collaboratives generally, and the flexibility of the BOOST model specifically, the BOOST authors are to be commended for undertaking any evaluation at all. Perhaps, not surprisingly, they encountered many of the problems associated with a multicenter studydropout of sites, problematic data, and limited evidence for adoption of the intervention at participating hospitals. Although these represent real‐world experiences of a quality‐improvement program, as a group they pose a number of problems that limit the study's robustness, and generate important caveats that readers should use to temper their interpretation of the authors' findings.

The first caveat relates to the substantial number of sites that either dropped out of BOOST or failed to submit data after enlisting in the collaborative. Although this may be common in quality improvement collaboratives, similar problems would not be permissible in a trial of a new drug or device. Dropout and selected ability to contribute data suggest that the ability to fully adopt BOOST may not be universal, and raises the possibility of bias, because the least successful sites may have had less interest in remaining engaged and submitting data.

The second caveat relates to how readmission rates were assessed. Because sites provided rates of readmissions at the unit level rather than the actual counts of admissions or readmissions, the authors were unable to conduct statistical analyses typically performed for these interventions, such as time series or difference‐in‐difference analyses. More importantly, one cannot discern whether their results are driven by a small absolute but large relative change in the number of readmissions at small sites. That is, large percentage changes of low statistical significance could have misleadingly affected the overall results. Conversely, we cannot identify large sites where a similar relative reduction could be statistically significant and more broadly interpreted as representing the real effectiveness of BOOST efforts.

The third caveat is in regard to the data describing the sites' performance. The effectiveness of BOOST in this analysis varied greatly among sites, with only 1 site showing a strong reduction in readmission rate, and nearly all others showing no statistical improvements. In fact, it appears that their overall results were almost entirely driven by the improvements at that 1 site.

Variable effectiveness of an intervention can be related to variable adoption or contextual factors (such as availability of personnel to implement the program). Although these authors have data on BOOST programmatic adoption, they do not have qualitative data on local barriers and facilitators to BOOST implementation, which at this stage of evaluation would be particularly valuable in understanding the results. Analyzing site‐level effectiveness is of growing relevance to multicenter quality improvement collaboratives,[8, 9] but this evaluation provides little insight into reasons for variable success across institutions.

Finally, their study design does not allow us to understand a number of key questions. How many patients were involved in the intervention? How many patients received all BOOST‐recommended interventions? Which of these interventions seemed most effective in which patients? To what degree did patient severity of illness, cognitive status, social supports, or access to primary care influence readmission risk? Such information would help frame cost‐effective deployment of BOOST or related tools.

In the end, it seems unlikely that this iteration of the BOOST program produced broad reductions in readmission rates. Having said this, the authors provide the necessary start down the road toward a fuller understanding of real‐world efforts to reduce readmissions. Stated alternately, the nuances and flaws of this study provide ample fodder for others working in the field. BOOST is in good stead with other care transition models that have not translated well from their initial research environment to real‐world practices. The question now is: Do any of these interventions actually work in clinical practice settings, and will we ever know? Even more fundamentally, how important and meaningful are these hospital‐based care transition interventions? Where is the engagement with primary care? Where are the primary care outcomes? Does BOOST truly impact outcomes other than readmission?[10]

Doing high‐quality research in the context of a rapidly evolving quality improvement program is hard. Doing it at more than 1 site is harder. BOOST's flexibility is both a great source of strength and a clear challenge to rigorous evaluation. However, when the costs of care transition programs are so high, and the potential consequences of high readmission rates are so great for patients and for hospitals, the need to address these issues with real data and better evidence is paramount. We look forward to the next phase of BOOST and to the growth and refinement of the evidence base for how to improve care coordination and transitions effectively.