Laura C Myers, MD, MPH

Adults admitted to general medical-surgical wards who experience in-hospital deterioration have a disproportionate effect on hospital mortality and length of stay.¹ Not long ago, systematic electronic capture of vital signs—arguably the most important predictors of impending deterioration—was restricted to intensive care units (ICUs). Deployment of comprehensive electronic health records (EHRs) and handheld charting tools have made vital signs data more accessible, expanding the possibilities of early detection.

In this issue, Peelen et al² report their scoping review of contemporary EHR-based predictive models for identifying ward patients at risk for deterioration. They identified 22 publications suitable for review. Impressively, some studies report extraordinary statistical performance, with positive predictive values (PPVs) exceeding 50% and with 12- to 24-hour lead times to prepare a clinician response. However, only five algorithms were implemented in an EHR and only three were used clinically. Peelen et al also quantified 48 barriers to and 54 facilitators of the implementation and use of these models. Improved statistical performance (higher PPVs) compared to manually assigned scores were the most important facilitators, while implementation in the context of daily practice (alarm fatigue, integration with existing workflows) were the most important barriers.

These reports invite an obvious question: If the models are this good, why have we not seen more reports of improved patient outcomes? Based on our own recent experience successfully deploying and evaluating the Advance Alert Monitor Program for early detection in a 21-hospital system,³ we suspect that there are several factors at play. Despite the relative computational ease of developing high-performing predictive models, it can be very challenging to create the right dataset (extracting and formatting data, standardizing variable definitions across different EHR builds). Investigators may also underestimate the difficulty of what can be implemented—and sustained—in real-world clinical practice. We encountered substantial difficulty, for example, around alarm fatigue mitigation and the relationship of alerts to end-of-life decisions. Greater attention to implementation is necessary to advance the field.

We suggest that four critical questions be considered when creating in-hospital predictive models. First, what are the statistical characteristics of a model around the likely clinical decision point? Simply having a high C-statistic is insufficient—what matters is the alert’s PPV at a clinically actionable threshold.⁴ Second, workflow burden—how many alerts per day at my hospital—must be measured, including other processes potentially affected by the new system. Third, will the extra work identify a meaningful proportion of the avoidable bad outcomes? Finally, how will model use affect care of patients near the end of life? Alerts for these patients may not make clinical sense and might even interfere with overall care (eg, by triggering an unwanted ICU transfer).

Implementation requires more than data scientists. Consideration must be given to system governance, predictive model maintenance (models can actually decalibrate over time!), and financing (not just the computation side—someone needs to pay for training clinicians and ensuring proper staffing of the clinical response).

Last, rigorous model evaluation must be undertaken. Given the increasing capabilities of comprehensive EHRs, patient-level randomization is becoming more feasible. But even randomized deployments present challenges. Since ward patients are a heterogeneous population, quantifying process-outcome relationships may be difficult. Alternative approaches to quantification of the impact of bundled interventions may need to be considered—not just for initial deployment, but on an ongoing basis. Peelen et al² have effectively summarized the state of published predictive models, which hold the tantalizing possibility of meaningful improvement: saved lives, decreased morbidity. Now, we must work together to address the identified gaps so that, one day, implementation of real-time models is routine, and the promise of in-hospital predictive analytics is fulfilled.

Adults admitted to general medical-surgical wards who experience in-hospital deterioration have a disproportionate effect on hospital mortality and length of stay.¹ Not long ago, systematic electronic capture of vital signs—arguably the most important predictors of impending deterioration—was restricted to intensive care units (ICUs). Deployment of comprehensive electronic health records (EHRs) and handheld charting tools have made vital signs data more accessible, expanding the possibilities of early detection.

In this issue, Peelen et al² report their scoping review of contemporary EHR-based predictive models for identifying ward patients at risk for deterioration. They identified 22 publications suitable for review. Impressively, some studies report extraordinary statistical performance, with positive predictive values (PPVs) exceeding 50% and with 12- to 24-hour lead times to prepare a clinician response. However, only five algorithms were implemented in an EHR and only three were used clinically. Peelen et al also quantified 48 barriers to and 54 facilitators of the implementation and use of these models. Improved statistical performance (higher PPVs) compared to manually assigned scores were the most important facilitators, while implementation in the context of daily practice (alarm fatigue, integration with existing workflows) were the most important barriers.

These reports invite an obvious question: If the models are this good, why have we not seen more reports of improved patient outcomes? Based on our own recent experience successfully deploying and evaluating the Advance Alert Monitor Program for early detection in a 21-hospital system,³ we suspect that there are several factors at play. Despite the relative computational ease of developing high-performing predictive models, it can be very challenging to create the right dataset (extracting and formatting data, standardizing variable definitions across different EHR builds). Investigators may also underestimate the difficulty of what can be implemented—and sustained—in real-world clinical practice. We encountered substantial difficulty, for example, around alarm fatigue mitigation and the relationship of alerts to end-of-life decisions. Greater attention to implementation is necessary to advance the field.

We suggest that four critical questions be considered when creating in-hospital predictive models. First, what are the statistical characteristics of a model around the likely clinical decision point? Simply having a high C-statistic is insufficient—what matters is the alert’s PPV at a clinically actionable threshold.⁴ Second, workflow burden—how many alerts per day at my hospital—must be measured, including other processes potentially affected by the new system. Third, will the extra work identify a meaningful proportion of the avoidable bad outcomes? Finally, how will model use affect care of patients near the end of life? Alerts for these patients may not make clinical sense and might even interfere with overall care (eg, by triggering an unwanted ICU transfer).

Implementation requires more than data scientists. Consideration must be given to system governance, predictive model maintenance (models can actually decalibrate over time!), and financing (not just the computation side—someone needs to pay for training clinicians and ensuring proper staffing of the clinical response).

Last, rigorous model evaluation must be undertaken. Given the increasing capabilities of comprehensive EHRs, patient-level randomization is becoming more feasible. But even randomized deployments present challenges. Since ward patients are a heterogeneous population, quantifying process-outcome relationships may be difficult. Alternative approaches to quantification of the impact of bundled interventions may need to be considered—not just for initial deployment, but on an ongoing basis. Peelen et al² have effectively summarized the state of published predictive models, which hold the tantalizing possibility of meaningful improvement: saved lives, decreased morbidity. Now, we must work together to address the identified gaps so that, one day, implementation of real-time models is routine, and the promise of in-hospital predictive analytics is fulfilled.

Adults admitted to general medical-surgical wards who experience in-hospital deterioration have a disproportionate effect on hospital mortality and length of stay.¹ Not long ago, systematic electronic capture of vital signs—arguably the most important predictors of impending deterioration—was restricted to intensive care units (ICUs). Deployment of comprehensive electronic health records (EHRs) and handheld charting tools have made vital signs data more accessible, expanding the possibilities of early detection.

In this issue, Peelen et al² report their scoping review of contemporary EHR-based predictive models for identifying ward patients at risk for deterioration. They identified 22 publications suitable for review. Impressively, some studies report extraordinary statistical performance, with positive predictive values (PPVs) exceeding 50% and with 12- to 24-hour lead times to prepare a clinician response. However, only five algorithms were implemented in an EHR and only three were used clinically. Peelen et al also quantified 48 barriers to and 54 facilitators of the implementation and use of these models. Improved statistical performance (higher PPVs) compared to manually assigned scores were the most important facilitators, while implementation in the context of daily practice (alarm fatigue, integration with existing workflows) were the most important barriers.

These reports invite an obvious question: If the models are this good, why have we not seen more reports of improved patient outcomes? Based on our own recent experience successfully deploying and evaluating the Advance Alert Monitor Program for early detection in a 21-hospital system,³ we suspect that there are several factors at play. Despite the relative computational ease of developing high-performing predictive models, it can be very challenging to create the right dataset (extracting and formatting data, standardizing variable definitions across different EHR builds). Investigators may also underestimate the difficulty of what can be implemented—and sustained—in real-world clinical practice. We encountered substantial difficulty, for example, around alarm fatigue mitigation and the relationship of alerts to end-of-life decisions. Greater attention to implementation is necessary to advance the field.

We suggest that four critical questions be considered when creating in-hospital predictive models. First, what are the statistical characteristics of a model around the likely clinical decision point? Simply having a high C-statistic is insufficient—what matters is the alert’s PPV at a clinically actionable threshold.⁴ Second, workflow burden—how many alerts per day at my hospital—must be measured, including other processes potentially affected by the new system. Third, will the extra work identify a meaningful proportion of the avoidable bad outcomes? Finally, how will model use affect care of patients near the end of life? Alerts for these patients may not make clinical sense and might even interfere with overall care (eg, by triggering an unwanted ICU transfer).

Implementation requires more than data scientists. Consideration must be given to system governance, predictive model maintenance (models can actually decalibrate over time!), and financing (not just the computation side—someone needs to pay for training clinicians and ensuring proper staffing of the clinical response).

Last, rigorous model evaluation must be undertaken. Given the increasing capabilities of comprehensive EHRs, patient-level randomization is becoming more feasible. But even randomized deployments present challenges. Since ward patients are a heterogeneous population, quantifying process-outcome relationships may be difficult. Alternative approaches to quantification of the impact of bundled interventions may need to be considered—not just for initial deployment, but on an ongoing basis. Peelen et al² have effectively summarized the state of published predictive models, which hold the tantalizing possibility of meaningful improvement: saved lives, decreased morbidity. Now, we must work together to address the identified gaps so that, one day, implementation of real-time models is routine, and the promise of in-hospital predictive analytics is fulfilled.