Original Research

Alarms from bedside continuous physiologic monitors (CPMs) occur frequently in children’s hospitals and can lead to harm. Recent studies conducted in children’s hospitals have identified alarm rates of up to 152 alarms per patient per day outside of the intensive care unit,^1-3 with as few as 1% of alarms being considered clinically important.⁴ Excessive alarms have been linked to alarm fatigue, when providers become desensitized to and may miss alarms indicating impending patient deterioration. Alarm fatigue has been identified by national patient safety organizations as a patient safety concern given the risk of patient harm.^5-7 Despite these concerns, CPMs are routinely used: up to 48% of pediatric patients in nonintensive care units at children’s hospitals are monitored.²

Although the low number of alarms that receive responses has been well-described,^8,9 the reasons why clinicians do or do not respond to alarms are unclear. A study conducted in an adult perioperative unit noted prolonged nurse response times for patients with high alarm rates.¹⁰ A second study conducted in the pediatric inpatient setting demonstrated a dose-response effect and noted progressively prolonged nurse response times with increased rates of nonactionable alarms.^4,11 Findings from another study suggested that underlying factors are highly complex and may be a result of excessive alarms, clinician characteristics, and working conditions (eg, workload and unit noise level).¹² Evidence also suggests that humans have difficulty distinguishing the importance of alarms in situations where multiple alarm tones are used, a common scenario in hospitals.^13,14 Understanding the factors that contribute to clinicians responding or not responding to CPM alarms will be crucial for addressing this serious patient safety issue.

An enhanced understanding of why nurses respond to alarms in daily practice will inform intervention development and improvement work. In the long term, this information could help improve systems for monitoring pediatric inpatients that are less prone to issues with alarm fatigue. The objective of this qualitative study, which employed structured observation, was to describe how bedside nurses think about and act upon bedside monitor alarms in a general pediatric inpatient unit.

METHODS

Study Design and Setting

This prospective observational study took place on a 48-bed hospital medicine unit at a large, freestanding children’s hospital with >650 beds and >19,000 annual admissions. General Electric (Little Chalfont, United Kingdom) physiologic monitors (models Dash 3000, 4000, and 5000) were used at the time of the study, and nurses could be notified of monitor alarms in four ways: First, an in-room auditory alarm sounds. Second, a light positioned above the door outside of each patient room blinks for alarms that are at a “warning” or “critical level” (eg ventricular tachycardia or low oxygen saturation). Third, audible alarms occur at the unit’s central monitoring station. Lastly, another staff member can notify the patient’s nurse via in-person conversion or secure smart phone communication. On the study unit, CPMs are initiated and discontinued through a physician order.

 

This study was reviewed and approved by the hospital’s institutional review board.

Study Population

We used a purposive recruitment strategy to enroll bedside nurses working on general hospital medicine units, stratified to ensure varying levels of experience and primary shifts (eg, day vs night). We planned to conduct approximately two observations with each participating nurse and to continue collecting data until we could no longer identify new insights in terms of responses to alarms (ie, thematic saturation¹⁵). Observations were targeted to cover times of day that coincided with increased rates of distraction. These times included just prior to and after the morning and evening change of shifts (7:00 am and 7:00 pm), during morning rounds (8:00 am-12:00 pm), and heavy admission times (12:00 pm-10:00 pm). After written informed consent, a nurse was eligible for observation during his/her shift if he/she was caring for at least one monitored patient. Enrolled nurses were made aware of the general study topic but were blinded to the study team’s hypotheses.

Data Sources

Prior to data collection, the research team, which consisted of physicians, bedside nurses, research coordinators, and a human factors expert, created a system for categorizing alarm responses. Categories for observed responses were based on the location and corresponding action taken. Initial categories were developed a priori from existing literature and expanded through input from the multidisciplinary study team, then vetted with bedside staff, and finally pilot tested through >4 hours of observations, thus producing the final categories. These categories were entered into a work-sampling program (WorkStudy by Quetech Ltd., Waterloo, Ontario, Canada) to facilitate quick data recording during observations.

The hospital uses a central alarm collection software (BedMasterEx by Anandic Medical Systems, Feuerthalen, Switzerland), which permitted the collection of date, time, trigger (eg, high heart rate), and level (eg, crisis, warning) of the generated CPM alarms. Alarms collected are based on thresholds preset at the bedside monitor. The central collection software does not differentiate between accurate (eg, correctly representing the physiologic state of the patient) and inaccurate alarms.

Observation Procedure

At the time of observation, nurse demographic information (eg, primary shift worked and years working as a nurse) was obtained. A brief preobservation questionnaire was administered to collect patient information (eg, age and diagnosis) and the nurses’ perspectives on the necessity of monitors for each monitored patient in his/her care.

The observer shadowed the nurse for a two-hour block of his/her shift. During this time, nurses were instructed to “think aloud” as they responded to alarms (eg, “I notice the oxygen saturation monitor alarming off, but the probe has fallen off”). A trained observer (AML or KMT) recorded responses verbalized by the nurse and his/her reaction by selecting the appropriate category using the work-sampling software. Data were also collected on the vital sign associated with the alarm (eg, heart rate). Moreover, the observer kept written notes to provide context for electronically recorded data. Alarms that were not verbalized by the nurse were not counted. Similarly, alarms that were noted outside of the room by the nurse were not classified by vital sign unless the nurse confirmed with the bedside monitor. Observers did not adjudicate the accuracy of the alarms. The session was stopped if monitors were discontinued during the observation period. Alarm data generated by the bedside monitor were pulled for each patient room after observations were completed.

 

Analysis

Descriptive statistics were used to assess the percentage of each nurse response category and each alarm type (eg, heart rate and respiratory rate). The observed alarm rate was calculated by taking the total number of observed alarms (ie, alarms noted by the nurse) divided by the total number of patient-hours observed. The monitor-generated alarm rate was calculated by taking the total number of alarms from the bedside-alarm generated data divided by the number of patient-hours observed.

Electronically recorded observations using the work-sampling program were cross-referenced with hand-written field notes to assess for any discrepancies or identify relevant events not captured by the program. Three study team members (AML, KMT, and ACS) reviewed each observation independently and compared field notes to ensure accurate categorization. Discrepancies were referred to the larger study group in cases of uncertainty.

RESULTS

Nine nurses had monitored patients during the available observations and participated in 19 observation sessions, which included 35 monitored patients for a total of 61.3 patient-hours of observation. Nurses were observed for a median of two times each (range 1-4). The median number of monitored patients during a single observation session was two (range 1-3). Observed nurses were female with a median of eight years of experience (range 0.5-26 years). Patients represented a broad range of age categories and were hospitalized with a variety of diagnoses (Table). Nurses, when queried at the start of the observation, felt that monitors were necessary for 29 (82.9%) of the observed patients given either patient condition or unit policy.

A total of 207 observed nurse responses to alarms occurred during the study period for a rate of 3.4 responses per patient per hour. Of the total number of responses, 45 (21.7%) were noted outside of a patient room, and in 15 (33.3%) the nurse chose to go to the room. The other 162 were recorded when the nurse was present in the room when the alarm activated. Of the 177 in-person nurse responses, 50 were related to a pulse oximetry alarm, 66 were related to a heart rate alarm, and 61 were related to a respiratory rate alarm. The most common observed in-person response to an alarm involved the nurse judging that no intervention was necessary (n = 152, 73.1%). Only 14 (7% of total responses) observed in-person responses involved a clinical intervention, such as suctioning or titrating supplemental oxygen. Findings are summarized in the Figure and describe nurse-verbalized reasons to further assess (or not) and then whether the nurse chose to take action (or not) after an alarm.

Alarm data were available for 17 of the 19 observation periods during the study. Technical issues with the central alarm collection software precluded alarm data collection for two of the observation sessions. A total of 483 alarms were recorded on bedside monitors during those 17 observation periods or 8.8 alarms per patient per hour, which was equivalent to 211.2 alarms per patient-day. A total of 175 observed responses were collected during these 17 observation periods. This number of responses was 36% of the number we would have expected on the basis of the alarm count from the central alarm software.

There were no patients transferred to the intensive care unit during the observation period. Nurses who chose not to respond to alarms outside the room most often cited the brevity of the alarm or other reassuring contextual details, such as that a family member was in the room to notify them if anything was truly wrong, that another member of the medical team was with the patient, or that they had recently assessed the patient and thought likely the alarm did not require any action. During three observations, the observed nurse cited the presence of family in the patient’s room in their decision not to conduct further assessment in response to the alarm, noting that the parent would be able to notify the nurse if something required attention. On two occasions in which a nurse had multiple monitored patients, the observed nurse noted that if the other monitored patients were alarming and she happened to be in another patient’s room, she would not be able to hear them. Four nurses cited policy as the reason a patient was on monitors (eg, patient was on respiratory support at night for obstructive sleep apnea).

 

DISCUSSION

We characterized responses to physiologic monitor alarms by a group of nurses with a range of experience levels. We found that most nurse responses to alarms in continuously monitored general pediatric patients involved no intervention, and further assessment was often not conducted for alarms that occurred outside of the room if the nurse noted otherwise reassuring clinical context. Observed responses occurred for 36% of alarms during the study period when compared with bedside monitor-alarm generated data. Overall, only 14 clinical interventions were noted among the observed responses. Nurses noted that they felt the monitors were necessary for 82.9% of monitored patients because of the clinical context or because of unit policy.

Our study findings highlight some potential contradictions in the current widespread use of CPMs in general pediatric units and how clinicians respond to them in practice.² First, while nurses reported that monitors were necessary for most of their patients, participating nurses deemed few alarms clinically actionable and often chose not to further assess when they noted alarms outside of the room. This is in line with findings from prior studies suggesting that clinicians overvalue the contribution of monitoring systems to patient safety.^16,17 Second, while this finding occurred in a minority of the observations, the presence of family members at the patient’s bedside was cited by nurses as a rationale for whether they responded to alarms. While family members are capable of identifying safety issues,¹⁸ formal systems to engage them in patient safety and physiologic monitoring are lacking. Finally, clinical interventions or responses to the alerts of deteriorating patients, which best represented the original intent of CPMs, were rare and accounted for just 7% of the responses. Further work elucidating why physicians and nurses choose to use CPMs may be helpful to identify interventions to reduce inappropriate monitor use and highlight gaps in frontline staff knowledge about the benefits and risks of CPM use.

Our findings provide a novel understanding of previously observed phenomena, such as long response times or nonresponses in settings with high alarm rates.^4,10 Similar to that in a prior study conducted in the pediatric setting,¹¹ alarms with an observed response constituted a minority of the total alarms that occurred in our study. This finding has previously been attributed to mental fatigue, caregiver apathy, and desensitization.⁸ However, even though a minority of observed responses in our study included an intervention, the nurse had a rationale for why the alarm did or did not need a response. This behavior and the verbalized rationale indicate that in his/her opinion, not responding to the alarm was clinically appropriate. Study participants also reflected on the difficulties of responding to alarms given the monitor system setup, in which they may not always be capable of hearing alarms for their patients. Without data from nurses regarding the alarms that had no observed response, we can only speculate; however, based on our findings, each of these factors could contribute to nonresponse. Finally, while high numbers of false alarms have been posited as an underlying cause of alarm fatigue, we noted that a majority of nonresponse was reported to be related to other clinical factors. This relationship suggests that from the nurse’s perspective, a more applicable framework for understanding alarms would be based on clinical actionability⁴ over physiologic accuracy.

In total, our findings suggest that a multifaceted approach will be necessary to improve alarm response rates. These interventions should include adjusting parameters such that alarms are highly likely to indicate a need for intervention coupled with educational interventions addressing clinician knowledge of the alarm system and bias about the actionability of alarms may improve response rates. Changes in the monitoring system setup such that nurses can easily be notified when alarms occur may also be indicated, in addition to formally engaging patients and families around response to alarms. Although secondary notification systems (eg, alarms transmitted to individual clinician’s devices) are one solution, the utilization of these systems needs to be balanced with the risks of contributing to existing alarm fatigue and the need to appropriately tailor monitoring thresholds and strategies to patients.

Our study has several limitations. First, nurses may have responded in a way they perceive to be socially desirable, and studies using in-person observers are also prone to a Hawthorne-like effect,^19-21 where the nurse may have tried to respond more frequently to alarms than usual during observations. However, given that the majority of bedside alarms did not receive a response and a substantial number of responses involved no action, these effects were likely weak. Second, we were unable to assess which alarms were accurately reflecting the patient’s physiologic status and which were not; we were also unable to link observed alarm response to monitor-recorded alarms. Third, despite the use of silent observers and an actual, rather than a simulated, clinical setting, by virtue of the data collection method we likely captured a more deliberate thought process (so-called System 2 thinking)²² rather than the subconscious processes that may predominate when nurses respond to alarms in the course of clinical care (System 1 thinking).²² Despite this limitation, our study findings, which reflect a nurse’s in-the-moment thinking, remain relevant to guiding the improvement of monitoring systems, and the development of nurse-facing interventions and education. Finally, we studied a small, purposive sample of nurses at a single hospital. Our study sample impacts the generalizability of our results and precluded a detailed analysis of the effect of nurse- and patient-level variables.

 

CONCLUSION

We found that nurses often deemed that no response was necessary for CPM alarms. Nurses cited contextual factors, including the duration of alarms and the presence of other providers or parents in their decision-making. Few (7%) of the alarm responses in our study included a clinical intervention. The number of observed alarm responses constituted roughly a third of the alarms recorded by bedside CPMs during the study. This result supports concerns about the nurse’s capacity to hear and process all CPM alarms given system limitations and a heavy clinical workload. Subsequent steps should include staff education, reducing overall alarm rates with appropriate monitor use and actionable alarm thresholds, and ensuring that patient alarms are easily recognizable for frontline staff.

Disclosures

The authors have no conflicts of interest to disclose.

Funding

This work was supported by the Place Outcomes Research Award from the Cincinnati Children’s Research Foundation. Dr. Brady is supported by the Agency for Healthcare Research and Quality under Award Number K08HS23827. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

Alarms from bedside continuous physiologic monitors (CPMs) occur frequently in children’s hospitals and can lead to harm. Recent studies conducted in children’s hospitals have identified alarm rates of up to 152 alarms per patient per day outside of the intensive care unit,^1-3 with as few as 1% of alarms being considered clinically important.⁴ Excessive alarms have been linked to alarm fatigue, when providers become desensitized to and may miss alarms indicating impending patient deterioration. Alarm fatigue has been identified by national patient safety organizations as a patient safety concern given the risk of patient harm.^5-7 Despite these concerns, CPMs are routinely used: up to 48% of pediatric patients in nonintensive care units at children’s hospitals are monitored.²

Although the low number of alarms that receive responses has been well-described,^8,9 the reasons why clinicians do or do not respond to alarms are unclear. A study conducted in an adult perioperative unit noted prolonged nurse response times for patients with high alarm rates.¹⁰ A second study conducted in the pediatric inpatient setting demonstrated a dose-response effect and noted progressively prolonged nurse response times with increased rates of nonactionable alarms.^4,11 Findings from another study suggested that underlying factors are highly complex and may be a result of excessive alarms, clinician characteristics, and working conditions (eg, workload and unit noise level).¹² Evidence also suggests that humans have difficulty distinguishing the importance of alarms in situations where multiple alarm tones are used, a common scenario in hospitals.^13,14 Understanding the factors that contribute to clinicians responding or not responding to CPM alarms will be crucial for addressing this serious patient safety issue.

An enhanced understanding of why nurses respond to alarms in daily practice will inform intervention development and improvement work. In the long term, this information could help improve systems for monitoring pediatric inpatients that are less prone to issues with alarm fatigue. The objective of this qualitative study, which employed structured observation, was to describe how bedside nurses think about and act upon bedside monitor alarms in a general pediatric inpatient unit.

METHODS

Study Design and Setting

This prospective observational study took place on a 48-bed hospital medicine unit at a large, freestanding children’s hospital with >650 beds and >19,000 annual admissions. General Electric (Little Chalfont, United Kingdom) physiologic monitors (models Dash 3000, 4000, and 5000) were used at the time of the study, and nurses could be notified of monitor alarms in four ways: First, an in-room auditory alarm sounds. Second, a light positioned above the door outside of each patient room blinks for alarms that are at a “warning” or “critical level” (eg ventricular tachycardia or low oxygen saturation). Third, audible alarms occur at the unit’s central monitoring station. Lastly, another staff member can notify the patient’s nurse via in-person conversion or secure smart phone communication. On the study unit, CPMs are initiated and discontinued through a physician order.

 

This study was reviewed and approved by the hospital’s institutional review board.

Study Population

We used a purposive recruitment strategy to enroll bedside nurses working on general hospital medicine units, stratified to ensure varying levels of experience and primary shifts (eg, day vs night). We planned to conduct approximately two observations with each participating nurse and to continue collecting data until we could no longer identify new insights in terms of responses to alarms (ie, thematic saturation¹⁵). Observations were targeted to cover times of day that coincided with increased rates of distraction. These times included just prior to and after the morning and evening change of shifts (7:00 am and 7:00 pm), during morning rounds (8:00 am-12:00 pm), and heavy admission times (12:00 pm-10:00 pm). After written informed consent, a nurse was eligible for observation during his/her shift if he/she was caring for at least one monitored patient. Enrolled nurses were made aware of the general study topic but were blinded to the study team’s hypotheses.

Data Sources

Prior to data collection, the research team, which consisted of physicians, bedside nurses, research coordinators, and a human factors expert, created a system for categorizing alarm responses. Categories for observed responses were based on the location and corresponding action taken. Initial categories were developed a priori from existing literature and expanded through input from the multidisciplinary study team, then vetted with bedside staff, and finally pilot tested through >4 hours of observations, thus producing the final categories. These categories were entered into a work-sampling program (WorkStudy by Quetech Ltd., Waterloo, Ontario, Canada) to facilitate quick data recording during observations.

The hospital uses a central alarm collection software (BedMasterEx by Anandic Medical Systems, Feuerthalen, Switzerland), which permitted the collection of date, time, trigger (eg, high heart rate), and level (eg, crisis, warning) of the generated CPM alarms. Alarms collected are based on thresholds preset at the bedside monitor. The central collection software does not differentiate between accurate (eg, correctly representing the physiologic state of the patient) and inaccurate alarms.

Observation Procedure

At the time of observation, nurse demographic information (eg, primary shift worked and years working as a nurse) was obtained. A brief preobservation questionnaire was administered to collect patient information (eg, age and diagnosis) and the nurses’ perspectives on the necessity of monitors for each monitored patient in his/her care.

The observer shadowed the nurse for a two-hour block of his/her shift. During this time, nurses were instructed to “think aloud” as they responded to alarms (eg, “I notice the oxygen saturation monitor alarming off, but the probe has fallen off”). A trained observer (AML or KMT) recorded responses verbalized by the nurse and his/her reaction by selecting the appropriate category using the work-sampling software. Data were also collected on the vital sign associated with the alarm (eg, heart rate). Moreover, the observer kept written notes to provide context for electronically recorded data. Alarms that were not verbalized by the nurse were not counted. Similarly, alarms that were noted outside of the room by the nurse were not classified by vital sign unless the nurse confirmed with the bedside monitor. Observers did not adjudicate the accuracy of the alarms. The session was stopped if monitors were discontinued during the observation period. Alarm data generated by the bedside monitor were pulled for each patient room after observations were completed.

 

Analysis

Descriptive statistics were used to assess the percentage of each nurse response category and each alarm type (eg, heart rate and respiratory rate). The observed alarm rate was calculated by taking the total number of observed alarms (ie, alarms noted by the nurse) divided by the total number of patient-hours observed. The monitor-generated alarm rate was calculated by taking the total number of alarms from the bedside-alarm generated data divided by the number of patient-hours observed.

Electronically recorded observations using the work-sampling program were cross-referenced with hand-written field notes to assess for any discrepancies or identify relevant events not captured by the program. Three study team members (AML, KMT, and ACS) reviewed each observation independently and compared field notes to ensure accurate categorization. Discrepancies were referred to the larger study group in cases of uncertainty.

RESULTS

Nine nurses had monitored patients during the available observations and participated in 19 observation sessions, which included 35 monitored patients for a total of 61.3 patient-hours of observation. Nurses were observed for a median of two times each (range 1-4). The median number of monitored patients during a single observation session was two (range 1-3). Observed nurses were female with a median of eight years of experience (range 0.5-26 years). Patients represented a broad range of age categories and were hospitalized with a variety of diagnoses (Table). Nurses, when queried at the start of the observation, felt that monitors were necessary for 29 (82.9%) of the observed patients given either patient condition or unit policy.

A total of 207 observed nurse responses to alarms occurred during the study period for a rate of 3.4 responses per patient per hour. Of the total number of responses, 45 (21.7%) were noted outside of a patient room, and in 15 (33.3%) the nurse chose to go to the room. The other 162 were recorded when the nurse was present in the room when the alarm activated. Of the 177 in-person nurse responses, 50 were related to a pulse oximetry alarm, 66 were related to a heart rate alarm, and 61 were related to a respiratory rate alarm. The most common observed in-person response to an alarm involved the nurse judging that no intervention was necessary (n = 152, 73.1%). Only 14 (7% of total responses) observed in-person responses involved a clinical intervention, such as suctioning or titrating supplemental oxygen. Findings are summarized in the Figure and describe nurse-verbalized reasons to further assess (or not) and then whether the nurse chose to take action (or not) after an alarm.

Alarm data were available for 17 of the 19 observation periods during the study. Technical issues with the central alarm collection software precluded alarm data collection for two of the observation sessions. A total of 483 alarms were recorded on bedside monitors during those 17 observation periods or 8.8 alarms per patient per hour, which was equivalent to 211.2 alarms per patient-day. A total of 175 observed responses were collected during these 17 observation periods. This number of responses was 36% of the number we would have expected on the basis of the alarm count from the central alarm software.

There were no patients transferred to the intensive care unit during the observation period. Nurses who chose not to respond to alarms outside the room most often cited the brevity of the alarm or other reassuring contextual details, such as that a family member was in the room to notify them if anything was truly wrong, that another member of the medical team was with the patient, or that they had recently assessed the patient and thought likely the alarm did not require any action. During three observations, the observed nurse cited the presence of family in the patient’s room in their decision not to conduct further assessment in response to the alarm, noting that the parent would be able to notify the nurse if something required attention. On two occasions in which a nurse had multiple monitored patients, the observed nurse noted that if the other monitored patients were alarming and she happened to be in another patient’s room, she would not be able to hear them. Four nurses cited policy as the reason a patient was on monitors (eg, patient was on respiratory support at night for obstructive sleep apnea).

 

DISCUSSION

We characterized responses to physiologic monitor alarms by a group of nurses with a range of experience levels. We found that most nurse responses to alarms in continuously monitored general pediatric patients involved no intervention, and further assessment was often not conducted for alarms that occurred outside of the room if the nurse noted otherwise reassuring clinical context. Observed responses occurred for 36% of alarms during the study period when compared with bedside monitor-alarm generated data. Overall, only 14 clinical interventions were noted among the observed responses. Nurses noted that they felt the monitors were necessary for 82.9% of monitored patients because of the clinical context or because of unit policy.

Our study findings highlight some potential contradictions in the current widespread use of CPMs in general pediatric units and how clinicians respond to them in practice.² First, while nurses reported that monitors were necessary for most of their patients, participating nurses deemed few alarms clinically actionable and often chose not to further assess when they noted alarms outside of the room. This is in line with findings from prior studies suggesting that clinicians overvalue the contribution of monitoring systems to patient safety.^16,17 Second, while this finding occurred in a minority of the observations, the presence of family members at the patient’s bedside was cited by nurses as a rationale for whether they responded to alarms. While family members are capable of identifying safety issues,¹⁸ formal systems to engage them in patient safety and physiologic monitoring are lacking. Finally, clinical interventions or responses to the alerts of deteriorating patients, which best represented the original intent of CPMs, were rare and accounted for just 7% of the responses. Further work elucidating why physicians and nurses choose to use CPMs may be helpful to identify interventions to reduce inappropriate monitor use and highlight gaps in frontline staff knowledge about the benefits and risks of CPM use.

Our findings provide a novel understanding of previously observed phenomena, such as long response times or nonresponses in settings with high alarm rates.^4,10 Similar to that in a prior study conducted in the pediatric setting,¹¹ alarms with an observed response constituted a minority of the total alarms that occurred in our study. This finding has previously been attributed to mental fatigue, caregiver apathy, and desensitization.⁸ However, even though a minority of observed responses in our study included an intervention, the nurse had a rationale for why the alarm did or did not need a response. This behavior and the verbalized rationale indicate that in his/her opinion, not responding to the alarm was clinically appropriate. Study participants also reflected on the difficulties of responding to alarms given the monitor system setup, in which they may not always be capable of hearing alarms for their patients. Without data from nurses regarding the alarms that had no observed response, we can only speculate; however, based on our findings, each of these factors could contribute to nonresponse. Finally, while high numbers of false alarms have been posited as an underlying cause of alarm fatigue, we noted that a majority of nonresponse was reported to be related to other clinical factors. This relationship suggests that from the nurse’s perspective, a more applicable framework for understanding alarms would be based on clinical actionability⁴ over physiologic accuracy.

In total, our findings suggest that a multifaceted approach will be necessary to improve alarm response rates. These interventions should include adjusting parameters such that alarms are highly likely to indicate a need for intervention coupled with educational interventions addressing clinician knowledge of the alarm system and bias about the actionability of alarms may improve response rates. Changes in the monitoring system setup such that nurses can easily be notified when alarms occur may also be indicated, in addition to formally engaging patients and families around response to alarms. Although secondary notification systems (eg, alarms transmitted to individual clinician’s devices) are one solution, the utilization of these systems needs to be balanced with the risks of contributing to existing alarm fatigue and the need to appropriately tailor monitoring thresholds and strategies to patients.

Our study has several limitations. First, nurses may have responded in a way they perceive to be socially desirable, and studies using in-person observers are also prone to a Hawthorne-like effect,^19-21 where the nurse may have tried to respond more frequently to alarms than usual during observations. However, given that the majority of bedside alarms did not receive a response and a substantial number of responses involved no action, these effects were likely weak. Second, we were unable to assess which alarms were accurately reflecting the patient’s physiologic status and which were not; we were also unable to link observed alarm response to monitor-recorded alarms. Third, despite the use of silent observers and an actual, rather than a simulated, clinical setting, by virtue of the data collection method we likely captured a more deliberate thought process (so-called System 2 thinking)²² rather than the subconscious processes that may predominate when nurses respond to alarms in the course of clinical care (System 1 thinking).²² Despite this limitation, our study findings, which reflect a nurse’s in-the-moment thinking, remain relevant to guiding the improvement of monitoring systems, and the development of nurse-facing interventions and education. Finally, we studied a small, purposive sample of nurses at a single hospital. Our study sample impacts the generalizability of our results and precluded a detailed analysis of the effect of nurse- and patient-level variables.

 

CONCLUSION

We found that nurses often deemed that no response was necessary for CPM alarms. Nurses cited contextual factors, including the duration of alarms and the presence of other providers or parents in their decision-making. Few (7%) of the alarm responses in our study included a clinical intervention. The number of observed alarm responses constituted roughly a third of the alarms recorded by bedside CPMs during the study. This result supports concerns about the nurse’s capacity to hear and process all CPM alarms given system limitations and a heavy clinical workload. Subsequent steps should include staff education, reducing overall alarm rates with appropriate monitor use and actionable alarm thresholds, and ensuring that patient alarms are easily recognizable for frontline staff.

Disclosures

The authors have no conflicts of interest to disclose.

Funding

This work was supported by the Place Outcomes Research Award from the Cincinnati Children’s Research Foundation. Dr. Brady is supported by the Agency for Healthcare Research and Quality under Award Number K08HS23827. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

Alarms from bedside continuous physiologic monitors (CPMs) occur frequently in children’s hospitals and can lead to harm. Recent studies conducted in children’s hospitals have identified alarm rates of up to 152 alarms per patient per day outside of the intensive care unit,^1-3 with as few as 1% of alarms being considered clinically important.⁴ Excessive alarms have been linked to alarm fatigue, when providers become desensitized to and may miss alarms indicating impending patient deterioration. Alarm fatigue has been identified by national patient safety organizations as a patient safety concern given the risk of patient harm.^5-7 Despite these concerns, CPMs are routinely used: up to 48% of pediatric patients in nonintensive care units at children’s hospitals are monitored.²

Although the low number of alarms that receive responses has been well-described,^8,9 the reasons why clinicians do or do not respond to alarms are unclear. A study conducted in an adult perioperative unit noted prolonged nurse response times for patients with high alarm rates.¹⁰ A second study conducted in the pediatric inpatient setting demonstrated a dose-response effect and noted progressively prolonged nurse response times with increased rates of nonactionable alarms.^4,11 Findings from another study suggested that underlying factors are highly complex and may be a result of excessive alarms, clinician characteristics, and working conditions (eg, workload and unit noise level).¹² Evidence also suggests that humans have difficulty distinguishing the importance of alarms in situations where multiple alarm tones are used, a common scenario in hospitals.^13,14 Understanding the factors that contribute to clinicians responding or not responding to CPM alarms will be crucial for addressing this serious patient safety issue.

An enhanced understanding of why nurses respond to alarms in daily practice will inform intervention development and improvement work. In the long term, this information could help improve systems for monitoring pediatric inpatients that are less prone to issues with alarm fatigue. The objective of this qualitative study, which employed structured observation, was to describe how bedside nurses think about and act upon bedside monitor alarms in a general pediatric inpatient unit.

METHODS

Study Design and Setting

This prospective observational study took place on a 48-bed hospital medicine unit at a large, freestanding children’s hospital with >650 beds and >19,000 annual admissions. General Electric (Little Chalfont, United Kingdom) physiologic monitors (models Dash 3000, 4000, and 5000) were used at the time of the study, and nurses could be notified of monitor alarms in four ways: First, an in-room auditory alarm sounds. Second, a light positioned above the door outside of each patient room blinks for alarms that are at a “warning” or “critical level” (eg ventricular tachycardia or low oxygen saturation). Third, audible alarms occur at the unit’s central monitoring station. Lastly, another staff member can notify the patient’s nurse via in-person conversion or secure smart phone communication. On the study unit, CPMs are initiated and discontinued through a physician order.

 

This study was reviewed and approved by the hospital’s institutional review board.

Study Population

We used a purposive recruitment strategy to enroll bedside nurses working on general hospital medicine units, stratified to ensure varying levels of experience and primary shifts (eg, day vs night). We planned to conduct approximately two observations with each participating nurse and to continue collecting data until we could no longer identify new insights in terms of responses to alarms (ie, thematic saturation¹⁵). Observations were targeted to cover times of day that coincided with increased rates of distraction. These times included just prior to and after the morning and evening change of shifts (7:00 am and 7:00 pm), during morning rounds (8:00 am-12:00 pm), and heavy admission times (12:00 pm-10:00 pm). After written informed consent, a nurse was eligible for observation during his/her shift if he/she was caring for at least one monitored patient. Enrolled nurses were made aware of the general study topic but were blinded to the study team’s hypotheses.

Data Sources

Prior to data collection, the research team, which consisted of physicians, bedside nurses, research coordinators, and a human factors expert, created a system for categorizing alarm responses. Categories for observed responses were based on the location and corresponding action taken. Initial categories were developed a priori from existing literature and expanded through input from the multidisciplinary study team, then vetted with bedside staff, and finally pilot tested through >4 hours of observations, thus producing the final categories. These categories were entered into a work-sampling program (WorkStudy by Quetech Ltd., Waterloo, Ontario, Canada) to facilitate quick data recording during observations.

The hospital uses a central alarm collection software (BedMasterEx by Anandic Medical Systems, Feuerthalen, Switzerland), which permitted the collection of date, time, trigger (eg, high heart rate), and level (eg, crisis, warning) of the generated CPM alarms. Alarms collected are based on thresholds preset at the bedside monitor. The central collection software does not differentiate between accurate (eg, correctly representing the physiologic state of the patient) and inaccurate alarms.

Observation Procedure

At the time of observation, nurse demographic information (eg, primary shift worked and years working as a nurse) was obtained. A brief preobservation questionnaire was administered to collect patient information (eg, age and diagnosis) and the nurses’ perspectives on the necessity of monitors for each monitored patient in his/her care.

The observer shadowed the nurse for a two-hour block of his/her shift. During this time, nurses were instructed to “think aloud” as they responded to alarms (eg, “I notice the oxygen saturation monitor alarming off, but the probe has fallen off”). A trained observer (AML or KMT) recorded responses verbalized by the nurse and his/her reaction by selecting the appropriate category using the work-sampling software. Data were also collected on the vital sign associated with the alarm (eg, heart rate). Moreover, the observer kept written notes to provide context for electronically recorded data. Alarms that were not verbalized by the nurse were not counted. Similarly, alarms that were noted outside of the room by the nurse were not classified by vital sign unless the nurse confirmed with the bedside monitor. Observers did not adjudicate the accuracy of the alarms. The session was stopped if monitors were discontinued during the observation period. Alarm data generated by the bedside monitor were pulled for each patient room after observations were completed.

 

Analysis

Descriptive statistics were used to assess the percentage of each nurse response category and each alarm type (eg, heart rate and respiratory rate). The observed alarm rate was calculated by taking the total number of observed alarms (ie, alarms noted by the nurse) divided by the total number of patient-hours observed. The monitor-generated alarm rate was calculated by taking the total number of alarms from the bedside-alarm generated data divided by the number of patient-hours observed.

Electronically recorded observations using the work-sampling program were cross-referenced with hand-written field notes to assess for any discrepancies or identify relevant events not captured by the program. Three study team members (AML, KMT, and ACS) reviewed each observation independently and compared field notes to ensure accurate categorization. Discrepancies were referred to the larger study group in cases of uncertainty.

RESULTS

Nine nurses had monitored patients during the available observations and participated in 19 observation sessions, which included 35 monitored patients for a total of 61.3 patient-hours of observation. Nurses were observed for a median of two times each (range 1-4). The median number of monitored patients during a single observation session was two (range 1-3). Observed nurses were female with a median of eight years of experience (range 0.5-26 years). Patients represented a broad range of age categories and were hospitalized with a variety of diagnoses (Table). Nurses, when queried at the start of the observation, felt that monitors were necessary for 29 (82.9%) of the observed patients given either patient condition or unit policy.

A total of 207 observed nurse responses to alarms occurred during the study period for a rate of 3.4 responses per patient per hour. Of the total number of responses, 45 (21.7%) were noted outside of a patient room, and in 15 (33.3%) the nurse chose to go to the room. The other 162 were recorded when the nurse was present in the room when the alarm activated. Of the 177 in-person nurse responses, 50 were related to a pulse oximetry alarm, 66 were related to a heart rate alarm, and 61 were related to a respiratory rate alarm. The most common observed in-person response to an alarm involved the nurse judging that no intervention was necessary (n = 152, 73.1%). Only 14 (7% of total responses) observed in-person responses involved a clinical intervention, such as suctioning or titrating supplemental oxygen. Findings are summarized in the Figure and describe nurse-verbalized reasons to further assess (or not) and then whether the nurse chose to take action (or not) after an alarm.

Alarm data were available for 17 of the 19 observation periods during the study. Technical issues with the central alarm collection software precluded alarm data collection for two of the observation sessions. A total of 483 alarms were recorded on bedside monitors during those 17 observation periods or 8.8 alarms per patient per hour, which was equivalent to 211.2 alarms per patient-day. A total of 175 observed responses were collected during these 17 observation periods. This number of responses was 36% of the number we would have expected on the basis of the alarm count from the central alarm software.

There were no patients transferred to the intensive care unit during the observation period. Nurses who chose not to respond to alarms outside the room most often cited the brevity of the alarm or other reassuring contextual details, such as that a family member was in the room to notify them if anything was truly wrong, that another member of the medical team was with the patient, or that they had recently assessed the patient and thought likely the alarm did not require any action. During three observations, the observed nurse cited the presence of family in the patient’s room in their decision not to conduct further assessment in response to the alarm, noting that the parent would be able to notify the nurse if something required attention. On two occasions in which a nurse had multiple monitored patients, the observed nurse noted that if the other monitored patients were alarming and she happened to be in another patient’s room, she would not be able to hear them. Four nurses cited policy as the reason a patient was on monitors (eg, patient was on respiratory support at night for obstructive sleep apnea).

 

DISCUSSION

We characterized responses to physiologic monitor alarms by a group of nurses with a range of experience levels. We found that most nurse responses to alarms in continuously monitored general pediatric patients involved no intervention, and further assessment was often not conducted for alarms that occurred outside of the room if the nurse noted otherwise reassuring clinical context. Observed responses occurred for 36% of alarms during the study period when compared with bedside monitor-alarm generated data. Overall, only 14 clinical interventions were noted among the observed responses. Nurses noted that they felt the monitors were necessary for 82.9% of monitored patients because of the clinical context or because of unit policy.

Our study findings highlight some potential contradictions in the current widespread use of CPMs in general pediatric units and how clinicians respond to them in practice.² First, while nurses reported that monitors were necessary for most of their patients, participating nurses deemed few alarms clinically actionable and often chose not to further assess when they noted alarms outside of the room. This is in line with findings from prior studies suggesting that clinicians overvalue the contribution of monitoring systems to patient safety.^16,17 Second, while this finding occurred in a minority of the observations, the presence of family members at the patient’s bedside was cited by nurses as a rationale for whether they responded to alarms. While family members are capable of identifying safety issues,¹⁸ formal systems to engage them in patient safety and physiologic monitoring are lacking. Finally, clinical interventions or responses to the alerts of deteriorating patients, which best represented the original intent of CPMs, were rare and accounted for just 7% of the responses. Further work elucidating why physicians and nurses choose to use CPMs may be helpful to identify interventions to reduce inappropriate monitor use and highlight gaps in frontline staff knowledge about the benefits and risks of CPM use.

Our findings provide a novel understanding of previously observed phenomena, such as long response times or nonresponses in settings with high alarm rates.^4,10 Similar to that in a prior study conducted in the pediatric setting,¹¹ alarms with an observed response constituted a minority of the total alarms that occurred in our study. This finding has previously been attributed to mental fatigue, caregiver apathy, and desensitization.⁸ However, even though a minority of observed responses in our study included an intervention, the nurse had a rationale for why the alarm did or did not need a response. This behavior and the verbalized rationale indicate that in his/her opinion, not responding to the alarm was clinically appropriate. Study participants also reflected on the difficulties of responding to alarms given the monitor system setup, in which they may not always be capable of hearing alarms for their patients. Without data from nurses regarding the alarms that had no observed response, we can only speculate; however, based on our findings, each of these factors could contribute to nonresponse. Finally, while high numbers of false alarms have been posited as an underlying cause of alarm fatigue, we noted that a majority of nonresponse was reported to be related to other clinical factors. This relationship suggests that from the nurse’s perspective, a more applicable framework for understanding alarms would be based on clinical actionability⁴ over physiologic accuracy.

In total, our findings suggest that a multifaceted approach will be necessary to improve alarm response rates. These interventions should include adjusting parameters such that alarms are highly likely to indicate a need for intervention coupled with educational interventions addressing clinician knowledge of the alarm system and bias about the actionability of alarms may improve response rates. Changes in the monitoring system setup such that nurses can easily be notified when alarms occur may also be indicated, in addition to formally engaging patients and families around response to alarms. Although secondary notification systems (eg, alarms transmitted to individual clinician’s devices) are one solution, the utilization of these systems needs to be balanced with the risks of contributing to existing alarm fatigue and the need to appropriately tailor monitoring thresholds and strategies to patients.

Our study has several limitations. First, nurses may have responded in a way they perceive to be socially desirable, and studies using in-person observers are also prone to a Hawthorne-like effect,^19-21 where the nurse may have tried to respond more frequently to alarms than usual during observations. However, given that the majority of bedside alarms did not receive a response and a substantial number of responses involved no action, these effects were likely weak. Second, we were unable to assess which alarms were accurately reflecting the patient’s physiologic status and which were not; we were also unable to link observed alarm response to monitor-recorded alarms. Third, despite the use of silent observers and an actual, rather than a simulated, clinical setting, by virtue of the data collection method we likely captured a more deliberate thought process (so-called System 2 thinking)²² rather than the subconscious processes that may predominate when nurses respond to alarms in the course of clinical care (System 1 thinking).²² Despite this limitation, our study findings, which reflect a nurse’s in-the-moment thinking, remain relevant to guiding the improvement of monitoring systems, and the development of nurse-facing interventions and education. Finally, we studied a small, purposive sample of nurses at a single hospital. Our study sample impacts the generalizability of our results and precluded a detailed analysis of the effect of nurse- and patient-level variables.

 

CONCLUSION

We found that nurses often deemed that no response was necessary for CPM alarms. Nurses cited contextual factors, including the duration of alarms and the presence of other providers or parents in their decision-making. Few (7%) of the alarm responses in our study included a clinical intervention. The number of observed alarm responses constituted roughly a third of the alarms recorded by bedside CPMs during the study. This result supports concerns about the nurse’s capacity to hear and process all CPM alarms given system limitations and a heavy clinical workload. Subsequent steps should include staff education, reducing overall alarm rates with appropriate monitor use and actionable alarm thresholds, and ensuring that patient alarms are easily recognizable for frontline staff.

Disclosures

The authors have no conflicts of interest to disclose.

Funding

This work was supported by the Place Outcomes Research Award from the Cincinnati Children’s Research Foundation. Dr. Brady is supported by the Agency for Healthcare Research and Quality under Award Number K08HS23827. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

Sepsis and heart failure are two common, costly, and deadly conditions. Among hospitalized Medicare patients, these conditions rank as the first and second most frequent principal diagnoses accounting for over $33 billion in spending across all payers.¹ One-third to one-half of all hospital deaths are estimated to occur in patients with sepsis,² and heart failure is listed as a contributing factor in over 10% of deaths in the United States.³

Previous research shows that evidence-based care decisions can impact the outcomes for these patients. For example, sepsis patients receiving intravenous fluids, blood cultures, broad-spectrum antibiotics, and lactate measurement within three hours of presentation have lower mortality rates.⁴ In heart failure, key interventions such as the appropriate use of ACE inhibitors, beta blockers, and referral to disease management programs reduce morbidity and mortality.⁵

However, rapid dissemination and adoption of evidence-based guidelines remain a challenge.^6,7 Policy makers have introduced incentives and penalties to support adoption, with varying levels of success. After four years of Centers for Medicare and Medicaid Services (CMS) penalties for hospitals with excess heart failure readmissions, only 21% performed well enough to avoid a penalty in 2017.⁸ CMS has been tracking sepsis bundle adherence as a core measure, but the rate in 2018 sat at just 54%.⁹ It is clear that new solutions are needed.¹⁰

AdventHealth (formerly Adventist Health System) is a growing, faith-based health system with hospitals across nine states. AdventHealth is a national leader in quality, safety, and patient satisfaction but is not immune to the challenges of delivering consistent, evidence-based care across an extensive network. To accelerate system-wide practice change, AdventHealth’s Office of Clinical Excellence (OCE) partnered with QURE Healthcare and Premier, Inc., to implement a physician engagement and care standardization collaboration involving nearly 100 hospitalists at eight facilities across five states.

This paper describes the results of the Adventist QURE Quality Project (AQQP), which used QURE’s validated, simulation-based measurement and feedback approach to engage hospitalists and standardize evidence-based practices for patients with sepsis and heart failure. We documented specific areas of variation identified in the simulations, how those practices changed through serial feedback, and the impact of those changes on real-world outcomes and costs.

METHODS

Setting

AdventHealth has its headquarters in Altamonte Springs, Florida. It has facilities in nine states, which includes 48 hospitals. The OCE is comprised of physician leaders, project managers, and data analysts who sponsored the project from July 2016 through July 2018.

Study Participants

AdventHealth hospitals were invited to enroll their hospitalists in AQQP; eight AdventHealth hospitals across five states, representing 91 physicians and 16 nurse practitioners/physician’s assistants (APPs), agreed to participate. Participants included both AdventHealth-employed providers and contracted hospitalist groups. Provider participation was voluntary and not tied to financial incentives; however, participants received Continuing Medical Education credit and, if applicable, Maintenance of Certification points through the American Board of Internal Medicine.

 

Quasi-experimental Design

We used AdventHealth hospitals not participating in AQQP as a quasi-experimental control group. We leveraged this to measure the impact of concurrent secular effects, such as order sets and other system-wide training, that could also improve practice and outcomes in our study.

Study Objectives and Approach

The explicit goals of AQQP were to (1) measure how sepsis and heart failure patients are cared for across AdventHealth using Clinical Performance and Value (CPV) case simulations, (2) provide a forum for hospitalists to discuss clinical variation, and (3) reduce unneeded variation to improve quality and reduce cost. QURE developed 12 CPV simulated patient cases (six sepsis and six heart failure cases) with case-specific evidenced-based scoring criteria tied to national and Advent­Health evidence-based guidelines. AdventHealth order sets were embedded in the cases and accessible by participants as they cared for their patients.

CPV vignettes are simulated patient cases administered online, and have been validated as an accurate and responsive measure of clinical decision-making in both ambulatory^11-13 and inpatient settings.^14,15 Cases take 20-30 minutes each to complete and simulate a typical clinical encounter: taking the medical history, performing a physical examination, ordering tests, making the diagnosis, implementing initial treatment, and outlining a follow-up plan. Each case has predefined, evidence-based scoring criteria for each care domain. Cases and scoring criteria were reviewed by AdventHealth hospitalist program leaders and physician leaders in OCE. Provider responses were double-scored by trained physician abstractors. Scores range from 0%-100%, with higher scores reflecting greater alignment with best practice recommendations.

In each round of the project, AQQP participants completed two CPV cases, received personalized online feedback reports on their care decisions, and met (at the various sites and via web conference) for a facilitated group discussion on areas of high group variation. The personal feedback reports included the participant’s case score compared to the group average, a list of high-priority personalized improvement opportunities, a summary of the cost of unneeded care items, and links to relevant references. The group discussions focused on six items of high variation. Six total rounds of CPV measurement and feedback were held, one every four months.

At the study’s conclusion, we administered a brief satisfaction survey, asking providers to rate various aspects of the project on a five-point Likert scale.

Data

The study used two primary data sources: (1) care decisions made in the CPV simulated cases and (2) patient-level utilization data from Premier Inc.’s QualityAdvisor^TM (QA) data system. QA integrates quality, safety, and financial data from AdventHealth’s electronic medical record, claims data, charge master, and other resources. QualityAdvisor also calculates expected performance for critical measures, including cost per case and length of stay (LOS), based on a proprietary algorithm, which uses DRG classification, severity-of-illness, risk-of-mortality, and other patient risk factors. We pulled patient-level observed and expected data from AQQP qualifying physicians, defined as physicians participating in a majority of CPV measurement rounds. Of the 107 total hospitalists who participated, six providers did not participate in enough CPV rounds, and 22 providers left AdventHealth and could not be included in a patient-level impact analysis. These providers were replaced with 21 new hospitalists who were enrolled in the study and included in the CPV analysis but who did not have patient-level data before AQQP enrollment. Overall, 58 providers met the qualifying criteria to be included in the impact analysis. We compared their performance to a group of 96 hospitalists at facilities that were not participating in the project. Comparator facilities were selected based on quantitative measures of size and demographic matching the AQQP-facilities ensuring that both sets of hospitals (comparator and AQQP) exhibited similar levels of engagement with Advent- Health quality activities such as quality dashboard performance and order set usage. Baseline patient-level cost and LOS data covered from October 2015 to June 2016 and were re-measured annually throughout the project, from July 2016 to June 2018.

 

Statistical Analyses

We analyzed three primary outcomes: (1) general CPV-measured improvements in each round (scored against evidence-based scoring criteria); (2) disease-specific CPV improvements over each round; and (3) changes in patient-level outcomes and economic savings among AdventHealth pneumonia/sepsis and heart failure patients from the aforementioned improvements. We used Student’s t-test to analyze continuous outcome variables (including CPV, cost of care, and length of stay data) and Fisher’s exact test for binary outcome data. All statistical analyses were performed using Stata 14.2 (StataCorp LLC, College Station, Texas).

RESULTS

Baseline Characteristics and Assessment

A total of 107 AdventHealth hospitalists participated in this study (Appendix Table 1). 78.1% of these providers rated the organization’s focus on quality and lowering unnecessary costs as either “good” or “excellent,” but 78.8% also reported that variation in care provided by the group was “moderate” to “very high”.

At baseline, we observed high variability in the care of pneumonia patients with sepsis (pneumonia/sepsis) and heart failure patients as measured by the care decisions obtained in the CPV cases. The overall quality score, which is a weighted average across all domains, averaged 61.9% ± 10.5% for the group (Table 1). Disaggregating scores by condition, we found an average overall score of 59.4% ± 10.9% for pneumonia/sepsis and 64.4% ± 9.4% for heart failure. The diagnosis and treatment domains, which require the most clinical judgment, had the lowest average domain scores of 53.4% ± 20.9% and 51.6% ± 15.1%, respectively.

Changes in CPV Scores

To determine the impact of serial measurement and feedback, we compared performance in the first two rounds of the project with the last two rounds. We found that overall CPV quality scores showed a 4.8%-point absolute improvement (P < .001; Table 1). We saw improvements in all care domains, and those increases were significant in all but the workup (P = .470); the most significant increase was in diagnostic accuracy (+19.1%; P < .001).

By condition, scores showed similar, statistically significant overall improvements: +4.4%-points for pneumonia/sepsis (P = .001) and +5.5%-points for heart failure (P < .001) driven by increases in the diagnosis and treatment domains. For example, providers increased appropriate identification of HF severity by 21.5%-points (P < .001) and primary diagnosis of pneumonia/sepsis by 3.6%-points (P = .385).

In the treatment domain, which included clinical decisions related to initial management and follow-up care, there were several specific improvements. For HF, we found that performing all the essential treatment elements—prescribing diuretics, ACE inhibitors and beta blockers for appropriate patients—improved by 13.9%-points (P = .038); ordering VTE prophylaxis increased more than threefold, from 16.6% to 51.0% (P < .001; Table 2). For pneumonia/sepsis patients, absolute adherence to all four elements of the 3-hour sepsis bundle improved by 11.7%-points (P = .034). We also saw a decrease in low-value diagnostic workup items for patient cases in which the guidelines suggest they are not needed, such as urinary antigen testing, which declined by 14.6%-points (P = .001) and sputum cultures, which declined 26.4%-points (P = .004). In addition, outlining an evidence-based discharge plan including a follow-up visit, patient education and medication reconciliation improved, especially for pneumonia/sepsis patients by 24.3%-points (P < .001).

Adherence to AdventHealth-preferred, evidence-based empiric antibiotic regimens was only 41.1% at baseline, but by the third round, adherence to preferred antibiotics had increased by 37% (P = .047). In the summer of 2017, after the third round, we updated scoring criteria for the cases to align with new Advent­Health-preferred antibiotic regimens. Not surprisingly, when the new antibiotic regimens were introduced, CPV-measured adherence to the new guidelines then regressed to nearly baseline levels (42.4%) as providers adjusted to the new recommendations. However, by the end of the final round, AdventHealth-preferred antibiotics orders improved by 12%.

Next, we explored whether the improvements seen were due to the best performers getting better, which was not the case. At baseline the bottom-half performers scored 10.7%-points less than top-half performers but, over the course of the study, we found that the bottom half performers had an absolute improvement nearly two times of those in the top half (+5.7%-points vs +2.9%-points; P = .006), indicating that these bottom performers were able to close the gap in quality-of-care provided. In particular, these bottom performers improved the accuracy of their primary diagnosis by 16.7%-points, compared to a 2.0%-point improvement for the top-half performers.

 

Patient-Level Impact on LOS and Cost Per Case

We took advantage of the quasi-experimental design, in which only a portion of AdventHealth facilities participated in the project, to compare patient-level results from AQQP-participating physicians against the engagement-matched cohort of hospitalists at nonparticipating AdventHealth facilities. We adjusted for potential differences in patient-level case mix between the two groups by comparing the observed/expected (O/E) LOS and cost per case ratios for pneumonia/sepsis and heart failure patients.

At baseline, AQQP-hospitalists performed better on geometric LOS versus the comparator group (O/E of 1.13 vs 1.22; P = .006) but at about the same on cost per case (O/E of 1.16 vs 1.14; P = .390). Throughout the project, as patient volumes and expected per patient costs rose for both groups, O/E ratios improved among both AQQP and non-AQQP providers.

To set apart the contribution of system-wide improvements from the AQQP project-specific impacts, we applied the O/E improvement rates seen in the comparator group to the AQQP group baseline performance. We then compared that to the actual changes seen in the AQQP throughout the project to see if there was any additional benefit from the simulation measurement and feedback (Figure).

From baseline through year one of the project, the O/E LOS ratio decreased by 8.0% in the AQQP group (1.13 to 1.04; P = .004) and only 2.5% in the comparator group (1.22 to 1.19; P = .480), which is an absolute difference-in-difference of 0.06 LOS O/E. In year 1, these improvements represent a reduction in 892 patient days among patients cared for by AQQP-hospitalists of which 570 appear to be driven by the AQQP intervention and 322 attributable to secular system-wide improvements (Table 3). In year two, both groups continued to improve with the comparator group catching up to the AQQP group.

Geometric mean O/E cost per case also decreased for both AQQP (1.16 Baseline vs 0.98 Year 2; P < .001) and comparator physicians (1.14 Baseline vs 1.01 Year 2; P = .002), for an absolute difference-in-difference of 0.05 cost O/E. However, the AQQP-hospitalists showed greater improvement (15% vs 12%; P = .346; Table 3). As in the LOS analysis, the AQQP-specific impact on cost was markedly accelerated in year one, accounting for $1.6 million of the estimated $2.6 million total savings that year. Over the two-year project, these combined improvements drove an estimated $6.2 million in total savings among AQQP-hospitalists: $3.8 million of this appear to be driven by secular system effects and, based upon our quasi-experimental design, an additional $2.4 million of which are attributable to participation in AQQP.

A Levene’s test for equality of variances on the log-transformed costs and LOS shows that the AQQP reductions in costs and LOS come from reduced variation among providers. Throughout the project, the standard deviation in LOS was reduced by 4.3%, from 3.8 days to 3.6 days (P = .046) and costs by 27.7%, from $9,391 to $6,793 (P < .001). The non-AQQP group saw a smaller, but still significant 14.6% reduction in cost variation (from $9,928 to $8,482), but saw a variation in LOS increase significantly by 20.6%, from 4.1 days to 5.0 days (P < .001).

 

Provider Satisfaction

At the project conclusion, we administered a brief survey. Participants were asked to rate aspects of the project (a five-point Likert scale with five being the highest), and 24 responded. The mean ratings of the relevance of the project to their practice and the overall quality of the material were 4.5 and 4.2, respectively. Providers found the individual feedback reports (3.9) slightly more helpful than the webcast group discussions (3.7; Appendix Table 2 ).

DISCUSSION

As health systems expand, the opportunity to standardize clinical practice within a system has the potential to enhance patient care and lower costs. However, achieving these goals is challenging when providers are dispersed across geographically separated sites and clinical decision-making is difficult to measure in a standardized way.^16,17 We brought together over 100 physicians and APPs from eight different-sized hospitals in five different states to prospectively determine if we could improve care using a standardized measurement and feedback system. At baseline, we found that care varied dramatically among providers. Care varied in terms of diagnostic accuracy and treatment, which directly relate to care quality and outcomes.⁴ After serial measurement and feedback, we saw reductions in unnecessary testing, more guideline-based treatment decisions, and better discharge planning in the clinical vignettes.

We confirmed that changes in CPV-measured practice translated into lower costs and shorter LOS at the patient level. We further validated the improvements through a quasi-experimental design that compared these changes to those at nonparticipating AdventHealth facilities. We saw more significant cost reductions and decreases in LOS in the simulation-based measurement and feedback cohort with the biggest impact early on. The overall savings to the system, attributable specifically to the AQQP approach, is estimated to be $2.4 million.

One advantage of the online case simulation approach is the ability to bring geographically remote sites together in a shared quality-of-care discussion. The interventions specifically sought to remove barriers between facilities. For example, individual feedback reports allowed providers to see how they compare with providers at other AdventHealth facilities and webcast results discussions enable providers across facilities to discuss specific care decisions.

There were several limitations to the study. While the quasi-experimental design allowed us to make informative comparisons between AQQP-participating facilities and nonparticipating facilities, the assignments were not random, and participants were generally from higher performing hospital medicine groups. The determination of secular versus CPV-related improvement is confounded by other system improvement initiatives that may have impacted cost and LOS results. This is mitigated by the observation that facilities that opted to participate performed better at baseline in risk-adjusted LOS but slightly worse in cost per case, indicating that baseline differences were not dramatic. While both groups improved over time, the QURE measurement and feedback approach led to larger and more rapid gains than those seen in the comparator group. However, we could not exclude the potential that project participation at the site level was biased to those groups disposed to performance improvement. In addition, our patient-level data analysis was limited to the metrics available and did not allow us to directly compare patient-level performance across the plethora of clinically relevant CPV data that showed improvement. Our inpatient cost per case analysis showed significant savings for the system but did not include all potentially favorable economic impacts such as lower follow-up care costs for patients, more accurate reimbursement through better coding or fewer lost days of productivity.

With continued consolidation in healthcare and broader health systems spanning multiple geographies, new tools are needed to support standardized, evidence-based care across sites. This standardization is especially important, both clinically and financially, for high-volume, high-cost diseases such as sepsis and heart failure. However, changing practice cannot happen without collaborative engagement with providers. Standardized patient vignettes are an opportunity to measure and provide feedback in a systematic way that engages providers and is particularly well-suited to large systems and common clinical conditions. This analysis, from a real-world study, shows that an approach that standardizes care and lowers costs may be particularly helpful for large systems needing to bring disparate sites together as they concurrently move toward value-based payment.

 

Disclosures

QURE, LLC, whose intellectual property was used to prepare the cases and collect the data, was contracted by AdventHealth. Otherwise, any of the study authors report no potential conflicts to disclose.

Funding

This work was funded by a contract between AdventHealth (formerly Adventist Health System) and QURE, LLC.

Sepsis and heart failure are two common, costly, and deadly conditions. Among hospitalized Medicare patients, these conditions rank as the first and second most frequent principal diagnoses accounting for over $33 billion in spending across all payers.¹ One-third to one-half of all hospital deaths are estimated to occur in patients with sepsis,² and heart failure is listed as a contributing factor in over 10% of deaths in the United States.³

Previous research shows that evidence-based care decisions can impact the outcomes for these patients. For example, sepsis patients receiving intravenous fluids, blood cultures, broad-spectrum antibiotics, and lactate measurement within three hours of presentation have lower mortality rates.⁴ In heart failure, key interventions such as the appropriate use of ACE inhibitors, beta blockers, and referral to disease management programs reduce morbidity and mortality.⁵

However, rapid dissemination and adoption of evidence-based guidelines remain a challenge.^6,7 Policy makers have introduced incentives and penalties to support adoption, with varying levels of success. After four years of Centers for Medicare and Medicaid Services (CMS) penalties for hospitals with excess heart failure readmissions, only 21% performed well enough to avoid a penalty in 2017.⁸ CMS has been tracking sepsis bundle adherence as a core measure, but the rate in 2018 sat at just 54%.⁹ It is clear that new solutions are needed.¹⁰

AdventHealth (formerly Adventist Health System) is a growing, faith-based health system with hospitals across nine states. AdventHealth is a national leader in quality, safety, and patient satisfaction but is not immune to the challenges of delivering consistent, evidence-based care across an extensive network. To accelerate system-wide practice change, AdventHealth’s Office of Clinical Excellence (OCE) partnered with QURE Healthcare and Premier, Inc., to implement a physician engagement and care standardization collaboration involving nearly 100 hospitalists at eight facilities across five states.

This paper describes the results of the Adventist QURE Quality Project (AQQP), which used QURE’s validated, simulation-based measurement and feedback approach to engage hospitalists and standardize evidence-based practices for patients with sepsis and heart failure. We documented specific areas of variation identified in the simulations, how those practices changed through serial feedback, and the impact of those changes on real-world outcomes and costs.

METHODS

Setting

AdventHealth has its headquarters in Altamonte Springs, Florida. It has facilities in nine states, which includes 48 hospitals. The OCE is comprised of physician leaders, project managers, and data analysts who sponsored the project from July 2016 through July 2018.

Study Participants

AdventHealth hospitals were invited to enroll their hospitalists in AQQP; eight AdventHealth hospitals across five states, representing 91 physicians and 16 nurse practitioners/physician’s assistants (APPs), agreed to participate. Participants included both AdventHealth-employed providers and contracted hospitalist groups. Provider participation was voluntary and not tied to financial incentives; however, participants received Continuing Medical Education credit and, if applicable, Maintenance of Certification points through the American Board of Internal Medicine.

 

Quasi-experimental Design

We used AdventHealth hospitals not participating in AQQP as a quasi-experimental control group. We leveraged this to measure the impact of concurrent secular effects, such as order sets and other system-wide training, that could also improve practice and outcomes in our study.

Study Objectives and Approach

The explicit goals of AQQP were to (1) measure how sepsis and heart failure patients are cared for across AdventHealth using Clinical Performance and Value (CPV) case simulations, (2) provide a forum for hospitalists to discuss clinical variation, and (3) reduce unneeded variation to improve quality and reduce cost. QURE developed 12 CPV simulated patient cases (six sepsis and six heart failure cases) with case-specific evidenced-based scoring criteria tied to national and Advent­Health evidence-based guidelines. AdventHealth order sets were embedded in the cases and accessible by participants as they cared for their patients.

CPV vignettes are simulated patient cases administered online, and have been validated as an accurate and responsive measure of clinical decision-making in both ambulatory^11-13 and inpatient settings.^14,15 Cases take 20-30 minutes each to complete and simulate a typical clinical encounter: taking the medical history, performing a physical examination, ordering tests, making the diagnosis, implementing initial treatment, and outlining a follow-up plan. Each case has predefined, evidence-based scoring criteria for each care domain. Cases and scoring criteria were reviewed by AdventHealth hospitalist program leaders and physician leaders in OCE. Provider responses were double-scored by trained physician abstractors. Scores range from 0%-100%, with higher scores reflecting greater alignment with best practice recommendations.

In each round of the project, AQQP participants completed two CPV cases, received personalized online feedback reports on their care decisions, and met (at the various sites and via web conference) for a facilitated group discussion on areas of high group variation. The personal feedback reports included the participant’s case score compared to the group average, a list of high-priority personalized improvement opportunities, a summary of the cost of unneeded care items, and links to relevant references. The group discussions focused on six items of high variation. Six total rounds of CPV measurement and feedback were held, one every four months.

At the study’s conclusion, we administered a brief satisfaction survey, asking providers to rate various aspects of the project on a five-point Likert scale.

Data

The study used two primary data sources: (1) care decisions made in the CPV simulated cases and (2) patient-level utilization data from Premier Inc.’s QualityAdvisor^TM (QA) data system. QA integrates quality, safety, and financial data from AdventHealth’s electronic medical record, claims data, charge master, and other resources. QualityAdvisor also calculates expected performance for critical measures, including cost per case and length of stay (LOS), based on a proprietary algorithm, which uses DRG classification, severity-of-illness, risk-of-mortality, and other patient risk factors. We pulled patient-level observed and expected data from AQQP qualifying physicians, defined as physicians participating in a majority of CPV measurement rounds. Of the 107 total hospitalists who participated, six providers did not participate in enough CPV rounds, and 22 providers left AdventHealth and could not be included in a patient-level impact analysis. These providers were replaced with 21 new hospitalists who were enrolled in the study and included in the CPV analysis but who did not have patient-level data before AQQP enrollment. Overall, 58 providers met the qualifying criteria to be included in the impact analysis. We compared their performance to a group of 96 hospitalists at facilities that were not participating in the project. Comparator facilities were selected based on quantitative measures of size and demographic matching the AQQP-facilities ensuring that both sets of hospitals (comparator and AQQP) exhibited similar levels of engagement with Advent- Health quality activities such as quality dashboard performance and order set usage. Baseline patient-level cost and LOS data covered from October 2015 to June 2016 and were re-measured annually throughout the project, from July 2016 to June 2018.

 

Statistical Analyses

We analyzed three primary outcomes: (1) general CPV-measured improvements in each round (scored against evidence-based scoring criteria); (2) disease-specific CPV improvements over each round; and (3) changes in patient-level outcomes and economic savings among AdventHealth pneumonia/sepsis and heart failure patients from the aforementioned improvements. We used Student’s t-test to analyze continuous outcome variables (including CPV, cost of care, and length of stay data) and Fisher’s exact test for binary outcome data. All statistical analyses were performed using Stata 14.2 (StataCorp LLC, College Station, Texas).

RESULTS

Baseline Characteristics and Assessment

A total of 107 AdventHealth hospitalists participated in this study (Appendix Table 1). 78.1% of these providers rated the organization’s focus on quality and lowering unnecessary costs as either “good” or “excellent,” but 78.8% also reported that variation in care provided by the group was “moderate” to “very high”.

At baseline, we observed high variability in the care of pneumonia patients with sepsis (pneumonia/sepsis) and heart failure patients as measured by the care decisions obtained in the CPV cases. The overall quality score, which is a weighted average across all domains, averaged 61.9% ± 10.5% for the group (Table 1). Disaggregating scores by condition, we found an average overall score of 59.4% ± 10.9% for pneumonia/sepsis and 64.4% ± 9.4% for heart failure. The diagnosis and treatment domains, which require the most clinical judgment, had the lowest average domain scores of 53.4% ± 20.9% and 51.6% ± 15.1%, respectively.

Changes in CPV Scores

To determine the impact of serial measurement and feedback, we compared performance in the first two rounds of the project with the last two rounds. We found that overall CPV quality scores showed a 4.8%-point absolute improvement (P < .001; Table 1). We saw improvements in all care domains, and those increases were significant in all but the workup (P = .470); the most significant increase was in diagnostic accuracy (+19.1%; P < .001).

By condition, scores showed similar, statistically significant overall improvements: +4.4%-points for pneumonia/sepsis (P = .001) and +5.5%-points for heart failure (P < .001) driven by increases in the diagnosis and treatment domains. For example, providers increased appropriate identification of HF severity by 21.5%-points (P < .001) and primary diagnosis of pneumonia/sepsis by 3.6%-points (P = .385).

In the treatment domain, which included clinical decisions related to initial management and follow-up care, there were several specific improvements. For HF, we found that performing all the essential treatment elements—prescribing diuretics, ACE inhibitors and beta blockers for appropriate patients—improved by 13.9%-points (P = .038); ordering VTE prophylaxis increased more than threefold, from 16.6% to 51.0% (P < .001; Table 2). For pneumonia/sepsis patients, absolute adherence to all four elements of the 3-hour sepsis bundle improved by 11.7%-points (P = .034). We also saw a decrease in low-value diagnostic workup items for patient cases in which the guidelines suggest they are not needed, such as urinary antigen testing, which declined by 14.6%-points (P = .001) and sputum cultures, which declined 26.4%-points (P = .004). In addition, outlining an evidence-based discharge plan including a follow-up visit, patient education and medication reconciliation improved, especially for pneumonia/sepsis patients by 24.3%-points (P < .001).

Adherence to AdventHealth-preferred, evidence-based empiric antibiotic regimens was only 41.1% at baseline, but by the third round, adherence to preferred antibiotics had increased by 37% (P = .047). In the summer of 2017, after the third round, we updated scoring criteria for the cases to align with new Advent­Health-preferred antibiotic regimens. Not surprisingly, when the new antibiotic regimens were introduced, CPV-measured adherence to the new guidelines then regressed to nearly baseline levels (42.4%) as providers adjusted to the new recommendations. However, by the end of the final round, AdventHealth-preferred antibiotics orders improved by 12%.

Next, we explored whether the improvements seen were due to the best performers getting better, which was not the case. At baseline the bottom-half performers scored 10.7%-points less than top-half performers but, over the course of the study, we found that the bottom half performers had an absolute improvement nearly two times of those in the top half (+5.7%-points vs +2.9%-points; P = .006), indicating that these bottom performers were able to close the gap in quality-of-care provided. In particular, these bottom performers improved the accuracy of their primary diagnosis by 16.7%-points, compared to a 2.0%-point improvement for the top-half performers.

 

Patient-Level Impact on LOS and Cost Per Case

We took advantage of the quasi-experimental design, in which only a portion of AdventHealth facilities participated in the project, to compare patient-level results from AQQP-participating physicians against the engagement-matched cohort of hospitalists at nonparticipating AdventHealth facilities. We adjusted for potential differences in patient-level case mix between the two groups by comparing the observed/expected (O/E) LOS and cost per case ratios for pneumonia/sepsis and heart failure patients.

At baseline, AQQP-hospitalists performed better on geometric LOS versus the comparator group (O/E of 1.13 vs 1.22; P = .006) but at about the same on cost per case (O/E of 1.16 vs 1.14; P = .390). Throughout the project, as patient volumes and expected per patient costs rose for both groups, O/E ratios improved among both AQQP and non-AQQP providers.

To set apart the contribution of system-wide improvements from the AQQP project-specific impacts, we applied the O/E improvement rates seen in the comparator group to the AQQP group baseline performance. We then compared that to the actual changes seen in the AQQP throughout the project to see if there was any additional benefit from the simulation measurement and feedback (Figure).

From baseline through year one of the project, the O/E LOS ratio decreased by 8.0% in the AQQP group (1.13 to 1.04; P = .004) and only 2.5% in the comparator group (1.22 to 1.19; P = .480), which is an absolute difference-in-difference of 0.06 LOS O/E. In year 1, these improvements represent a reduction in 892 patient days among patients cared for by AQQP-hospitalists of which 570 appear to be driven by the AQQP intervention and 322 attributable to secular system-wide improvements (Table 3). In year two, both groups continued to improve with the comparator group catching up to the AQQP group.

Geometric mean O/E cost per case also decreased for both AQQP (1.16 Baseline vs 0.98 Year 2; P < .001) and comparator physicians (1.14 Baseline vs 1.01 Year 2; P = .002), for an absolute difference-in-difference of 0.05 cost O/E. However, the AQQP-hospitalists showed greater improvement (15% vs 12%; P = .346; Table 3). As in the LOS analysis, the AQQP-specific impact on cost was markedly accelerated in year one, accounting for $1.6 million of the estimated $2.6 million total savings that year. Over the two-year project, these combined improvements drove an estimated $6.2 million in total savings among AQQP-hospitalists: $3.8 million of this appear to be driven by secular system effects and, based upon our quasi-experimental design, an additional $2.4 million of which are attributable to participation in AQQP.

A Levene’s test for equality of variances on the log-transformed costs and LOS shows that the AQQP reductions in costs and LOS come from reduced variation among providers. Throughout the project, the standard deviation in LOS was reduced by 4.3%, from 3.8 days to 3.6 days (P = .046) and costs by 27.7%, from $9,391 to $6,793 (P < .001). The non-AQQP group saw a smaller, but still significant 14.6% reduction in cost variation (from $9,928 to $8,482), but saw a variation in LOS increase significantly by 20.6%, from 4.1 days to 5.0 days (P < .001).

 

Provider Satisfaction

At the project conclusion, we administered a brief survey. Participants were asked to rate aspects of the project (a five-point Likert scale with five being the highest), and 24 responded. The mean ratings of the relevance of the project to their practice and the overall quality of the material were 4.5 and 4.2, respectively. Providers found the individual feedback reports (3.9) slightly more helpful than the webcast group discussions (3.7; Appendix Table 2 ).

DISCUSSION

As health systems expand, the opportunity to standardize clinical practice within a system has the potential to enhance patient care and lower costs. However, achieving these goals is challenging when providers are dispersed across geographically separated sites and clinical decision-making is difficult to measure in a standardized way.^16,17 We brought together over 100 physicians and APPs from eight different-sized hospitals in five different states to prospectively determine if we could improve care using a standardized measurement and feedback system. At baseline, we found that care varied dramatically among providers. Care varied in terms of diagnostic accuracy and treatment, which directly relate to care quality and outcomes.⁴ After serial measurement and feedback, we saw reductions in unnecessary testing, more guideline-based treatment decisions, and better discharge planning in the clinical vignettes.

We confirmed that changes in CPV-measured practice translated into lower costs and shorter LOS at the patient level. We further validated the improvements through a quasi-experimental design that compared these changes to those at nonparticipating AdventHealth facilities. We saw more significant cost reductions and decreases in LOS in the simulation-based measurement and feedback cohort with the biggest impact early on. The overall savings to the system, attributable specifically to the AQQP approach, is estimated to be $2.4 million.

One advantage of the online case simulation approach is the ability to bring geographically remote sites together in a shared quality-of-care discussion. The interventions specifically sought to remove barriers between facilities. For example, individual feedback reports allowed providers to see how they compare with providers at other AdventHealth facilities and webcast results discussions enable providers across facilities to discuss specific care decisions.

There were several limitations to the study. While the quasi-experimental design allowed us to make informative comparisons between AQQP-participating facilities and nonparticipating facilities, the assignments were not random, and participants were generally from higher performing hospital medicine groups. The determination of secular versus CPV-related improvement is confounded by other system improvement initiatives that may have impacted cost and LOS results. This is mitigated by the observation that facilities that opted to participate performed better at baseline in risk-adjusted LOS but slightly worse in cost per case, indicating that baseline differences were not dramatic. While both groups improved over time, the QURE measurement and feedback approach led to larger and more rapid gains than those seen in the comparator group. However, we could not exclude the potential that project participation at the site level was biased to those groups disposed to performance improvement. In addition, our patient-level data analysis was limited to the metrics available and did not allow us to directly compare patient-level performance across the plethora of clinically relevant CPV data that showed improvement. Our inpatient cost per case analysis showed significant savings for the system but did not include all potentially favorable economic impacts such as lower follow-up care costs for patients, more accurate reimbursement through better coding or fewer lost days of productivity.

With continued consolidation in healthcare and broader health systems spanning multiple geographies, new tools are needed to support standardized, evidence-based care across sites. This standardization is especially important, both clinically and financially, for high-volume, high-cost diseases such as sepsis and heart failure. However, changing practice cannot happen without collaborative engagement with providers. Standardized patient vignettes are an opportunity to measure and provide feedback in a systematic way that engages providers and is particularly well-suited to large systems and common clinical conditions. This analysis, from a real-world study, shows that an approach that standardizes care and lowers costs may be particularly helpful for large systems needing to bring disparate sites together as they concurrently move toward value-based payment.

 

Disclosures

QURE, LLC, whose intellectual property was used to prepare the cases and collect the data, was contracted by AdventHealth. Otherwise, any of the study authors report no potential conflicts to disclose.

Funding

This work was funded by a contract between AdventHealth (formerly Adventist Health System) and QURE, LLC.

Sepsis and heart failure are two common, costly, and deadly conditions. Among hospitalized Medicare patients, these conditions rank as the first and second most frequent principal diagnoses accounting for over $33 billion in spending across all payers.¹ One-third to one-half of all hospital deaths are estimated to occur in patients with sepsis,² and heart failure is listed as a contributing factor in over 10% of deaths in the United States.³

Previous research shows that evidence-based care decisions can impact the outcomes for these patients. For example, sepsis patients receiving intravenous fluids, blood cultures, broad-spectrum antibiotics, and lactate measurement within three hours of presentation have lower mortality rates.⁴ In heart failure, key interventions such as the appropriate use of ACE inhibitors, beta blockers, and referral to disease management programs reduce morbidity and mortality.⁵

However, rapid dissemination and adoption of evidence-based guidelines remain a challenge.^6,7 Policy makers have introduced incentives and penalties to support adoption, with varying levels of success. After four years of Centers for Medicare and Medicaid Services (CMS) penalties for hospitals with excess heart failure readmissions, only 21% performed well enough to avoid a penalty in 2017.⁸ CMS has been tracking sepsis bundle adherence as a core measure, but the rate in 2018 sat at just 54%.⁹ It is clear that new solutions are needed.¹⁰

AdventHealth (formerly Adventist Health System) is a growing, faith-based health system with hospitals across nine states. AdventHealth is a national leader in quality, safety, and patient satisfaction but is not immune to the challenges of delivering consistent, evidence-based care across an extensive network. To accelerate system-wide practice change, AdventHealth’s Office of Clinical Excellence (OCE) partnered with QURE Healthcare and Premier, Inc., to implement a physician engagement and care standardization collaboration involving nearly 100 hospitalists at eight facilities across five states.

This paper describes the results of the Adventist QURE Quality Project (AQQP), which used QURE’s validated, simulation-based measurement and feedback approach to engage hospitalists and standardize evidence-based practices for patients with sepsis and heart failure. We documented specific areas of variation identified in the simulations, how those practices changed through serial feedback, and the impact of those changes on real-world outcomes and costs.

METHODS

Setting

AdventHealth has its headquarters in Altamonte Springs, Florida. It has facilities in nine states, which includes 48 hospitals. The OCE is comprised of physician leaders, project managers, and data analysts who sponsored the project from July 2016 through July 2018.

Study Participants

AdventHealth hospitals were invited to enroll their hospitalists in AQQP; eight AdventHealth hospitals across five states, representing 91 physicians and 16 nurse practitioners/physician’s assistants (APPs), agreed to participate. Participants included both AdventHealth-employed providers and contracted hospitalist groups. Provider participation was voluntary and not tied to financial incentives; however, participants received Continuing Medical Education credit and, if applicable, Maintenance of Certification points through the American Board of Internal Medicine.

 

Quasi-experimental Design

We used AdventHealth hospitals not participating in AQQP as a quasi-experimental control group. We leveraged this to measure the impact of concurrent secular effects, such as order sets and other system-wide training, that could also improve practice and outcomes in our study.

Study Objectives and Approach

The explicit goals of AQQP were to (1) measure how sepsis and heart failure patients are cared for across AdventHealth using Clinical Performance and Value (CPV) case simulations, (2) provide a forum for hospitalists to discuss clinical variation, and (3) reduce unneeded variation to improve quality and reduce cost. QURE developed 12 CPV simulated patient cases (six sepsis and six heart failure cases) with case-specific evidenced-based scoring criteria tied to national and Advent­Health evidence-based guidelines. AdventHealth order sets were embedded in the cases and accessible by participants as they cared for their patients.

CPV vignettes are simulated patient cases administered online, and have been validated as an accurate and responsive measure of clinical decision-making in both ambulatory^11-13 and inpatient settings.^14,15 Cases take 20-30 minutes each to complete and simulate a typical clinical encounter: taking the medical history, performing a physical examination, ordering tests, making the diagnosis, implementing initial treatment, and outlining a follow-up plan. Each case has predefined, evidence-based scoring criteria for each care domain. Cases and scoring criteria were reviewed by AdventHealth hospitalist program leaders and physician leaders in OCE. Provider responses were double-scored by trained physician abstractors. Scores range from 0%-100%, with higher scores reflecting greater alignment with best practice recommendations.

In each round of the project, AQQP participants completed two CPV cases, received personalized online feedback reports on their care decisions, and met (at the various sites and via web conference) for a facilitated group discussion on areas of high group variation. The personal feedback reports included the participant’s case score compared to the group average, a list of high-priority personalized improvement opportunities, a summary of the cost of unneeded care items, and links to relevant references. The group discussions focused on six items of high variation. Six total rounds of CPV measurement and feedback were held, one every four months.

At the study’s conclusion, we administered a brief satisfaction survey, asking providers to rate various aspects of the project on a five-point Likert scale.

Data

The study used two primary data sources: (1) care decisions made in the CPV simulated cases and (2) patient-level utilization data from Premier Inc.’s QualityAdvisor^TM (QA) data system. QA integrates quality, safety, and financial data from AdventHealth’s electronic medical record, claims data, charge master, and other resources. QualityAdvisor also calculates expected performance for critical measures, including cost per case and length of stay (LOS), based on a proprietary algorithm, which uses DRG classification, severity-of-illness, risk-of-mortality, and other patient risk factors. We pulled patient-level observed and expected data from AQQP qualifying physicians, defined as physicians participating in a majority of CPV measurement rounds. Of the 107 total hospitalists who participated, six providers did not participate in enough CPV rounds, and 22 providers left AdventHealth and could not be included in a patient-level impact analysis. These providers were replaced with 21 new hospitalists who were enrolled in the study and included in the CPV analysis but who did not have patient-level data before AQQP enrollment. Overall, 58 providers met the qualifying criteria to be included in the impact analysis. We compared their performance to a group of 96 hospitalists at facilities that were not participating in the project. Comparator facilities were selected based on quantitative measures of size and demographic matching the AQQP-facilities ensuring that both sets of hospitals (comparator and AQQP) exhibited similar levels of engagement with Advent- Health quality activities such as quality dashboard performance and order set usage. Baseline patient-level cost and LOS data covered from October 2015 to June 2016 and were re-measured annually throughout the project, from July 2016 to June 2018.

 

Statistical Analyses

We analyzed three primary outcomes: (1) general CPV-measured improvements in each round (scored against evidence-based scoring criteria); (2) disease-specific CPV improvements over each round; and (3) changes in patient-level outcomes and economic savings among AdventHealth pneumonia/sepsis and heart failure patients from the aforementioned improvements. We used Student’s t-test to analyze continuous outcome variables (including CPV, cost of care, and length of stay data) and Fisher’s exact test for binary outcome data. All statistical analyses were performed using Stata 14.2 (StataCorp LLC, College Station, Texas).

RESULTS

Baseline Characteristics and Assessment

A total of 107 AdventHealth hospitalists participated in this study (Appendix Table 1). 78.1% of these providers rated the organization’s focus on quality and lowering unnecessary costs as either “good” or “excellent,” but 78.8% also reported that variation in care provided by the group was “moderate” to “very high”.

At baseline, we observed high variability in the care of pneumonia patients with sepsis (pneumonia/sepsis) and heart failure patients as measured by the care decisions obtained in the CPV cases. The overall quality score, which is a weighted average across all domains, averaged 61.9% ± 10.5% for the group (Table 1). Disaggregating scores by condition, we found an average overall score of 59.4% ± 10.9% for pneumonia/sepsis and 64.4% ± 9.4% for heart failure. The diagnosis and treatment domains, which require the most clinical judgment, had the lowest average domain scores of 53.4% ± 20.9% and 51.6% ± 15.1%, respectively.

Changes in CPV Scores

To determine the impact of serial measurement and feedback, we compared performance in the first two rounds of the project with the last two rounds. We found that overall CPV quality scores showed a 4.8%-point absolute improvement (P < .001; Table 1). We saw improvements in all care domains, and those increases were significant in all but the workup (P = .470); the most significant increase was in diagnostic accuracy (+19.1%; P < .001).

By condition, scores showed similar, statistically significant overall improvements: +4.4%-points for pneumonia/sepsis (P = .001) and +5.5%-points for heart failure (P < .001) driven by increases in the diagnosis and treatment domains. For example, providers increased appropriate identification of HF severity by 21.5%-points (P < .001) and primary diagnosis of pneumonia/sepsis by 3.6%-points (P = .385).

In the treatment domain, which included clinical decisions related to initial management and follow-up care, there were several specific improvements. For HF, we found that performing all the essential treatment elements—prescribing diuretics, ACE inhibitors and beta blockers for appropriate patients—improved by 13.9%-points (P = .038); ordering VTE prophylaxis increased more than threefold, from 16.6% to 51.0% (P < .001; Table 2). For pneumonia/sepsis patients, absolute adherence to all four elements of the 3-hour sepsis bundle improved by 11.7%-points (P = .034). We also saw a decrease in low-value diagnostic workup items for patient cases in which the guidelines suggest they are not needed, such as urinary antigen testing, which declined by 14.6%-points (P = .001) and sputum cultures, which declined 26.4%-points (P = .004). In addition, outlining an evidence-based discharge plan including a follow-up visit, patient education and medication reconciliation improved, especially for pneumonia/sepsis patients by 24.3%-points (P < .001).

Adherence to AdventHealth-preferred, evidence-based empiric antibiotic regimens was only 41.1% at baseline, but by the third round, adherence to preferred antibiotics had increased by 37% (P = .047). In the summer of 2017, after the third round, we updated scoring criteria for the cases to align with new Advent­Health-preferred antibiotic regimens. Not surprisingly, when the new antibiotic regimens were introduced, CPV-measured adherence to the new guidelines then regressed to nearly baseline levels (42.4%) as providers adjusted to the new recommendations. However, by the end of the final round, AdventHealth-preferred antibiotics orders improved by 12%.

Next, we explored whether the improvements seen were due to the best performers getting better, which was not the case. At baseline the bottom-half performers scored 10.7%-points less than top-half performers but, over the course of the study, we found that the bottom half performers had an absolute improvement nearly two times of those in the top half (+5.7%-points vs +2.9%-points; P = .006), indicating that these bottom performers were able to close the gap in quality-of-care provided. In particular, these bottom performers improved the accuracy of their primary diagnosis by 16.7%-points, compared to a 2.0%-point improvement for the top-half performers.

 

Patient-Level Impact on LOS and Cost Per Case

We took advantage of the quasi-experimental design, in which only a portion of AdventHealth facilities participated in the project, to compare patient-level results from AQQP-participating physicians against the engagement-matched cohort of hospitalists at nonparticipating AdventHealth facilities. We adjusted for potential differences in patient-level case mix between the two groups by comparing the observed/expected (O/E) LOS and cost per case ratios for pneumonia/sepsis and heart failure patients.

At baseline, AQQP-hospitalists performed better on geometric LOS versus the comparator group (O/E of 1.13 vs 1.22; P = .006) but at about the same on cost per case (O/E of 1.16 vs 1.14; P = .390). Throughout the project, as patient volumes and expected per patient costs rose for both groups, O/E ratios improved among both AQQP and non-AQQP providers.

To set apart the contribution of system-wide improvements from the AQQP project-specific impacts, we applied the O/E improvement rates seen in the comparator group to the AQQP group baseline performance. We then compared that to the actual changes seen in the AQQP throughout the project to see if there was any additional benefit from the simulation measurement and feedback (Figure).

From baseline through year one of the project, the O/E LOS ratio decreased by 8.0% in the AQQP group (1.13 to 1.04; P = .004) and only 2.5% in the comparator group (1.22 to 1.19; P = .480), which is an absolute difference-in-difference of 0.06 LOS O/E. In year 1, these improvements represent a reduction in 892 patient days among patients cared for by AQQP-hospitalists of which 570 appear to be driven by the AQQP intervention and 322 attributable to secular system-wide improvements (Table 3). In year two, both groups continued to improve with the comparator group catching up to the AQQP group.

Geometric mean O/E cost per case also decreased for both AQQP (1.16 Baseline vs 0.98 Year 2; P < .001) and comparator physicians (1.14 Baseline vs 1.01 Year 2; P = .002), for an absolute difference-in-difference of 0.05 cost O/E. However, the AQQP-hospitalists showed greater improvement (15% vs 12%; P = .346; Table 3). As in the LOS analysis, the AQQP-specific impact on cost was markedly accelerated in year one, accounting for $1.6 million of the estimated $2.6 million total savings that year. Over the two-year project, these combined improvements drove an estimated $6.2 million in total savings among AQQP-hospitalists: $3.8 million of this appear to be driven by secular system effects and, based upon our quasi-experimental design, an additional $2.4 million of which are attributable to participation in AQQP.

A Levene’s test for equality of variances on the log-transformed costs and LOS shows that the AQQP reductions in costs and LOS come from reduced variation among providers. Throughout the project, the standard deviation in LOS was reduced by 4.3%, from 3.8 days to 3.6 days (P = .046) and costs by 27.7%, from $9,391 to $6,793 (P < .001). The non-AQQP group saw a smaller, but still significant 14.6% reduction in cost variation (from $9,928 to $8,482), but saw a variation in LOS increase significantly by 20.6%, from 4.1 days to 5.0 days (P < .001).

 

Provider Satisfaction

At the project conclusion, we administered a brief survey. Participants were asked to rate aspects of the project (a five-point Likert scale with five being the highest), and 24 responded. The mean ratings of the relevance of the project to their practice and the overall quality of the material were 4.5 and 4.2, respectively. Providers found the individual feedback reports (3.9) slightly more helpful than the webcast group discussions (3.7; Appendix Table 2 ).

DISCUSSION

As health systems expand, the opportunity to standardize clinical practice within a system has the potential to enhance patient care and lower costs. However, achieving these goals is challenging when providers are dispersed across geographically separated sites and clinical decision-making is difficult to measure in a standardized way.^16,17 We brought together over 100 physicians and APPs from eight different-sized hospitals in five different states to prospectively determine if we could improve care using a standardized measurement and feedback system. At baseline, we found that care varied dramatically among providers. Care varied in terms of diagnostic accuracy and treatment, which directly relate to care quality and outcomes.⁴ After serial measurement and feedback, we saw reductions in unnecessary testing, more guideline-based treatment decisions, and better discharge planning in the clinical vignettes.

We confirmed that changes in CPV-measured practice translated into lower costs and shorter LOS at the patient level. We further validated the improvements through a quasi-experimental design that compared these changes to those at nonparticipating AdventHealth facilities. We saw more significant cost reductions and decreases in LOS in the simulation-based measurement and feedback cohort with the biggest impact early on. The overall savings to the system, attributable specifically to the AQQP approach, is estimated to be $2.4 million.

One advantage of the online case simulation approach is the ability to bring geographically remote sites together in a shared quality-of-care discussion. The interventions specifically sought to remove barriers between facilities. For example, individual feedback reports allowed providers to see how they compare with providers at other AdventHealth facilities and webcast results discussions enable providers across facilities to discuss specific care decisions.

There were several limitations to the study. While the quasi-experimental design allowed us to make informative comparisons between AQQP-participating facilities and nonparticipating facilities, the assignments were not random, and participants were generally from higher performing hospital medicine groups. The determination of secular versus CPV-related improvement is confounded by other system improvement initiatives that may have impacted cost and LOS results. This is mitigated by the observation that facilities that opted to participate performed better at baseline in risk-adjusted LOS but slightly worse in cost per case, indicating that baseline differences were not dramatic. While both groups improved over time, the QURE measurement and feedback approach led to larger and more rapid gains than those seen in the comparator group. However, we could not exclude the potential that project participation at the site level was biased to those groups disposed to performance improvement. In addition, our patient-level data analysis was limited to the metrics available and did not allow us to directly compare patient-level performance across the plethora of clinically relevant CPV data that showed improvement. Our inpatient cost per case analysis showed significant savings for the system but did not include all potentially favorable economic impacts such as lower follow-up care costs for patients, more accurate reimbursement through better coding or fewer lost days of productivity.

With continued consolidation in healthcare and broader health systems spanning multiple geographies, new tools are needed to support standardized, evidence-based care across sites. This standardization is especially important, both clinically and financially, for high-volume, high-cost diseases such as sepsis and heart failure. However, changing practice cannot happen without collaborative engagement with providers. Standardized patient vignettes are an opportunity to measure and provide feedback in a systematic way that engages providers and is particularly well-suited to large systems and common clinical conditions. This analysis, from a real-world study, shows that an approach that standardizes care and lowers costs may be particularly helpful for large systems needing to bring disparate sites together as they concurrently move toward value-based payment.

 

Disclosures

QURE, LLC, whose intellectual property was used to prepare the cases and collect the data, was contracted by AdventHealth. Otherwise, any of the study authors report no potential conflicts to disclose.

Funding

This work was funded by a contract between AdventHealth (formerly Adventist Health System) and QURE, LLC.

Approximately 60,000 hospitalists were working in the United States in 2018.¹ Hospitalist groups work collaboratively because of the shiftwork required for 24/7 patient coverage, and first-rate clinical documentation is essential for quality care.² Thoughtful clinical documentation not only transmits one provider’s clinical reasoning to other providers but is a professional responsibility.³ Hospitalists spend two-thirds of their time in indirect patient-care activities and approximately one quarter of their time on documentation in electronic health records (EHRs).⁴ Despite documentation occupying a substantial portion of the clinician’s time, published literature on the best practices for the documentation of clinical reasoning in hospital medicine or its assessment remains scant.^5-7

Clinical reasoning involves establishing a diagnosis and developing a therapeutic plan that fits the unique circumstances and needs of the patient.⁸ Inpatient providers who admit patients to the hospital end the admission note with their assessment and plan (A&P) after reflecting about a patient’s presenting illness. The A&P generally represents the interpretations, deductions, and clinical reasoning of the inpatient providers; this is the section of the note that fellow physicians concentrate on over others.⁹ The documentation of clinical reasoning in the A&P allows for many to consider how the recorded interpretations relate to their own elucidations resulting in distributed cognition.¹⁰

Disorganized documentation can contribute to cognitive overload and impede thoughtful consideration about the clinical presentation.³ The assessment of clinical documentation may translate into reduced medical errors and improved note quality.^11,12 Studies that have formally evaluated the documentation of clinical reasoning have focused exclusively on medical students.^13-15 The nonexistence of a detailed rubric for evaluating clinical reasoning in the A&Ps of hospitalists represents a missed opportunity for evaluating what hospitalists “do”; if this evolves into a mechanism for offering formative feedback, such professional development would impact the highest level of Miller’s assessment pyramid.¹⁶ We therefore undertook this study to establish a metric to assess the hospitalist providers’ documentation of clinical reasoning in the A&P of an admission note.

METHODS

Study Design, Setting, and Subjects

This was a retrospective study that reviewed the admission notes of hospitalists for patients admitted over the period of January 2014 and October 2017 at three hospitals in Maryland. One is a community hospital (Hospital A) and two are academic medical centers (Hospital B and Hospital C). Even though these three hospitals are part of one health system, they have distinct cultures and leadership, serve different populations, and are staffed by different provider teams.

 

The notes of physicians working for the hospitalist groups at each of the three hospitals were the focus of the analysis in this study.

Development of the Documentation Assessment Rubric

A team was assembled to develop the Clinical Reasoning in Admission Note Assessment & PLan (CRANAPL) tool. The CRANAPL was designed to assess the comprehensiveness and thoughtfulness of the clinical reasoning documented in the A&P sections of the notes of patients who were admitted to the hospital with an acute illness. Validity evidence for CRANAPL was summarized on the basis of Messick’s unified validity framework by using four of the five sources of validity: content, response process, internal structure, and relations to other variables.¹⁷

Content Validity

The development team consisted of members who have an average of 10 years of clinical experience in hospital medicine; have studied clinical excellence and clinical reasoning; and have expertise in feedback, assessment, and professional development.^18-22 The development of the CRANAPL tool by the team was informed by a review of the clinical reasoning literature, with particular attention paid to the standards and competencies outlined by the Liaison Committee on Medical Education, the Association of American Medical Colleges, the Accreditation Council on Graduate Medical Education, the Internal Medicine Milestone Project, and the Society of Hospital Medicine.^23-26 For each of these parties, diagnostic reasoning and its impact on clinical decision-making are considered to be a core competency. Several works that heavily influenced the CRANAPL tool’s development were Baker’s Interpretive Summary, Differential Diagnosis, Explanation of Reasoning, And Alternatives (IDEA) assessment tool;¹⁴ King’s Pediatric History and Physical Exam Evaluation (P-HAPEE) rubric;¹⁵ and three other studies related to diagnostic reasoning.^16,27,28 These manuscripts and other works substantively informed the preliminary behavioral-based anchors that formed the initial foundation for the tool under development. The CRANAPL tool was shown to colleagues at other institutions who are leaders on clinical reasoning and was presented at academic conferences in the Division of General Internal Medicine and the Division of Hospital Medicine of our institution. Feedback resulted in iterative revisions. The aforementioned methods established content validity evidence for the CRANAPL tool.

Response Process Validity

Several of the authors pilot-tested earlier iterations on admission notes that were excluded from the sample when refining the CRANAPL tool. The weaknesses and sources of confusion with specific items were addressed by scoring 10 A&Ps individually and then comparing data captured on the tool. This cycle was repeated three times for the iterative enhancement and finalization of the CRANAPL tool. On several occasions when two authors were piloting the near-final CRANAPL tool, a third author interviewed each of the two authors about reactivity while assessing individual items and exploring with probes how their own clinical documentation practices were being considered when scoring the notes. The reasonable and thoughtful answers provided by the two authors as they explained and justified the scores they were selecting during the pilot testing served to confer response process validity evidence.

Finalizing the CRANAPL Tool

The nine-item CRANAPL tool includes elements for problem representation, leading diagnosis, uncertainty, differential diagnosis, plans for diagnosis and treatment, estimated length of stay (LOS), potential for upgrade in status to a higher level of care, and consideration of disposition. Although the final three items are not core clinical reasoning domains in the medical education literature, they represent clinical judgments that are especially relevant for the delivery of the high-quality and cost-effective care of hospitalized patients. Given that the probabilities and estimations of these three elements evolve over the course of any hospitalization on the basis of test results and response to therapy, the documentation of initial expectations on these fronts can facilitate distributed cognition with all individuals becoming wiser from shared insights.¹⁰ The tool uses two- and three-point rating scales, with each number score being clearly defined by specific written criteria (total score range: 0-14; Appendix).

 

Data Collection

Hospitalists’ admission notes from the three hospitals were used to validate the CRANAPL tool. Admission notes from patients hospitalized to the general medical floors with an admission diagnosis of either fever, syncope/dizziness, or abdominal pain were used. These diagnoses were purposefully examined because they (1) have a wide differential diagnosis, (2) are common presenting symptoms, and (3) are prone to diagnostic errors.^29-32

The centralized EHR system across the three hospitals identified admission notes with one of these primary diagnoses of patients admitted over the period of January 2014 to October 2017. We submitted a request for 650 admission notes to be randomly selected from the centralized institutional records system. The notes were stratified by hospital and diagnosis. The sample size of our study was comparable with that of prior psychometric validation studies.^33,34 Upon reviewing the A&Ps associated with these admissions, 365 notes were excluded for one of three reasons: (1) the note was written by a nurse practitioner, physician assistant, resident, or medical student; (2) the admission diagnosis had been definitively confirmed in the emergency department (eg, abdominal pain due to diverticulitis seen on CT); and (3) the note represented the fourth or more note by any single provider (to sample notes of many providers, no more than three notes written by any single provider were analyzed). A total of 285 admission notes were ultimately included in the sample.

Data were deidentified, and the A&P sections of the admission notes were each copied from the EHR into a unique Word document. Patient and hospital demographic data (including age, gender, race, number of comorbid conditions, LOS, hospital charges, and readmission to the same health system within 30 days) were collected separately from the EHR. Select physician characteristics were also collected from the hospitalist groups at each of the three hospitals, as was the length (word count) of each A&P.

The study was approved by our institutional review board.

Data Analysis

Two authors scored all deidentified A&Ps by using the finalized version of the CRANAPL tool. Prior to using the CRANAPL tool on each of the notes, these raters read each A&P and scored them by using two single-item rating scales: a global clinical reasoning and a global readability/clarity measure. Both of these global scales used three-item Likert scales (below average, average, and above average). These global rating scales collected the reviewers’ gestalt about the quality and clarity of the A&P. The use of gestalt ratings as comparators is supported by other research.³⁵

Descriptive statistics were computed for all variables. Each rater rescored a sample of 48 records (one month after the initial scoring) and intraclass correlations (ICCs) were computed for intrarater reliability. ICCs were calculated for each item and for the CRANAPL total to determine interrater reliability.

The averaged ratings from the two raters were used for all other analyses. For CRANAPL’s internal structure validity evidence, Cronbach’s alpha was calculated as a measure of internal consistency. For relations to other variables validity evidence, CRANAPL total scores were compared with the two global assessment variables with linear regressions.

Bivariate analyses were performed by applying parametric and nonparametric tests as appropriate. A series of multivariate linear regressions, controlling for diagnosis and clustered variance by hospital site, were performed using CRANAPL total as the dependent variable and patient variables as predictors.

All data were analyzed using Stata (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, Texas: StataCorp LP.)

 

RESULTS

The admission notes of 120 hospitalists were evaluated (Table 1). A total of 39 (33%) physicians were moonlighters with primary appointments outside of the hospitalist division, and 81 (68%) were full-time hospitalists. Among the 120 hospitalists, 48 (40%) were female, 60 (50%) were international medical graduates, and 90 (75%) were of nonwhite race. Most hospitalist physicians (n = 47, 58%) had worked in our health system for less than five years, and 64 hospitalists (53%) devoted greater than 50% of their time to patient care.

Approximately equal numbers of patient admission notes were pulled from each of the three hospitals. The average age of patients was 67.2 (SD 13.6) years, 145 (51%) were female, and 120 (42%) were of nonwhite race. The mean LOS for all patients was 4.0 (SD 3.4) days. A total of 44 (15%) patients were readmitted to the same health system within 30 days of discharge. None of the patients died during the incident hospitalization. The average charge for each of the hospitalizations was $10,646 (SD $9,964).

CRANAPL Data

Figure 1 shows the distribution of the scores given by each rater for each of the nine items. The mean of the total CRANAPL score given by both raters was 6.4 (SD 2.2). Scoring for some items were high (eg, summary statement: 1.5/2), whereas performance on others were low (eg, estimating LOS: 0.1/1 and describing the potential need for upgrade in care: 0.0/1).

Validity of the CRANAPL Tool’s Internal Structure

Cronbach’s alpha, which was used to measure internal consistency within the CRANAPL tool, was 0.43. The ICC, which was applied to measure the interrater reliability for both raters for the total CRANAPL score, was 0.83 (95% CI:  0.76-0.87). The ICC values for intrarater reliability for raters 1 and 2 were 0.73 (95% CI: 0.60-0.83) and 0.73 (95% CI: 0.45-0.86), respectively.

Relations to Other Variables Validity

Associations between CRANAPL total scores, global clinical reasoning, and global scores for note readability/clarity were statistically significant (P < .001), Figure 2.

Eight out of nine CRANAPL variables were statistically significantly different across the three hospitals (P <. 01) when data were analyzed by hospital site. Hospital C had the highest mean score of 7.4 (SD 2.0), followed by Hospital B with a score of 6.6 (SD 2.1), and Hospital A had the lowest total CRANAPL score of 5.2 (SD 1.9). This difference was statistically significant (P < .001). Five variables with respect to admission diagnoses (uncertainty acknowledged, differential diagnosis, plan for diagnosis, plan for treatment, and upgrade plan) were statistically significantly different across notes. Notes for syncope/dizziness generally yielded higher scores than those for abdominal pain and fever.

Factors Associated with High CRANAPL Scores

Table 2 shows the associations between CRANAPL scores and several covariates. Before adjustment, high CRANAPL scores were associated with high word counts of A&Ps (P < .001) and high hospital charges (P < .05). These associations were no longer significant after adjusting for hospital site and admitting diagnoses.

 

DISCUSSION

We reviewed the documentation of clinical reasoning in 285 admission notes at three different hospitals written by hospitalist physicians during routine clinical care. To our knowledge, this is the first study that assessed the documentation of hospitalists’ clinical reasoning with real patient notes. Wide variability exists in the documentation of clinical reasoning within the A&Ps of hospitalists’ admission notes. We have provided validity evidence to support the use of the user-friendly CRANAPL tool.

Prior studies have described rubrics for evaluating the clinical reasoning skills of medical students.^14,15 The ICCs for the IDEA rubric used to assess medical students’ documentation of clinical reasoning were fair to moderate (0.29-0.67), whereas the ICC for the CRANAPL tool was high at 0.83. This measure of reliability is similar to that for the P-HAPEE rubric used to assess medical students’ documentation of pediatric history and physical notes.¹⁵ These data are markedly different from the data in previous studies that have found low interrater reliability for psychometric evaluations related to judgment and decision-making.^36-39 CRANAPL was also found to have high intrarater reliability, which shows the reproducibility of an individual’s assessment over time. The strong association between the total CRANAPL score and global clinical reasoning assessment found in the present study is similar to that found in previous studies that have also embedded global rating scales as comparators when assessing clinical reasoning.^13,,15,40,41 Global rating scales represent an overarching structure for comparison given the absence of an accepted method or gold standard for assessing clinical reasoning documentation. High-quality provider notes are defined by clarity, thoroughness, and accuracy;³⁵ and effective documentation promotes communication and the coordination of care among the members of the care team.³

The total CRANAPL scores varied by hospital site with academic hospitals (B and C) scoring higher than the community hospital (A) in our study. Similarly, lengthy A&Ps were associated with high CRANAPL scores (P < .001) prior to adjustment for hospital site. Healthcare providers consider that the thoroughness of documentation denotes quality and attention to detail.^35,42 Comprehensive documentation takes time; the longer notes by academic hospitalists than those by community hospitalists may be attributed to the fewer number of patients generally carried by hospitalists at academic centers than that by hospitalists at community hospitals.⁴³

The documentation of the estimations of LOS, possibility of potential upgrade, and thoughts about disposition were consistently poorly described across all hospital sites and diagnoses. In contrast to CRANAPL, other clinical reasoning rubrics have excluded these items or discussed uncertainty.^14,15,44 These elements represent the forward thinking that may be essential for high-quality progressive care by hospitalists. Physicians’s difficulty in acknowledging uncertainty has been associated with resource overuse, including the excessive ordering of tests, iatrogenic injury, and heavy financial burden on the healthcare system.^45,46 The lack of thoughtful clinical and management reasoning at the time of admission is believed to be associated with medical errors.⁴⁷ If used as a guide, the CRANAPL tool may promote reflection on the part of the admitting physician. The estimations of LOS, potential for upgrade to a higher level of care, and disposition are markers of optimal inpatient care, especially for hospitalists who work in shifts with embedded handoffs. When shared with colleagues (through documentation), there is the potential for distributed cognition¹⁰ to extend throughout the social network of the hospitalist group. The fact that so few providers are currently including these items in their A&P’s show that the providers are either not performing or documenting the ‘reasoning’. Either way, this is an opportunity that has been highlighted by the CRANAPL tool.

Several limitations of this study should be considered. First, the CRANAPL tool may not have captured elements of optimal clinical reasoning documentation. The reliance on multiple methods and an iterative process in the refinement of the CRANAPL tool should have minimized this. Second, this study was conducted across a single healthcare system that uses the same EHR; this EHR or institutional culture may influence documentation practices and behaviors. Given that using the CRANAPL tool to score an A&P is quick and easy, the benefit of giving providers feedback on their notes remains to be seen—here and at other hospitals. Third, our sample size could limit the generalizability of the results and the significance of the associations. However, the sample assessed in our study was significantly larger than that assessed in other studies that have validated clinical reasoning rubrics.^14,15 Fourth, clinical reasoning is a broad and multidimensional construct. The CRANAPL tool focuses exclusively on hospitalists’ documentation of clinical reasoning and therefore does not assess aspects of clinical reasoning occurring in the physicians’ minds. Finally, given our goal to optimally validate the CRANAPL tool, we chose to test the tool on specific presentations that are known to be associated with diagnostic practice variation and errors. We may have observed different results had we chosen a different set of diagnoses from each hospital. Further validity evidence will be established when applying the CRANPL tool to different diagnoses and to notes from other clinical settings.

In conclusion, this study focuses on the development and validation of the CRANAPL tool that assesses how hospitalists document their clinical reasoning in the A&P section of admission notes. Our results show that wide variability exists in the documentation of clinical reasoning by hospitalists within and across hospitals. Given the CRANAPL tool’s ease-of-use and its versatility, hospitalist divisions in academic and nonacademic settings may use the CRANAPL tool to assess and provide feedback on the documentation of hospitalists’ clinical reasoning. Beyond studying whether physicians can be taught to improve their notes with feedback based on the CRANAPL tool, future studies may explore whether enhancing clinical reasoning documentation may be associated with improvements in patient care and clinical outcomes.

 

Acknowledgments

Dr. Wright is the Anne Gaines and G. Thomas Miller Professor of Medicine which is supported through Hopkins’ Center for Innovative Medicine.

The authors thank Christine Caufield-Noll, MLIS, AHIP (Johns Hopkins Bayview Medical Center, Baltimore, Maryland) for her assistance with this project.

Disclosures

The authors have nothing to disclose.

 

Approximately 60,000 hospitalists were working in the United States in 2018.¹ Hospitalist groups work collaboratively because of the shiftwork required for 24/7 patient coverage, and first-rate clinical documentation is essential for quality care.² Thoughtful clinical documentation not only transmits one provider’s clinical reasoning to other providers but is a professional responsibility.³ Hospitalists spend two-thirds of their time in indirect patient-care activities and approximately one quarter of their time on documentation in electronic health records (EHRs).⁴ Despite documentation occupying a substantial portion of the clinician’s time, published literature on the best practices for the documentation of clinical reasoning in hospital medicine or its assessment remains scant.^5-7

Clinical reasoning involves establishing a diagnosis and developing a therapeutic plan that fits the unique circumstances and needs of the patient.⁸ Inpatient providers who admit patients to the hospital end the admission note with their assessment and plan (A&P) after reflecting about a patient’s presenting illness. The A&P generally represents the interpretations, deductions, and clinical reasoning of the inpatient providers; this is the section of the note that fellow physicians concentrate on over others.⁹ The documentation of clinical reasoning in the A&P allows for many to consider how the recorded interpretations relate to their own elucidations resulting in distributed cognition.¹⁰

Disorganized documentation can contribute to cognitive overload and impede thoughtful consideration about the clinical presentation.³ The assessment of clinical documentation may translate into reduced medical errors and improved note quality.^11,12 Studies that have formally evaluated the documentation of clinical reasoning have focused exclusively on medical students.^13-15 The nonexistence of a detailed rubric for evaluating clinical reasoning in the A&Ps of hospitalists represents a missed opportunity for evaluating what hospitalists “do”; if this evolves into a mechanism for offering formative feedback, such professional development would impact the highest level of Miller’s assessment pyramid.¹⁶ We therefore undertook this study to establish a metric to assess the hospitalist providers’ documentation of clinical reasoning in the A&P of an admission note.

METHODS

Study Design, Setting, and Subjects

This was a retrospective study that reviewed the admission notes of hospitalists for patients admitted over the period of January 2014 and October 2017 at three hospitals in Maryland. One is a community hospital (Hospital A) and two are academic medical centers (Hospital B and Hospital C). Even though these three hospitals are part of one health system, they have distinct cultures and leadership, serve different populations, and are staffed by different provider teams.

 

The notes of physicians working for the hospitalist groups at each of the three hospitals were the focus of the analysis in this study.

Development of the Documentation Assessment Rubric

A team was assembled to develop the Clinical Reasoning in Admission Note Assessment & PLan (CRANAPL) tool. The CRANAPL was designed to assess the comprehensiveness and thoughtfulness of the clinical reasoning documented in the A&P sections of the notes of patients who were admitted to the hospital with an acute illness. Validity evidence for CRANAPL was summarized on the basis of Messick’s unified validity framework by using four of the five sources of validity: content, response process, internal structure, and relations to other variables.¹⁷

Content Validity

The development team consisted of members who have an average of 10 years of clinical experience in hospital medicine; have studied clinical excellence and clinical reasoning; and have expertise in feedback, assessment, and professional development.^18-22 The development of the CRANAPL tool by the team was informed by a review of the clinical reasoning literature, with particular attention paid to the standards and competencies outlined by the Liaison Committee on Medical Education, the Association of American Medical Colleges, the Accreditation Council on Graduate Medical Education, the Internal Medicine Milestone Project, and the Society of Hospital Medicine.^23-26 For each of these parties, diagnostic reasoning and its impact on clinical decision-making are considered to be a core competency. Several works that heavily influenced the CRANAPL tool’s development were Baker’s Interpretive Summary, Differential Diagnosis, Explanation of Reasoning, And Alternatives (IDEA) assessment tool;¹⁴ King’s Pediatric History and Physical Exam Evaluation (P-HAPEE) rubric;¹⁵ and three other studies related to diagnostic reasoning.^16,27,28 These manuscripts and other works substantively informed the preliminary behavioral-based anchors that formed the initial foundation for the tool under development. The CRANAPL tool was shown to colleagues at other institutions who are leaders on clinical reasoning and was presented at academic conferences in the Division of General Internal Medicine and the Division of Hospital Medicine of our institution. Feedback resulted in iterative revisions. The aforementioned methods established content validity evidence for the CRANAPL tool.

Response Process Validity

Several of the authors pilot-tested earlier iterations on admission notes that were excluded from the sample when refining the CRANAPL tool. The weaknesses and sources of confusion with specific items were addressed by scoring 10 A&Ps individually and then comparing data captured on the tool. This cycle was repeated three times for the iterative enhancement and finalization of the CRANAPL tool. On several occasions when two authors were piloting the near-final CRANAPL tool, a third author interviewed each of the two authors about reactivity while assessing individual items and exploring with probes how their own clinical documentation practices were being considered when scoring the notes. The reasonable and thoughtful answers provided by the two authors as they explained and justified the scores they were selecting during the pilot testing served to confer response process validity evidence.

Finalizing the CRANAPL Tool

The nine-item CRANAPL tool includes elements for problem representation, leading diagnosis, uncertainty, differential diagnosis, plans for diagnosis and treatment, estimated length of stay (LOS), potential for upgrade in status to a higher level of care, and consideration of disposition. Although the final three items are not core clinical reasoning domains in the medical education literature, they represent clinical judgments that are especially relevant for the delivery of the high-quality and cost-effective care of hospitalized patients. Given that the probabilities and estimations of these three elements evolve over the course of any hospitalization on the basis of test results and response to therapy, the documentation of initial expectations on these fronts can facilitate distributed cognition with all individuals becoming wiser from shared insights.¹⁰ The tool uses two- and three-point rating scales, with each number score being clearly defined by specific written criteria (total score range: 0-14; Appendix).

 

Data Collection

Hospitalists’ admission notes from the three hospitals were used to validate the CRANAPL tool. Admission notes from patients hospitalized to the general medical floors with an admission diagnosis of either fever, syncope/dizziness, or abdominal pain were used. These diagnoses were purposefully examined because they (1) have a wide differential diagnosis, (2) are common presenting symptoms, and (3) are prone to diagnostic errors.^29-32

The centralized EHR system across the three hospitals identified admission notes with one of these primary diagnoses of patients admitted over the period of January 2014 to October 2017. We submitted a request for 650 admission notes to be randomly selected from the centralized institutional records system. The notes were stratified by hospital and diagnosis. The sample size of our study was comparable with that of prior psychometric validation studies.^33,34 Upon reviewing the A&Ps associated with these admissions, 365 notes were excluded for one of three reasons: (1) the note was written by a nurse practitioner, physician assistant, resident, or medical student; (2) the admission diagnosis had been definitively confirmed in the emergency department (eg, abdominal pain due to diverticulitis seen on CT); and (3) the note represented the fourth or more note by any single provider (to sample notes of many providers, no more than three notes written by any single provider were analyzed). A total of 285 admission notes were ultimately included in the sample.

Data were deidentified, and the A&P sections of the admission notes were each copied from the EHR into a unique Word document. Patient and hospital demographic data (including age, gender, race, number of comorbid conditions, LOS, hospital charges, and readmission to the same health system within 30 days) were collected separately from the EHR. Select physician characteristics were also collected from the hospitalist groups at each of the three hospitals, as was the length (word count) of each A&P.

The study was approved by our institutional review board.

Data Analysis

Two authors scored all deidentified A&Ps by using the finalized version of the CRANAPL tool. Prior to using the CRANAPL tool on each of the notes, these raters read each A&P and scored them by using two single-item rating scales: a global clinical reasoning and a global readability/clarity measure. Both of these global scales used three-item Likert scales (below average, average, and above average). These global rating scales collected the reviewers’ gestalt about the quality and clarity of the A&P. The use of gestalt ratings as comparators is supported by other research.³⁵

Descriptive statistics were computed for all variables. Each rater rescored a sample of 48 records (one month after the initial scoring) and intraclass correlations (ICCs) were computed for intrarater reliability. ICCs were calculated for each item and for the CRANAPL total to determine interrater reliability.

The averaged ratings from the two raters were used for all other analyses. For CRANAPL’s internal structure validity evidence, Cronbach’s alpha was calculated as a measure of internal consistency. For relations to other variables validity evidence, CRANAPL total scores were compared with the two global assessment variables with linear regressions.

Bivariate analyses were performed by applying parametric and nonparametric tests as appropriate. A series of multivariate linear regressions, controlling for diagnosis and clustered variance by hospital site, were performed using CRANAPL total as the dependent variable and patient variables as predictors.

All data were analyzed using Stata (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, Texas: StataCorp LP.)

 

RESULTS

The admission notes of 120 hospitalists were evaluated (Table 1). A total of 39 (33%) physicians were moonlighters with primary appointments outside of the hospitalist division, and 81 (68%) were full-time hospitalists. Among the 120 hospitalists, 48 (40%) were female, 60 (50%) were international medical graduates, and 90 (75%) were of nonwhite race. Most hospitalist physicians (n = 47, 58%) had worked in our health system for less than five years, and 64 hospitalists (53%) devoted greater than 50% of their time to patient care.

Approximately equal numbers of patient admission notes were pulled from each of the three hospitals. The average age of patients was 67.2 (SD 13.6) years, 145 (51%) were female, and 120 (42%) were of nonwhite race. The mean LOS for all patients was 4.0 (SD 3.4) days. A total of 44 (15%) patients were readmitted to the same health system within 30 days of discharge. None of the patients died during the incident hospitalization. The average charge for each of the hospitalizations was $10,646 (SD $9,964).

CRANAPL Data

Figure 1 shows the distribution of the scores given by each rater for each of the nine items. The mean of the total CRANAPL score given by both raters was 6.4 (SD 2.2). Scoring for some items were high (eg, summary statement: 1.5/2), whereas performance on others were low (eg, estimating LOS: 0.1/1 and describing the potential need for upgrade in care: 0.0/1).

Validity of the CRANAPL Tool’s Internal Structure

Cronbach’s alpha, which was used to measure internal consistency within the CRANAPL tool, was 0.43. The ICC, which was applied to measure the interrater reliability for both raters for the total CRANAPL score, was 0.83 (95% CI:  0.76-0.87). The ICC values for intrarater reliability for raters 1 and 2 were 0.73 (95% CI: 0.60-0.83) and 0.73 (95% CI: 0.45-0.86), respectively.

Relations to Other Variables Validity

Associations between CRANAPL total scores, global clinical reasoning, and global scores for note readability/clarity were statistically significant (P < .001), Figure 2.

Eight out of nine CRANAPL variables were statistically significantly different across the three hospitals (P <. 01) when data were analyzed by hospital site. Hospital C had the highest mean score of 7.4 (SD 2.0), followed by Hospital B with a score of 6.6 (SD 2.1), and Hospital A had the lowest total CRANAPL score of 5.2 (SD 1.9). This difference was statistically significant (P < .001). Five variables with respect to admission diagnoses (uncertainty acknowledged, differential diagnosis, plan for diagnosis, plan for treatment, and upgrade plan) were statistically significantly different across notes. Notes for syncope/dizziness generally yielded higher scores than those for abdominal pain and fever.

Factors Associated with High CRANAPL Scores

Table 2 shows the associations between CRANAPL scores and several covariates. Before adjustment, high CRANAPL scores were associated with high word counts of A&Ps (P < .001) and high hospital charges (P < .05). These associations were no longer significant after adjusting for hospital site and admitting diagnoses.

 

DISCUSSION

We reviewed the documentation of clinical reasoning in 285 admission notes at three different hospitals written by hospitalist physicians during routine clinical care. To our knowledge, this is the first study that assessed the documentation of hospitalists’ clinical reasoning with real patient notes. Wide variability exists in the documentation of clinical reasoning within the A&Ps of hospitalists’ admission notes. We have provided validity evidence to support the use of the user-friendly CRANAPL tool.

Prior studies have described rubrics for evaluating the clinical reasoning skills of medical students.^14,15 The ICCs for the IDEA rubric used to assess medical students’ documentation of clinical reasoning were fair to moderate (0.29-0.67), whereas the ICC for the CRANAPL tool was high at 0.83. This measure of reliability is similar to that for the P-HAPEE rubric used to assess medical students’ documentation of pediatric history and physical notes.¹⁵ These data are markedly different from the data in previous studies that have found low interrater reliability for psychometric evaluations related to judgment and decision-making.^36-39 CRANAPL was also found to have high intrarater reliability, which shows the reproducibility of an individual’s assessment over time. The strong association between the total CRANAPL score and global clinical reasoning assessment found in the present study is similar to that found in previous studies that have also embedded global rating scales as comparators when assessing clinical reasoning.^13,,15,40,41 Global rating scales represent an overarching structure for comparison given the absence of an accepted method or gold standard for assessing clinical reasoning documentation. High-quality provider notes are defined by clarity, thoroughness, and accuracy;³⁵ and effective documentation promotes communication and the coordination of care among the members of the care team.³

The total CRANAPL scores varied by hospital site with academic hospitals (B and C) scoring higher than the community hospital (A) in our study. Similarly, lengthy A&Ps were associated with high CRANAPL scores (P < .001) prior to adjustment for hospital site. Healthcare providers consider that the thoroughness of documentation denotes quality and attention to detail.^35,42 Comprehensive documentation takes time; the longer notes by academic hospitalists than those by community hospitalists may be attributed to the fewer number of patients generally carried by hospitalists at academic centers than that by hospitalists at community hospitals.⁴³

The documentation of the estimations of LOS, possibility of potential upgrade, and thoughts about disposition were consistently poorly described across all hospital sites and diagnoses. In contrast to CRANAPL, other clinical reasoning rubrics have excluded these items or discussed uncertainty.^14,15,44 These elements represent the forward thinking that may be essential for high-quality progressive care by hospitalists. Physicians’s difficulty in acknowledging uncertainty has been associated with resource overuse, including the excessive ordering of tests, iatrogenic injury, and heavy financial burden on the healthcare system.^45,46 The lack of thoughtful clinical and management reasoning at the time of admission is believed to be associated with medical errors.⁴⁷ If used as a guide, the CRANAPL tool may promote reflection on the part of the admitting physician. The estimations of LOS, potential for upgrade to a higher level of care, and disposition are markers of optimal inpatient care, especially for hospitalists who work in shifts with embedded handoffs. When shared with colleagues (through documentation), there is the potential for distributed cognition¹⁰ to extend throughout the social network of the hospitalist group. The fact that so few providers are currently including these items in their A&P’s show that the providers are either not performing or documenting the ‘reasoning’. Either way, this is an opportunity that has been highlighted by the CRANAPL tool.

Several limitations of this study should be considered. First, the CRANAPL tool may not have captured elements of optimal clinical reasoning documentation. The reliance on multiple methods and an iterative process in the refinement of the CRANAPL tool should have minimized this. Second, this study was conducted across a single healthcare system that uses the same EHR; this EHR or institutional culture may influence documentation practices and behaviors. Given that using the CRANAPL tool to score an A&P is quick and easy, the benefit of giving providers feedback on their notes remains to be seen—here and at other hospitals. Third, our sample size could limit the generalizability of the results and the significance of the associations. However, the sample assessed in our study was significantly larger than that assessed in other studies that have validated clinical reasoning rubrics.^14,15 Fourth, clinical reasoning is a broad and multidimensional construct. The CRANAPL tool focuses exclusively on hospitalists’ documentation of clinical reasoning and therefore does not assess aspects of clinical reasoning occurring in the physicians’ minds. Finally, given our goal to optimally validate the CRANAPL tool, we chose to test the tool on specific presentations that are known to be associated with diagnostic practice variation and errors. We may have observed different results had we chosen a different set of diagnoses from each hospital. Further validity evidence will be established when applying the CRANPL tool to different diagnoses and to notes from other clinical settings.

In conclusion, this study focuses on the development and validation of the CRANAPL tool that assesses how hospitalists document their clinical reasoning in the A&P section of admission notes. Our results show that wide variability exists in the documentation of clinical reasoning by hospitalists within and across hospitals. Given the CRANAPL tool’s ease-of-use and its versatility, hospitalist divisions in academic and nonacademic settings may use the CRANAPL tool to assess and provide feedback on the documentation of hospitalists’ clinical reasoning. Beyond studying whether physicians can be taught to improve their notes with feedback based on the CRANAPL tool, future studies may explore whether enhancing clinical reasoning documentation may be associated with improvements in patient care and clinical outcomes.

 

Acknowledgments

Dr. Wright is the Anne Gaines and G. Thomas Miller Professor of Medicine which is supported through Hopkins’ Center for Innovative Medicine.

The authors thank Christine Caufield-Noll, MLIS, AHIP (Johns Hopkins Bayview Medical Center, Baltimore, Maryland) for her assistance with this project.

Disclosures

The authors have nothing to disclose.

 

Approximately 60,000 hospitalists were working in the United States in 2018.¹ Hospitalist groups work collaboratively because of the shiftwork required for 24/7 patient coverage, and first-rate clinical documentation is essential for quality care.² Thoughtful clinical documentation not only transmits one provider’s clinical reasoning to other providers but is a professional responsibility.³ Hospitalists spend two-thirds of their time in indirect patient-care activities and approximately one quarter of their time on documentation in electronic health records (EHRs).⁴ Despite documentation occupying a substantial portion of the clinician’s time, published literature on the best practices for the documentation of clinical reasoning in hospital medicine or its assessment remains scant.^5-7

Clinical reasoning involves establishing a diagnosis and developing a therapeutic plan that fits the unique circumstances and needs of the patient.⁸ Inpatient providers who admit patients to the hospital end the admission note with their assessment and plan (A&P) after reflecting about a patient’s presenting illness. The A&P generally represents the interpretations, deductions, and clinical reasoning of the inpatient providers; this is the section of the note that fellow physicians concentrate on over others.⁹ The documentation of clinical reasoning in the A&P allows for many to consider how the recorded interpretations relate to their own elucidations resulting in distributed cognition.¹⁰

Disorganized documentation can contribute to cognitive overload and impede thoughtful consideration about the clinical presentation.³ The assessment of clinical documentation may translate into reduced medical errors and improved note quality.^11,12 Studies that have formally evaluated the documentation of clinical reasoning have focused exclusively on medical students.^13-15 The nonexistence of a detailed rubric for evaluating clinical reasoning in the A&Ps of hospitalists represents a missed opportunity for evaluating what hospitalists “do”; if this evolves into a mechanism for offering formative feedback, such professional development would impact the highest level of Miller’s assessment pyramid.¹⁶ We therefore undertook this study to establish a metric to assess the hospitalist providers’ documentation of clinical reasoning in the A&P of an admission note.

METHODS

Study Design, Setting, and Subjects

This was a retrospective study that reviewed the admission notes of hospitalists for patients admitted over the period of January 2014 and October 2017 at three hospitals in Maryland. One is a community hospital (Hospital A) and two are academic medical centers (Hospital B and Hospital C). Even though these three hospitals are part of one health system, they have distinct cultures and leadership, serve different populations, and are staffed by different provider teams.

 

The notes of physicians working for the hospitalist groups at each of the three hospitals were the focus of the analysis in this study.

Development of the Documentation Assessment Rubric

A team was assembled to develop the Clinical Reasoning in Admission Note Assessment & PLan (CRANAPL) tool. The CRANAPL was designed to assess the comprehensiveness and thoughtfulness of the clinical reasoning documented in the A&P sections of the notes of patients who were admitted to the hospital with an acute illness. Validity evidence for CRANAPL was summarized on the basis of Messick’s unified validity framework by using four of the five sources of validity: content, response process, internal structure, and relations to other variables.¹⁷

Content Validity

The development team consisted of members who have an average of 10 years of clinical experience in hospital medicine; have studied clinical excellence and clinical reasoning; and have expertise in feedback, assessment, and professional development.^18-22 The development of the CRANAPL tool by the team was informed by a review of the clinical reasoning literature, with particular attention paid to the standards and competencies outlined by the Liaison Committee on Medical Education, the Association of American Medical Colleges, the Accreditation Council on Graduate Medical Education, the Internal Medicine Milestone Project, and the Society of Hospital Medicine.^23-26 For each of these parties, diagnostic reasoning and its impact on clinical decision-making are considered to be a core competency. Several works that heavily influenced the CRANAPL tool’s development were Baker’s Interpretive Summary, Differential Diagnosis, Explanation of Reasoning, And Alternatives (IDEA) assessment tool;¹⁴ King’s Pediatric History and Physical Exam Evaluation (P-HAPEE) rubric;¹⁵ and three other studies related to diagnostic reasoning.^16,27,28 These manuscripts and other works substantively informed the preliminary behavioral-based anchors that formed the initial foundation for the tool under development. The CRANAPL tool was shown to colleagues at other institutions who are leaders on clinical reasoning and was presented at academic conferences in the Division of General Internal Medicine and the Division of Hospital Medicine of our institution. Feedback resulted in iterative revisions. The aforementioned methods established content validity evidence for the CRANAPL tool.

Response Process Validity

Several of the authors pilot-tested earlier iterations on admission notes that were excluded from the sample when refining the CRANAPL tool. The weaknesses and sources of confusion with specific items were addressed by scoring 10 A&Ps individually and then comparing data captured on the tool. This cycle was repeated three times for the iterative enhancement and finalization of the CRANAPL tool. On several occasions when two authors were piloting the near-final CRANAPL tool, a third author interviewed each of the two authors about reactivity while assessing individual items and exploring with probes how their own clinical documentation practices were being considered when scoring the notes. The reasonable and thoughtful answers provided by the two authors as they explained and justified the scores they were selecting during the pilot testing served to confer response process validity evidence.

Finalizing the CRANAPL Tool

The nine-item CRANAPL tool includes elements for problem representation, leading diagnosis, uncertainty, differential diagnosis, plans for diagnosis and treatment, estimated length of stay (LOS), potential for upgrade in status to a higher level of care, and consideration of disposition. Although the final three items are not core clinical reasoning domains in the medical education literature, they represent clinical judgments that are especially relevant for the delivery of the high-quality and cost-effective care of hospitalized patients. Given that the probabilities and estimations of these three elements evolve over the course of any hospitalization on the basis of test results and response to therapy, the documentation of initial expectations on these fronts can facilitate distributed cognition with all individuals becoming wiser from shared insights.¹⁰ The tool uses two- and three-point rating scales, with each number score being clearly defined by specific written criteria (total score range: 0-14; Appendix).

 

Data Collection

Hospitalists’ admission notes from the three hospitals were used to validate the CRANAPL tool. Admission notes from patients hospitalized to the general medical floors with an admission diagnosis of either fever, syncope/dizziness, or abdominal pain were used. These diagnoses were purposefully examined because they (1) have a wide differential diagnosis, (2) are common presenting symptoms, and (3) are prone to diagnostic errors.^29-32

The centralized EHR system across the three hospitals identified admission notes with one of these primary diagnoses of patients admitted over the period of January 2014 to October 2017. We submitted a request for 650 admission notes to be randomly selected from the centralized institutional records system. The notes were stratified by hospital and diagnosis. The sample size of our study was comparable with that of prior psychometric validation studies.^33,34 Upon reviewing the A&Ps associated with these admissions, 365 notes were excluded for one of three reasons: (1) the note was written by a nurse practitioner, physician assistant, resident, or medical student; (2) the admission diagnosis had been definitively confirmed in the emergency department (eg, abdominal pain due to diverticulitis seen on CT); and (3) the note represented the fourth or more note by any single provider (to sample notes of many providers, no more than three notes written by any single provider were analyzed). A total of 285 admission notes were ultimately included in the sample.

Data were deidentified, and the A&P sections of the admission notes were each copied from the EHR into a unique Word document. Patient and hospital demographic data (including age, gender, race, number of comorbid conditions, LOS, hospital charges, and readmission to the same health system within 30 days) were collected separately from the EHR. Select physician characteristics were also collected from the hospitalist groups at each of the three hospitals, as was the length (word count) of each A&P.

The study was approved by our institutional review board.

Data Analysis

Two authors scored all deidentified A&Ps by using the finalized version of the CRANAPL tool. Prior to using the CRANAPL tool on each of the notes, these raters read each A&P and scored them by using two single-item rating scales: a global clinical reasoning and a global readability/clarity measure. Both of these global scales used three-item Likert scales (below average, average, and above average). These global rating scales collected the reviewers’ gestalt about the quality and clarity of the A&P. The use of gestalt ratings as comparators is supported by other research.³⁵

Descriptive statistics were computed for all variables. Each rater rescored a sample of 48 records (one month after the initial scoring) and intraclass correlations (ICCs) were computed for intrarater reliability. ICCs were calculated for each item and for the CRANAPL total to determine interrater reliability.

The averaged ratings from the two raters were used for all other analyses. For CRANAPL’s internal structure validity evidence, Cronbach’s alpha was calculated as a measure of internal consistency. For relations to other variables validity evidence, CRANAPL total scores were compared with the two global assessment variables with linear regressions.

Bivariate analyses were performed by applying parametric and nonparametric tests as appropriate. A series of multivariate linear regressions, controlling for diagnosis and clustered variance by hospital site, were performed using CRANAPL total as the dependent variable and patient variables as predictors.

All data were analyzed using Stata (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, Texas: StataCorp LP.)

 

RESULTS

The admission notes of 120 hospitalists were evaluated (Table 1). A total of 39 (33%) physicians were moonlighters with primary appointments outside of the hospitalist division, and 81 (68%) were full-time hospitalists. Among the 120 hospitalists, 48 (40%) were female, 60 (50%) were international medical graduates, and 90 (75%) were of nonwhite race. Most hospitalist physicians (n = 47, 58%) had worked in our health system for less than five years, and 64 hospitalists (53%) devoted greater than 50% of their time to patient care.

Approximately equal numbers of patient admission notes were pulled from each of the three hospitals. The average age of patients was 67.2 (SD 13.6) years, 145 (51%) were female, and 120 (42%) were of nonwhite race. The mean LOS for all patients was 4.0 (SD 3.4) days. A total of 44 (15%) patients were readmitted to the same health system within 30 days of discharge. None of the patients died during the incident hospitalization. The average charge for each of the hospitalizations was $10,646 (SD $9,964).

CRANAPL Data

Figure 1 shows the distribution of the scores given by each rater for each of the nine items. The mean of the total CRANAPL score given by both raters was 6.4 (SD 2.2). Scoring for some items were high (eg, summary statement: 1.5/2), whereas performance on others were low (eg, estimating LOS: 0.1/1 and describing the potential need for upgrade in care: 0.0/1).

Validity of the CRANAPL Tool’s Internal Structure

Cronbach’s alpha, which was used to measure internal consistency within the CRANAPL tool, was 0.43. The ICC, which was applied to measure the interrater reliability for both raters for the total CRANAPL score, was 0.83 (95% CI:  0.76-0.87). The ICC values for intrarater reliability for raters 1 and 2 were 0.73 (95% CI: 0.60-0.83) and 0.73 (95% CI: 0.45-0.86), respectively.

Relations to Other Variables Validity

Associations between CRANAPL total scores, global clinical reasoning, and global scores for note readability/clarity were statistically significant (P < .001), Figure 2.

Eight out of nine CRANAPL variables were statistically significantly different across the three hospitals (P <. 01) when data were analyzed by hospital site. Hospital C had the highest mean score of 7.4 (SD 2.0), followed by Hospital B with a score of 6.6 (SD 2.1), and Hospital A had the lowest total CRANAPL score of 5.2 (SD 1.9). This difference was statistically significant (P < .001). Five variables with respect to admission diagnoses (uncertainty acknowledged, differential diagnosis, plan for diagnosis, plan for treatment, and upgrade plan) were statistically significantly different across notes. Notes for syncope/dizziness generally yielded higher scores than those for abdominal pain and fever.

Factors Associated with High CRANAPL Scores

Table 2 shows the associations between CRANAPL scores and several covariates. Before adjustment, high CRANAPL scores were associated with high word counts of A&Ps (P < .001) and high hospital charges (P < .05). These associations were no longer significant after adjusting for hospital site and admitting diagnoses.

 

DISCUSSION

We reviewed the documentation of clinical reasoning in 285 admission notes at three different hospitals written by hospitalist physicians during routine clinical care. To our knowledge, this is the first study that assessed the documentation of hospitalists’ clinical reasoning with real patient notes. Wide variability exists in the documentation of clinical reasoning within the A&Ps of hospitalists’ admission notes. We have provided validity evidence to support the use of the user-friendly CRANAPL tool.

Prior studies have described rubrics for evaluating the clinical reasoning skills of medical students.^14,15 The ICCs for the IDEA rubric used to assess medical students’ documentation of clinical reasoning were fair to moderate (0.29-0.67), whereas the ICC for the CRANAPL tool was high at 0.83. This measure of reliability is similar to that for the P-HAPEE rubric used to assess medical students’ documentation of pediatric history and physical notes.¹⁵ These data are markedly different from the data in previous studies that have found low interrater reliability for psychometric evaluations related to judgment and decision-making.^36-39 CRANAPL was also found to have high intrarater reliability, which shows the reproducibility of an individual’s assessment over time. The strong association between the total CRANAPL score and global clinical reasoning assessment found in the present study is similar to that found in previous studies that have also embedded global rating scales as comparators when assessing clinical reasoning.^13,,15,40,41 Global rating scales represent an overarching structure for comparison given the absence of an accepted method or gold standard for assessing clinical reasoning documentation. High-quality provider notes are defined by clarity, thoroughness, and accuracy;³⁵ and effective documentation promotes communication and the coordination of care among the members of the care team.³

The total CRANAPL scores varied by hospital site with academic hospitals (B and C) scoring higher than the community hospital (A) in our study. Similarly, lengthy A&Ps were associated with high CRANAPL scores (P < .001) prior to adjustment for hospital site. Healthcare providers consider that the thoroughness of documentation denotes quality and attention to detail.^35,42 Comprehensive documentation takes time; the longer notes by academic hospitalists than those by community hospitalists may be attributed to the fewer number of patients generally carried by hospitalists at academic centers than that by hospitalists at community hospitals.⁴³

The documentation of the estimations of LOS, possibility of potential upgrade, and thoughts about disposition were consistently poorly described across all hospital sites and diagnoses. In contrast to CRANAPL, other clinical reasoning rubrics have excluded these items or discussed uncertainty.^14,15,44 These elements represent the forward thinking that may be essential for high-quality progressive care by hospitalists. Physicians’s difficulty in acknowledging uncertainty has been associated with resource overuse, including the excessive ordering of tests, iatrogenic injury, and heavy financial burden on the healthcare system.^45,46 The lack of thoughtful clinical and management reasoning at the time of admission is believed to be associated with medical errors.⁴⁷ If used as a guide, the CRANAPL tool may promote reflection on the part of the admitting physician. The estimations of LOS, potential for upgrade to a higher level of care, and disposition are markers of optimal inpatient care, especially for hospitalists who work in shifts with embedded handoffs. When shared with colleagues (through documentation), there is the potential for distributed cognition¹⁰ to extend throughout the social network of the hospitalist group. The fact that so few providers are currently including these items in their A&P’s show that the providers are either not performing or documenting the ‘reasoning’. Either way, this is an opportunity that has been highlighted by the CRANAPL tool.

Several limitations of this study should be considered. First, the CRANAPL tool may not have captured elements of optimal clinical reasoning documentation. The reliance on multiple methods and an iterative process in the refinement of the CRANAPL tool should have minimized this. Second, this study was conducted across a single healthcare system that uses the same EHR; this EHR or institutional culture may influence documentation practices and behaviors. Given that using the CRANAPL tool to score an A&P is quick and easy, the benefit of giving providers feedback on their notes remains to be seen—here and at other hospitals. Third, our sample size could limit the generalizability of the results and the significance of the associations. However, the sample assessed in our study was significantly larger than that assessed in other studies that have validated clinical reasoning rubrics.^14,15 Fourth, clinical reasoning is a broad and multidimensional construct. The CRANAPL tool focuses exclusively on hospitalists’ documentation of clinical reasoning and therefore does not assess aspects of clinical reasoning occurring in the physicians’ minds. Finally, given our goal to optimally validate the CRANAPL tool, we chose to test the tool on specific presentations that are known to be associated with diagnostic practice variation and errors. We may have observed different results had we chosen a different set of diagnoses from each hospital. Further validity evidence will be established when applying the CRANPL tool to different diagnoses and to notes from other clinical settings.

In conclusion, this study focuses on the development and validation of the CRANAPL tool that assesses how hospitalists document their clinical reasoning in the A&P section of admission notes. Our results show that wide variability exists in the documentation of clinical reasoning by hospitalists within and across hospitals. Given the CRANAPL tool’s ease-of-use and its versatility, hospitalist divisions in academic and nonacademic settings may use the CRANAPL tool to assess and provide feedback on the documentation of hospitalists’ clinical reasoning. Beyond studying whether physicians can be taught to improve their notes with feedback based on the CRANAPL tool, future studies may explore whether enhancing clinical reasoning documentation may be associated with improvements in patient care and clinical outcomes.

 

Acknowledgments

Dr. Wright is the Anne Gaines and G. Thomas Miller Professor of Medicine which is supported through Hopkins’ Center for Innovative Medicine.

The authors thank Christine Caufield-Noll, MLIS, AHIP (Johns Hopkins Bayview Medical Center, Baltimore, Maryland) for her assistance with this project.

Disclosures

The authors have nothing to disclose.

 

ABSTRACT

Purpose To avoid disruption of administrative and clinical workflow in an increasingly complex system of health information technology, health care systems and providers have started using medical scribes. The purpose of this study was to investigate the impact of medical scribes on patient satisfaction, physician satisfaction, and quality measure documentation in a family medicine office.

Methods We reviewed 1000 electronic health records for documentation of specified quality measures in the family medicine setting, before and after the use of medical scribes. We surveyed 150 patients on attitude, comfort, and acceptance of medical scribes during their visit. Five physicians shared their perceptions related to productivity, efficiency, and overall job satisfaction on working with medical scribes.

Results Documentation of 4 quality measures improved with the use of scribes, demonstrating statistical significance: fall risk assessment (odds ratio [OR] = 5.5; P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Patients reported comfort with scribes in the examination room (96%, n = 144), a more focused health care provider (76%, n = 113), increased efficiency (74%, n = 109), and a higher degree of satisfaction with the office visit (61%, n = 90). Physicians believed they were providing better care and developing better relationships with patients while spending less time documenting and experiencing less stress.

Conclusions Use of medical scribes in a primary care setting was associated with higher patient and physician satisfaction. Patients felt comfortable with a medical scribe in the room, attested to their professionalism, and understood their purpose during the visit. The use of medical scribes in this primary care setting improved documentation of 4 quality measures.

[polldaddy:10339849]

The widespread implementation and adoption of electronic health records (EHRs) continues to increase, primarily motivated by federal incentives through the Centers for Medicare and Medicaid Services to positively impact patient care. Physician use of the EHR in the exam room has the potential to affect the patient-physician relationship, patient satisfaction, physician satisfaction, physician productivity, and physician reimbursement. In the United States, the Health Information Technology for Economic and Clinical Health Act of 2009 established incentive programs to promote meaningful use of EHRs in primary care.¹ Integrating EHRs into physician practice, adoption of meaningful use, and the increasing challenge of pay-for-performance quality measures have generated additional hours of administrative work for health care providers. These intrusions on routine clinical care, while hypothesized to improve care, have diminished physician satisfaction, increased stress, and contributed to physician burnout.²

The expanded role of clinicians incentivized to capture metrics for value-based care introduces an unprecedented level of multitasking required at the point of care. In a clinical setting, multitasking undermines the core clinical activities of observation, communication, problem solving, and, ultimately, the development of trusting relationships.^3,4 EHR documentation creates a barrier to patient engagement and may contribute to patients feeling isolated when unable to view data being entered.^5,6

Potential benefits of scribes. One means of increasing physician satisfaction and productivity may be the integration of medical scribes into health care systems. Medical scribes do not operate independently but are able to document activities or receive dictation critical for patient management—eg, recording patient histories, documenting physical examination findings and procedures, and following up on lab reports.⁷

Continue to: In a 2015 systematic review...

 

In a 2015 systematic review, Shultz and Holmstrom found that medical scribes in specialty settings may improve clinician satisfaction, productivity, time-related efficiency, revenue, and patient-clinician interactions.⁸ The use of scribes in one study increased the number of patients seen and time saved by emergency physicians, thereby increasing physician productivity.⁹ Studies have also shown that physicians were more satisfied during scribe engagement, related to increased time spent with patients, decreased work-related stress, and increased overall workplace satisfaction.^10-12

Sixty-one percent of patients were more satisfied with their office visit with a scribe present.

Studies on the use of medical scribes have mainly focused on physician satisfaction and productivity; however, the data on patient satisfaction are limited. Data about the use of the medical scribe in the primary care setting are also limited. The aim of our research was threefold. We wanted to evaluate the effects of using a medical scribe on: (1) patient satisfaction, (2) documentation of primary care pay-for-performance quality measures, and (3) physicians’ perceptions of the use of scribes in the primary care setting.

METHODS

Data collection

This study was conducted at Family Practice Group in Arlington, Massachusetts, where 5 part-time physicians and 3 full-time physician assistants see approximately 400 patients each week. The representative patient population is approximately 80% privately insured, 10% Medicaid, and 10% Medicare. The EHR system is eClinicalWorks.

The scribes were undergraduate college students who were interested in careers as health care professionals. They had no scribe training or experience working in a medical office. These scribes underwent 4 hours of training in EHR functionality, pay-for-performance quality measures, and risk coding (using appropriate medical codes that capture the patient’s level of medical complexity). The Independent Physician Association affiliated with Family Practice Group provided this training at no cost to the practice. The 3 scribes worked full-time with the 5 part-time physicians in the study. Scribes were not required to have had a medical background prior to entering the program.

After the aforementioned training, scribes began working full-time with physicians during patient visits and continued learning on the job through feedback from supervising physicians. Scribes documented the patient encounters, recording medical and social histories and physical exam findings, and transcribing discussions of treatment plans and physicians’ instructions to patients.

Continue to: We reviewed patient EHRs...

 

We reviewed patient EHRs of 5 family physicians over 2 time periods: the 3 months prior to having a medical scribe and the 3 months after beginning to work with a medical scribe. Chart data extraction occurred from 4/11/13 to 8/28/14. We reviewed 1000 patient EHRs—100 EHRs each for the 5 participating physicians before and after scribe use. Selected EHRs ran chronologically from the start of each 3-month period. Reviewing EHRs at 3 months after the onset of the medical scribe program allowed time for the scribes to be fully integrated into the practice and confident in their job responsibilities. Chart review was performed by an office administrator who was blinded as to whether documentation had been done with or without a scribe present during the visit.

Eight quality measures were evaluated in chart review. These measures were drawn from the Healthcare Effectiveness Data and Information Set (HEDIS), a tool used to measure performance in medical care and service.

We surveyed 30 patients of each of the 5 providers, yielding a total of 150 survey responses. A medical assistant gave surveys to patients in the exam room following each office visit, to be completed anonymously and privately. Patients were told that surveys would take less than 2 minutes to complete. Office visits included episodic visits, physical exams, and chronic disease management.

Physicians believed they were saving, on average, 1.5 hours each day with the use of a scribe.

After the trial period, we surveyed participating physicians regarding medical scribe assistance with documentation. We also asked the physicians 3 open-ended questions regarding their experiences with their medical scribe.

This study was reviewed and approved (IRB Approval #11424) by the Tufts Health Science Campus Institutional Review Board.

Continue to: Data analysis

 

 

Data analysis

During chart review, we assessed the rate at which documentation was completed for 8 quality outcome measures commonly used in the primary care setting (TABLE 1), before and after the introduction of medical scribes. These quality measures and pertinent descriptors are listed in TABLE 2.¹³ Presence or absence of documentation on all quality measures was noted for all applicable patients.

One hundred fifty patients were surveyed immediately after their office visit on their perceptions of medical scribes, including their attitude toward, comfort with, and acceptance of medical scribes (TABLE 3). Five participating physicians were surveyed to assess their perceptions related to productivity and job satisfaction with the use of medical scribes (TABLE 4), and regarding time saved and additional patients seen. Those who collected and analyzed the data from the surveys were blinded to patient and physician identifiers.

Statistical analysis

Using chi-squared tests, we compared the number of positive documentations for the 8 outcome measures before and after the use of medical scribes. Two-sided P values < .05 were considered statistically significant. All statistical analyses were performed with the use of STATA version 9 (StataCorp LP. College Station, Tex).

Physician survey data were calculated on a Likert scale, with a score of 1 corresponding to “strongly disagree,” 2 “disagree,” 3 “neither agree nor disagree,” 4 “agree,” and 5 “strongly agree.” Using the 5 answers generated from the 5 physicians, we calculated the mean for each question.

RESULTS

The use of scribes demonstrated a statistically significant improvement in the documentation of 4 (out of 8) pay-for-performance measures (TABLE 1): fall risk assessment (odds ratio [OR] = 5.5, P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index (BMI) plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Sample sizes of each quality measure vary as there were differing numbers of applicable patients for each quality measure within the overall 1000 charts.

Continue to: We established at the beginning...

 

We established at the beginning of the study a target of obtaining surveys from 30 patients of each of the 5 physicians (total of 150). Response rates for surveys were 100% for both the 150 patients and the 5 physicians. No patients declined to complete the survey, although some did not answer every question.

Patients generally had positive experiences with medical scribes (TABLE 3). The majority of patients (96%, n = 144) felt comfortable with the scribe in the room during the visit with their provider. Patients felt that the provider focused on them “a little to a lot more” (75.8%, n = 113) and thought their visit was more efficient (73.6%, n = 109) as a result of the scribe being present vs not being present. Most patients were more satisfied with their office visit with the scribe being present (60.8%, n = 90).

Physicians felt that working with a medical scribe helped them connect with their patients, made patients feel that their physician was more attentive to them, contributed to better patient care, decreased the time they spent documenting in EHR, and contributed to faster work flow (TABLE 4). The physicians also believed they had saved a mean of 1.5 hours each day with the use of a medical scribe, and that they did not have to change their schedule in any way to accommodate additional patients as a result of having a scribe.

DISCUSSION

Documentation of fall risk assessment, follow-up tobacco screening, follow-up BMI plan, and follow-up blood pressure plan all demonstrated statistically significant increases with the use of medical scribes compared with practice before scribes. Follow-up depression screen and transition of care management had relatively high ORs (3.2 and 8, respectively), but did not yield statistically significant values, in part due to small sample sizes as the number of patients who were hospitalized and the number of patients who screened positive for depression were relatively small out of the total group of 1000 patients. The use of scribes had little effect on depression screen and tobacco screen. This is likely due to the fact that there were already effective office systems in place at the practice that alerted medical assistants to complete these screens for each appropriate patient.

We found that the use of medical scribes in a primary care setting was associated with both higher patient and physician satisfaction. Although the 5 physicians in this study chose not to see additional patients when using a medical scribe, they believed they were saving, on average, 1.5 hours of time each day with the use of a scribe. All 5 physicians reported that medical scribes enabled them to provide better patient care and to help patients feel as though they had more of the physician’s attention. Patient respondents attested to their provider focusing on them more during the visit. According to patient surveys, 40.4% of respondents felt that physicians addressed their concerns more thoroughly during the visit, while the remainder of patients did not.

Continue to: Some concerns...

 

Some concerns of introducing medical scribes into a health care system include possible patient discomfort with a third party being present during the visit and the cost of employing medical scribes. In this study, the vast majority of patients (96%) felt comfortable with a scribe in the room. Future research could compare patient discomfort due to the presence of a medical scribe with patient discomfort due to a physician using a computer during the visit.

Limitations of this study include the small sample size of both physicians and patients; a lack of validated measures for calculating productivity, time/efficiency, and overall satisfaction; and short time periods leading up to and following the introduction of medical scribes. In addition, EHRs of patients were chosen sequentially and not randomly, which could be a confounder. Participating physicians were aware of being studied; therefore, documentation could have been affected by the Hawthorne effect. The study also was limited to one family medicine site. Although improved documentation of primary care pay-for-performance quality measures was reported, wide confidence intervals and small patient numbers hindered generalizability of findings.

Documentation of 4 out of 8 pay-forperformance measures showed statistically significant improvement with the use of scribes.

Additional studies are needed with a robust analytic plan sufficient to demonstrate baseline provider familiarity with EHRs, accuracy of medical scribe documentation, and improved documentation of pay-for-performance quality measures. Additional investigation regarding the variable competency of different medical scribes could be useful in measuring the effects of the scribe on a variety of outcomes related to both the physician and patient.

It is possible that the improved documentation yielded by the use of medical scribes could generate billing codes that reimburse physicians at a higher level (eg, a higher ratio of 99214 to 99213), leading to increased pay. Future research could aim to quantify this source of increased revenue. Furthermore, investigations could aim to quantify the revenue that medical scribes generate via improved quality measure pay-for-performance documentation.

CORRESPONDENCE
Jessica Platt, MD, 195 Canal Street, Malden, MA 02148; [email protected].

ABSTRACT

Purpose To avoid disruption of administrative and clinical workflow in an increasingly complex system of health information technology, health care systems and providers have started using medical scribes. The purpose of this study was to investigate the impact of medical scribes on patient satisfaction, physician satisfaction, and quality measure documentation in a family medicine office.

Methods We reviewed 1000 electronic health records for documentation of specified quality measures in the family medicine setting, before and after the use of medical scribes. We surveyed 150 patients on attitude, comfort, and acceptance of medical scribes during their visit. Five physicians shared their perceptions related to productivity, efficiency, and overall job satisfaction on working with medical scribes.

Results Documentation of 4 quality measures improved with the use of scribes, demonstrating statistical significance: fall risk assessment (odds ratio [OR] = 5.5; P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Patients reported comfort with scribes in the examination room (96%, n = 144), a more focused health care provider (76%, n = 113), increased efficiency (74%, n = 109), and a higher degree of satisfaction with the office visit (61%, n = 90). Physicians believed they were providing better care and developing better relationships with patients while spending less time documenting and experiencing less stress.

Conclusions Use of medical scribes in a primary care setting was associated with higher patient and physician satisfaction. Patients felt comfortable with a medical scribe in the room, attested to their professionalism, and understood their purpose during the visit. The use of medical scribes in this primary care setting improved documentation of 4 quality measures.

[polldaddy:10339849]

The widespread implementation and adoption of electronic health records (EHRs) continues to increase, primarily motivated by federal incentives through the Centers for Medicare and Medicaid Services to positively impact patient care. Physician use of the EHR in the exam room has the potential to affect the patient-physician relationship, patient satisfaction, physician satisfaction, physician productivity, and physician reimbursement. In the United States, the Health Information Technology for Economic and Clinical Health Act of 2009 established incentive programs to promote meaningful use of EHRs in primary care.¹ Integrating EHRs into physician practice, adoption of meaningful use, and the increasing challenge of pay-for-performance quality measures have generated additional hours of administrative work for health care providers. These intrusions on routine clinical care, while hypothesized to improve care, have diminished physician satisfaction, increased stress, and contributed to physician burnout.²

The expanded role of clinicians incentivized to capture metrics for value-based care introduces an unprecedented level of multitasking required at the point of care. In a clinical setting, multitasking undermines the core clinical activities of observation, communication, problem solving, and, ultimately, the development of trusting relationships.^3,4 EHR documentation creates a barrier to patient engagement and may contribute to patients feeling isolated when unable to view data being entered.^5,6

Potential benefits of scribes. One means of increasing physician satisfaction and productivity may be the integration of medical scribes into health care systems. Medical scribes do not operate independently but are able to document activities or receive dictation critical for patient management—eg, recording patient histories, documenting physical examination findings and procedures, and following up on lab reports.⁷

Continue to: In a 2015 systematic review...

 

In a 2015 systematic review, Shultz and Holmstrom found that medical scribes in specialty settings may improve clinician satisfaction, productivity, time-related efficiency, revenue, and patient-clinician interactions.⁸ The use of scribes in one study increased the number of patients seen and time saved by emergency physicians, thereby increasing physician productivity.⁹ Studies have also shown that physicians were more satisfied during scribe engagement, related to increased time spent with patients, decreased work-related stress, and increased overall workplace satisfaction.^10-12

Sixty-one percent of patients were more satisfied with their office visit with a scribe present.

Studies on the use of medical scribes have mainly focused on physician satisfaction and productivity; however, the data on patient satisfaction are limited. Data about the use of the medical scribe in the primary care setting are also limited. The aim of our research was threefold. We wanted to evaluate the effects of using a medical scribe on: (1) patient satisfaction, (2) documentation of primary care pay-for-performance quality measures, and (3) physicians’ perceptions of the use of scribes in the primary care setting.

METHODS

Data collection

This study was conducted at Family Practice Group in Arlington, Massachusetts, where 5 part-time physicians and 3 full-time physician assistants see approximately 400 patients each week. The representative patient population is approximately 80% privately insured, 10% Medicaid, and 10% Medicare. The EHR system is eClinicalWorks.

The scribes were undergraduate college students who were interested in careers as health care professionals. They had no scribe training or experience working in a medical office. These scribes underwent 4 hours of training in EHR functionality, pay-for-performance quality measures, and risk coding (using appropriate medical codes that capture the patient’s level of medical complexity). The Independent Physician Association affiliated with Family Practice Group provided this training at no cost to the practice. The 3 scribes worked full-time with the 5 part-time physicians in the study. Scribes were not required to have had a medical background prior to entering the program.

After the aforementioned training, scribes began working full-time with physicians during patient visits and continued learning on the job through feedback from supervising physicians. Scribes documented the patient encounters, recording medical and social histories and physical exam findings, and transcribing discussions of treatment plans and physicians’ instructions to patients.

Continue to: We reviewed patient EHRs...

 

We reviewed patient EHRs of 5 family physicians over 2 time periods: the 3 months prior to having a medical scribe and the 3 months after beginning to work with a medical scribe. Chart data extraction occurred from 4/11/13 to 8/28/14. We reviewed 1000 patient EHRs—100 EHRs each for the 5 participating physicians before and after scribe use. Selected EHRs ran chronologically from the start of each 3-month period. Reviewing EHRs at 3 months after the onset of the medical scribe program allowed time for the scribes to be fully integrated into the practice and confident in their job responsibilities. Chart review was performed by an office administrator who was blinded as to whether documentation had been done with or without a scribe present during the visit.

Eight quality measures were evaluated in chart review. These measures were drawn from the Healthcare Effectiveness Data and Information Set (HEDIS), a tool used to measure performance in medical care and service.

We surveyed 30 patients of each of the 5 providers, yielding a total of 150 survey responses. A medical assistant gave surveys to patients in the exam room following each office visit, to be completed anonymously and privately. Patients were told that surveys would take less than 2 minutes to complete. Office visits included episodic visits, physical exams, and chronic disease management.

Physicians believed they were saving, on average, 1.5 hours each day with the use of a scribe.

After the trial period, we surveyed participating physicians regarding medical scribe assistance with documentation. We also asked the physicians 3 open-ended questions regarding their experiences with their medical scribe.

This study was reviewed and approved (IRB Approval #11424) by the Tufts Health Science Campus Institutional Review Board.

Continue to: Data analysis

 

 

Data analysis

During chart review, we assessed the rate at which documentation was completed for 8 quality outcome measures commonly used in the primary care setting (TABLE 1), before and after the introduction of medical scribes. These quality measures and pertinent descriptors are listed in TABLE 2.¹³ Presence or absence of documentation on all quality measures was noted for all applicable patients.

One hundred fifty patients were surveyed immediately after their office visit on their perceptions of medical scribes, including their attitude toward, comfort with, and acceptance of medical scribes (TABLE 3). Five participating physicians were surveyed to assess their perceptions related to productivity and job satisfaction with the use of medical scribes (TABLE 4), and regarding time saved and additional patients seen. Those who collected and analyzed the data from the surveys were blinded to patient and physician identifiers.

Statistical analysis

Using chi-squared tests, we compared the number of positive documentations for the 8 outcome measures before and after the use of medical scribes. Two-sided P values < .05 were considered statistically significant. All statistical analyses were performed with the use of STATA version 9 (StataCorp LP. College Station, Tex).

Physician survey data were calculated on a Likert scale, with a score of 1 corresponding to “strongly disagree,” 2 “disagree,” 3 “neither agree nor disagree,” 4 “agree,” and 5 “strongly agree.” Using the 5 answers generated from the 5 physicians, we calculated the mean for each question.

RESULTS

The use of scribes demonstrated a statistically significant improvement in the documentation of 4 (out of 8) pay-for-performance measures (TABLE 1): fall risk assessment (odds ratio [OR] = 5.5, P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index (BMI) plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Sample sizes of each quality measure vary as there were differing numbers of applicable patients for each quality measure within the overall 1000 charts.

Continue to: We established at the beginning...

 

We established at the beginning of the study a target of obtaining surveys from 30 patients of each of the 5 physicians (total of 150). Response rates for surveys were 100% for both the 150 patients and the 5 physicians. No patients declined to complete the survey, although some did not answer every question.

Patients generally had positive experiences with medical scribes (TABLE 3). The majority of patients (96%, n = 144) felt comfortable with the scribe in the room during the visit with their provider. Patients felt that the provider focused on them “a little to a lot more” (75.8%, n = 113) and thought their visit was more efficient (73.6%, n = 109) as a result of the scribe being present vs not being present. Most patients were more satisfied with their office visit with the scribe being present (60.8%, n = 90).

Physicians felt that working with a medical scribe helped them connect with their patients, made patients feel that their physician was more attentive to them, contributed to better patient care, decreased the time they spent documenting in EHR, and contributed to faster work flow (TABLE 4). The physicians also believed they had saved a mean of 1.5 hours each day with the use of a medical scribe, and that they did not have to change their schedule in any way to accommodate additional patients as a result of having a scribe.

DISCUSSION

Documentation of fall risk assessment, follow-up tobacco screening, follow-up BMI plan, and follow-up blood pressure plan all demonstrated statistically significant increases with the use of medical scribes compared with practice before scribes. Follow-up depression screen and transition of care management had relatively high ORs (3.2 and 8, respectively), but did not yield statistically significant values, in part due to small sample sizes as the number of patients who were hospitalized and the number of patients who screened positive for depression were relatively small out of the total group of 1000 patients. The use of scribes had little effect on depression screen and tobacco screen. This is likely due to the fact that there were already effective office systems in place at the practice that alerted medical assistants to complete these screens for each appropriate patient.

We found that the use of medical scribes in a primary care setting was associated with both higher patient and physician satisfaction. Although the 5 physicians in this study chose not to see additional patients when using a medical scribe, they believed they were saving, on average, 1.5 hours of time each day with the use of a scribe. All 5 physicians reported that medical scribes enabled them to provide better patient care and to help patients feel as though they had more of the physician’s attention. Patient respondents attested to their provider focusing on them more during the visit. According to patient surveys, 40.4% of respondents felt that physicians addressed their concerns more thoroughly during the visit, while the remainder of patients did not.

Continue to: Some concerns...

 

Some concerns of introducing medical scribes into a health care system include possible patient discomfort with a third party being present during the visit and the cost of employing medical scribes. In this study, the vast majority of patients (96%) felt comfortable with a scribe in the room. Future research could compare patient discomfort due to the presence of a medical scribe with patient discomfort due to a physician using a computer during the visit.

Limitations of this study include the small sample size of both physicians and patients; a lack of validated measures for calculating productivity, time/efficiency, and overall satisfaction; and short time periods leading up to and following the introduction of medical scribes. In addition, EHRs of patients were chosen sequentially and not randomly, which could be a confounder. Participating physicians were aware of being studied; therefore, documentation could have been affected by the Hawthorne effect. The study also was limited to one family medicine site. Although improved documentation of primary care pay-for-performance quality measures was reported, wide confidence intervals and small patient numbers hindered generalizability of findings.

Documentation of 4 out of 8 pay-forperformance measures showed statistically significant improvement with the use of scribes.

Additional studies are needed with a robust analytic plan sufficient to demonstrate baseline provider familiarity with EHRs, accuracy of medical scribe documentation, and improved documentation of pay-for-performance quality measures. Additional investigation regarding the variable competency of different medical scribes could be useful in measuring the effects of the scribe on a variety of outcomes related to both the physician and patient.

It is possible that the improved documentation yielded by the use of medical scribes could generate billing codes that reimburse physicians at a higher level (eg, a higher ratio of 99214 to 99213), leading to increased pay. Future research could aim to quantify this source of increased revenue. Furthermore, investigations could aim to quantify the revenue that medical scribes generate via improved quality measure pay-for-performance documentation.

CORRESPONDENCE
Jessica Platt, MD, 195 Canal Street, Malden, MA 02148; [email protected].

ABSTRACT

Purpose To avoid disruption of administrative and clinical workflow in an increasingly complex system of health information technology, health care systems and providers have started using medical scribes. The purpose of this study was to investigate the impact of medical scribes on patient satisfaction, physician satisfaction, and quality measure documentation in a family medicine office.

Methods We reviewed 1000 electronic health records for documentation of specified quality measures in the family medicine setting, before and after the use of medical scribes. We surveyed 150 patients on attitude, comfort, and acceptance of medical scribes during their visit. Five physicians shared their perceptions related to productivity, efficiency, and overall job satisfaction on working with medical scribes.

Results Documentation of 4 quality measures improved with the use of scribes, demonstrating statistical significance: fall risk assessment (odds ratio [OR] = 5.5; P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Patients reported comfort with scribes in the examination room (96%, n = 144), a more focused health care provider (76%, n = 113), increased efficiency (74%, n = 109), and a higher degree of satisfaction with the office visit (61%, n = 90). Physicians believed they were providing better care and developing better relationships with patients while spending less time documenting and experiencing less stress.

Conclusions Use of medical scribes in a primary care setting was associated with higher patient and physician satisfaction. Patients felt comfortable with a medical scribe in the room, attested to their professionalism, and understood their purpose during the visit. The use of medical scribes in this primary care setting improved documentation of 4 quality measures.

[polldaddy:10339849]

The widespread implementation and adoption of electronic health records (EHRs) continues to increase, primarily motivated by federal incentives through the Centers for Medicare and Medicaid Services to positively impact patient care. Physician use of the EHR in the exam room has the potential to affect the patient-physician relationship, patient satisfaction, physician satisfaction, physician productivity, and physician reimbursement. In the United States, the Health Information Technology for Economic and Clinical Health Act of 2009 established incentive programs to promote meaningful use of EHRs in primary care.¹ Integrating EHRs into physician practice, adoption of meaningful use, and the increasing challenge of pay-for-performance quality measures have generated additional hours of administrative work for health care providers. These intrusions on routine clinical care, while hypothesized to improve care, have diminished physician satisfaction, increased stress, and contributed to physician burnout.²

The expanded role of clinicians incentivized to capture metrics for value-based care introduces an unprecedented level of multitasking required at the point of care. In a clinical setting, multitasking undermines the core clinical activities of observation, communication, problem solving, and, ultimately, the development of trusting relationships.^3,4 EHR documentation creates a barrier to patient engagement and may contribute to patients feeling isolated when unable to view data being entered.^5,6

Potential benefits of scribes. One means of increasing physician satisfaction and productivity may be the integration of medical scribes into health care systems. Medical scribes do not operate independently but are able to document activities or receive dictation critical for patient management—eg, recording patient histories, documenting physical examination findings and procedures, and following up on lab reports.⁷

Continue to: In a 2015 systematic review...

 

In a 2015 systematic review, Shultz and Holmstrom found that medical scribes in specialty settings may improve clinician satisfaction, productivity, time-related efficiency, revenue, and patient-clinician interactions.⁸ The use of scribes in one study increased the number of patients seen and time saved by emergency physicians, thereby increasing physician productivity.⁹ Studies have also shown that physicians were more satisfied during scribe engagement, related to increased time spent with patients, decreased work-related stress, and increased overall workplace satisfaction.^10-12

Sixty-one percent of patients were more satisfied with their office visit with a scribe present.

Studies on the use of medical scribes have mainly focused on physician satisfaction and productivity; however, the data on patient satisfaction are limited. Data about the use of the medical scribe in the primary care setting are also limited. The aim of our research was threefold. We wanted to evaluate the effects of using a medical scribe on: (1) patient satisfaction, (2) documentation of primary care pay-for-performance quality measures, and (3) physicians’ perceptions of the use of scribes in the primary care setting.

METHODS

Data collection

This study was conducted at Family Practice Group in Arlington, Massachusetts, where 5 part-time physicians and 3 full-time physician assistants see approximately 400 patients each week. The representative patient population is approximately 80% privately insured, 10% Medicaid, and 10% Medicare. The EHR system is eClinicalWorks.

The scribes were undergraduate college students who were interested in careers as health care professionals. They had no scribe training or experience working in a medical office. These scribes underwent 4 hours of training in EHR functionality, pay-for-performance quality measures, and risk coding (using appropriate medical codes that capture the patient’s level of medical complexity). The Independent Physician Association affiliated with Family Practice Group provided this training at no cost to the practice. The 3 scribes worked full-time with the 5 part-time physicians in the study. Scribes were not required to have had a medical background prior to entering the program.

After the aforementioned training, scribes began working full-time with physicians during patient visits and continued learning on the job through feedback from supervising physicians. Scribes documented the patient encounters, recording medical and social histories and physical exam findings, and transcribing discussions of treatment plans and physicians’ instructions to patients.

Continue to: We reviewed patient EHRs...

 

We reviewed patient EHRs of 5 family physicians over 2 time periods: the 3 months prior to having a medical scribe and the 3 months after beginning to work with a medical scribe. Chart data extraction occurred from 4/11/13 to 8/28/14. We reviewed 1000 patient EHRs—100 EHRs each for the 5 participating physicians before and after scribe use. Selected EHRs ran chronologically from the start of each 3-month period. Reviewing EHRs at 3 months after the onset of the medical scribe program allowed time for the scribes to be fully integrated into the practice and confident in their job responsibilities. Chart review was performed by an office administrator who was blinded as to whether documentation had been done with or without a scribe present during the visit.

Eight quality measures were evaluated in chart review. These measures were drawn from the Healthcare Effectiveness Data and Information Set (HEDIS), a tool used to measure performance in medical care and service.

We surveyed 30 patients of each of the 5 providers, yielding a total of 150 survey responses. A medical assistant gave surveys to patients in the exam room following each office visit, to be completed anonymously and privately. Patients were told that surveys would take less than 2 minutes to complete. Office visits included episodic visits, physical exams, and chronic disease management.

Physicians believed they were saving, on average, 1.5 hours each day with the use of a scribe.

After the trial period, we surveyed participating physicians regarding medical scribe assistance with documentation. We also asked the physicians 3 open-ended questions regarding their experiences with their medical scribe.

This study was reviewed and approved (IRB Approval #11424) by the Tufts Health Science Campus Institutional Review Board.

Continue to: Data analysis

 

 

Data analysis

During chart review, we assessed the rate at which documentation was completed for 8 quality outcome measures commonly used in the primary care setting (TABLE 1), before and after the introduction of medical scribes. These quality measures and pertinent descriptors are listed in TABLE 2.¹³ Presence or absence of documentation on all quality measures was noted for all applicable patients.

One hundred fifty patients were surveyed immediately after their office visit on their perceptions of medical scribes, including their attitude toward, comfort with, and acceptance of medical scribes (TABLE 3). Five participating physicians were surveyed to assess their perceptions related to productivity and job satisfaction with the use of medical scribes (TABLE 4), and regarding time saved and additional patients seen. Those who collected and analyzed the data from the surveys were blinded to patient and physician identifiers.

Statistical analysis

Using chi-squared tests, we compared the number of positive documentations for the 8 outcome measures before and after the use of medical scribes. Two-sided P values < .05 were considered statistically significant. All statistical analyses were performed with the use of STATA version 9 (StataCorp LP. College Station, Tex).

Physician survey data were calculated on a Likert scale, with a score of 1 corresponding to “strongly disagree,” 2 “disagree,” 3 “neither agree nor disagree,” 4 “agree,” and 5 “strongly agree.” Using the 5 answers generated from the 5 physicians, we calculated the mean for each question.

RESULTS

The use of scribes demonstrated a statistically significant improvement in the documentation of 4 (out of 8) pay-for-performance measures (TABLE 1): fall risk assessment (odds ratio [OR] = 5.5, P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index (BMI) plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Sample sizes of each quality measure vary as there were differing numbers of applicable patients for each quality measure within the overall 1000 charts.

Continue to: We established at the beginning...

 

We established at the beginning of the study a target of obtaining surveys from 30 patients of each of the 5 physicians (total of 150). Response rates for surveys were 100% for both the 150 patients and the 5 physicians. No patients declined to complete the survey, although some did not answer every question.

Patients generally had positive experiences with medical scribes (TABLE 3). The majority of patients (96%, n = 144) felt comfortable with the scribe in the room during the visit with their provider. Patients felt that the provider focused on them “a little to a lot more” (75.8%, n = 113) and thought their visit was more efficient (73.6%, n = 109) as a result of the scribe being present vs not being present. Most patients were more satisfied with their office visit with the scribe being present (60.8%, n = 90).

Physicians felt that working with a medical scribe helped them connect with their patients, made patients feel that their physician was more attentive to them, contributed to better patient care, decreased the time they spent documenting in EHR, and contributed to faster work flow (TABLE 4). The physicians also believed they had saved a mean of 1.5 hours each day with the use of a medical scribe, and that they did not have to change their schedule in any way to accommodate additional patients as a result of having a scribe.

DISCUSSION

Documentation of fall risk assessment, follow-up tobacco screening, follow-up BMI plan, and follow-up blood pressure plan all demonstrated statistically significant increases with the use of medical scribes compared with practice before scribes. Follow-up depression screen and transition of care management had relatively high ORs (3.2 and 8, respectively), but did not yield statistically significant values, in part due to small sample sizes as the number of patients who were hospitalized and the number of patients who screened positive for depression were relatively small out of the total group of 1000 patients. The use of scribes had little effect on depression screen and tobacco screen. This is likely due to the fact that there were already effective office systems in place at the practice that alerted medical assistants to complete these screens for each appropriate patient.

We found that the use of medical scribes in a primary care setting was associated with both higher patient and physician satisfaction. Although the 5 physicians in this study chose not to see additional patients when using a medical scribe, they believed they were saving, on average, 1.5 hours of time each day with the use of a scribe. All 5 physicians reported that medical scribes enabled them to provide better patient care and to help patients feel as though they had more of the physician’s attention. Patient respondents attested to their provider focusing on them more during the visit. According to patient surveys, 40.4% of respondents felt that physicians addressed their concerns more thoroughly during the visit, while the remainder of patients did not.

Continue to: Some concerns...

 

Some concerns of introducing medical scribes into a health care system include possible patient discomfort with a third party being present during the visit and the cost of employing medical scribes. In this study, the vast majority of patients (96%) felt comfortable with a scribe in the room. Future research could compare patient discomfort due to the presence of a medical scribe with patient discomfort due to a physician using a computer during the visit.

Limitations of this study include the small sample size of both physicians and patients; a lack of validated measures for calculating productivity, time/efficiency, and overall satisfaction; and short time periods leading up to and following the introduction of medical scribes. In addition, EHRs of patients were chosen sequentially and not randomly, which could be a confounder. Participating physicians were aware of being studied; therefore, documentation could have been affected by the Hawthorne effect. The study also was limited to one family medicine site. Although improved documentation of primary care pay-for-performance quality measures was reported, wide confidence intervals and small patient numbers hindered generalizability of findings.

Documentation of 4 out of 8 pay-forperformance measures showed statistically significant improvement with the use of scribes.

Additional studies are needed with a robust analytic plan sufficient to demonstrate baseline provider familiarity with EHRs, accuracy of medical scribe documentation, and improved documentation of pay-for-performance quality measures. Additional investigation regarding the variable competency of different medical scribes could be useful in measuring the effects of the scribe on a variety of outcomes related to both the physician and patient.

It is possible that the improved documentation yielded by the use of medical scribes could generate billing codes that reimburse physicians at a higher level (eg, a higher ratio of 99214 to 99213), leading to increased pay. Future research could aim to quantify this source of increased revenue. Furthermore, investigations could aim to quantify the revenue that medical scribes generate via improved quality measure pay-for-performance documentation.

CORRESPONDENCE
Jessica Platt, MD, 195 Canal Street, Malden, MA 02148; [email protected].

This article is part of a series that illustrates strategies intended to redesign primary care education at the Veterans Health Administration (VHA), using interprofessional workplace learning. All have been implemented in the VA Centers of Excellence in Primary Care Education (CoEPCE). These models embody visionary transformation of clinical and educational environments that have potential for replication and dissemination throughout VA and other primary care clinical educational environments. For an introduction to the series see Klink K. Transforming primary care clinical learning environments to optimize education, outcomes, and satisfaction. Fed Pract. 2018;35(9):8-10.

Background

In 2011, 5 US Department of Veterans Affairs (VA) medical centers were selected by the VA Office of Academic Affiliations (OAA) to establish Centers of Excellence in Primary Care Education (CoEPCE). Part of the New Models of Care initiative, the 5 CoEPCEs use VA primary care settings to develop and test innovative approaches to prepare physician residents, medical students, advanced practice registered nurses, undergraduate nursing students, and other health professions’ trainees, such as social workers, pharmacists, psychologists, and physician assistants, for improved primary care practice. The CoEPCEs are interprofessional Academic PACTs (iAPACTs) with ≥ 2 professions of trainees engaged in learning on the PACT team.

The VA Puget Sound Seattle CoEPCE curriculum is embedded in a well-established academic VA primary care training site.¹ Trainees include doctor of nursing practice (DNP) students in adult, family, and psychiatric mental health nurse practitioner (NP) programs; NP residents; internal medicine physician residents; postgraduate pharmacy residents; and other health professions’ trainees. A Seattle CoEPCE priority is to provide DNP students, DNP residents, and physician residents with a longitudinal experience in team-based care as well as interprofessional education and collaborative practice (IPECP). Learners spend the majority of CoEPCE time in supervised, direct patient care, including primary care, women’s health, deployment health, homeless care, and home care. Formal IPECP activities comprise about 20% of time, supported by 3 educational strategies: (1) Panel management (PM)/quality improvement (QI); (2) Team building/ communications; and (3) Clinical content seminars to expand trainee clinical knowledge and skills and curriculum developed with the CoEPCE enterprise core domains in mind (Table).

Panel Management

Clinicians are increasingly being required to proactively optimize the health of an assigned population of patients in addition to assessing and managing the health of individual patients presenting for care. To address the objectives of increased accountability for population health outcomes and improved face-to-face care, Seattle CoEPCE developed curriculum for trainees to learn PM, a set of tools and processes that can be applied in the primary care setting.

PM clinical providers use data to proactively provide care to their patients between traditional clinic visits. The process is proactive in that gaps are identified whether or not an in-person visit occurs and involves an outreach mechanism to increase continuity of care, such as follow-up communications with the patients.² PM also has been associated with improvements in chronic disease care.^3-5

The Seattle CoEPCE developed an interprofessional team approach to PM that teaches trainees about the tools and resources used to close the gaps in care, including the use of clinical team members as health care systems subject matter experts. CoEPCE trainees are taught to analyze the care they provide to their panel of veterans (eg, identifying patients who have not refilled chronic medications or those who use the emergency department [ED] for nonacute conditions) and take action to improve care. PM yields rich discussions on systems resources and processes and is easily applied to a range of health conditions as well as delivery system issues. PM gives learners the tools they can use to close these gaps, such as the expertise of their peers, clinical team, and specialists.⁶

Planning and Implementation

In addition to completing a literature review to determine the state of PM practice and models, CoEPCE faculty polled recent graduates inquiring about strategies they did not learn prior to graduation. Based on their responses, CoEPCE faculty identified 2 skill deficits: management of chronic diseases and proficiency with data and statistics about performance improvement in panel patient care over time. Addressing these unmet needs became the impetus for developing curriculum for conducting PM. Planning and launching the CoEPCE approach to PM took about 3 months and involved CoEPCE faculty, a data manager, and administrative support. The learning objectives of Seattle’s PM initiative are to:

• Promote preventive health and chronic disease care by use performance data;
• Develop individual- and populationfocused action plans;
• Work collaboratively, strategically, and effectively with an interprofessional care team; and
• Learn how to effectively use system resources.

Curriculum

The PM curriculum is a longitudinal, experiential approach to learning how to manage chronic diseases between visits by using patient data. It is designed for trainees in a continuity clinic to review the care of their patients on a regular basis. Seattle CoEPCE medicine residents are assigned patient panels, which increase from 70 patients in the first year to about 140 patients by the end of the third year. DNP postgraduate trainees are assigned an initial panel of 50 patients that increases incrementally over the year-long residency.

CoEPCE faculty determined the focus of PM sessions to be diabetes mellitus (DM), hypertension, obesity, chronic opioid therapy, and low-acuity ED use. Because PM sessions are designed to allow participants to identify systems issues that may affect multiple patients, some of these topics have expanded into QI projects. PM sessions run 2 to 3 hours per session and are held 4 to 6 times a year. Each session is repeated twice to accommodate diverse trainee schedules. PM participants must have their patient visit time blocked for each session (Appendix).

Faculty Roles and Development

PM faculty involved in any individual session may include a combination of a CoEPCE clinical pharmacy specialist, a registered nurse (RN) care manager, a social worker, a NP, a physician, a clinical psychologist, and a medicine outpatient chief resident (PGY4, termed clinician-teacher fellow at Seattle VA medical center). The chief resident is a medicine residency graduate and takes on teaching responsibilities depending on the topic of the session. The CoEPCE clinical pharmacist role varies depending on the session topic: They may facilitate the session or provide recommendations for medication management for individual cases. The RN care manager often knows the patients and brings a unique perspective that complements that of the primary care providers and ideally participates in every session. The patients of multiple RN care managers may be presented at each session, and it was not feasible to include all RN care managers in every session. After case discussions, trainees often communicated with the RN care managers about the case, using instant messaging, and CoEPCE provides other avenues for patient care discussion through huddles involving the provider, RN care manager, clinical pharmacist, and other clinical professions.

Resources

The primary resource required to support PM is an information technology (IT) system that provides relevant health outcome and health care utilization data on patients assigned to trainees. PM sessions include teaching trainees how to access patient data. Since discussion about the care of panel patients during the learning sessions often results in real-time adjustments in the care plan, modest administrative support required post-PM sessions, such as clerical scheduling of the requested clinic or telephone follow-up with the physician, nurse, or pharmacist.

Monitoring and Assessment

Panel performance is evaluated at each educational session. To assess the CoEPCE PM curriculum, participants provide feedback in 8 questions over 3 domains: trainee perception of curriculum content, confidence in performing PM involving completion of a PM workshop, and likelihood of using PM techniques in the future. CoEPCE faculty use the feedback to improve their instruction of panel management skill and develop new sessions that target additional population groups. Evaluation of the curriculum also includes monitoring of panel patients’ chronic disease measures.

Several partnerships have contributed to the success and integrations of PM into facility activities. First, having the primary care clinic director as a member of the Co- EPCE faculty has encouraged faculty and staff to operationalize and implement PM broadly by distributing data monthly to all clinic staff. Second, high facility staff interest outside the CoEPCE and primary care clinic has facilitated establishing communications outside the CoEPCE regarding clinic data.

Challenges and Solutions

Trainees at earlier academic levels often desire more instruction in clinical knowledge, such as treatment options for DM or goals of therapy in hypertension. In contrast, advanced trainees are able to review patient data, brainstorm, and optimize solutions. Seattle CoEPCE balances these different learning needs via a flexible approach to the 3-hour sessions. For example, advanced trainees progress from structured short lectures to informal sessions, which train them to perform PM on their own. In addition, the flexible design integrates trainees with diverse schedules, particularly among DNP students and residents, pharmacy residents, and physician residents. Some of this work falls on the RN care management team and administrative support staff.

Competing Priorities

The demand for direct patient care points to the importance of indirect patient care activities like PM to demonstrate improved results. Managing chronic conditions and matching appropriate services and resources should improve clinical outcomes and efficiency longterm. In the interim, it is important to note that PM demonstrates the continuous aspect of clinical care, particularly for trainees who have strict guidelines defining clinical care for the experiences to count toward eligibility for licensure. Additionally, PM results in trainees who are making decisions with VA patients and are more efficiently providing and supporting patient care. Therefore, it is critical to secure important resources, such as provider time for conducting PM.

Data Access

No single data system in VA covers the broad range of topics covered in the PM sessions, and not all trainees have their own assigned panels. For example, health professions students are not assigned a panel of patients. While they do not have access to panel data such as those generated by Primary Care Almanac in VSSC (a data source in the VA Support Service Center database),the Seattle CoEPCE data manager pulls a set of patient data from the students’ paired faculty preceptors’ panels for review. Thus they learn PM principles and strategies for improving patient care via PM as part of the unique VA longitudinal clinic experience and the opportunity to learn from a multidisciplinary team that is not available at other clinical sites. Postgraduate NP residents in CoEPCE training have their own panels of patients and thus the ability to directly access their panel performance data.

Success Factors

A key success factor includes CoEPCE faculty’s ability to develop and operationalize a panel management model that simultaneously aligns with the educational goals of an interprofessional education training program and supports VA adoption of the medical home or patient aligned care teams (PACT). The CoEPCE contributes staff expertise in accessing and reporting patient data, accessing appropriate teaching space, managing panels of patients with chronic diseases, and facilitating a team-based approach to care. Additionally, the CoEPCE brand is helpful for getting buy-in from the clinical and academic stakeholders necessary for moving PM forward.

Colocating CoEPCE trainees and faculty in the primary care clinic promotes team identity around the RN care managers and facilitated communications with non-CoEPCE clinical teams that have trainees from other professions. RN care managers serve as the locus of highquality PM since they share patient panels with the trainees and already track admissions, ED visits, and numerous chronic health care metrics. RN care managers offer a level of insight into chronic disease that other providers may not possess, such as the specific details on medication adherence and the impact of adverse effects (AEs) for that particular patient. RN care managers are able to teach about their team role and responsibilities, strengthening the model.

PM is an opportunity to expand CoEPCE interprofessional education capacity by creating colocation of different trainee and faculty professions during the PM sessions; the sharing of data with trainees; and sharing and reflecting on data, strengthening communications between professions and within the PACT. The Seattle CoEPCE now has systems in place that allow the RN care manager to send notes to a physician and DNP resident, and the resident is expected to respond. In addition, the PM approach provides experience with analyzing data to improve care in an interprofessional team setting, which is a requirement of the Accreditation Council for Graduate Medical Education.

Interprofessional Collaboration

PM sessions are intentionally designed to improve communication among team members and foster a team approach to care. PM sessions provide an opportunity for trainees and clinician faculty to be together and learn about each profession’s perspectives. For example, early in the process physician and DNP trainees learn about the importance of clinical pharmacists to the team who prescribe and make medication adjustments within their scope of practice as well as the importance of making appropriate pharmacy referrals. Additionally, the RN care manager and clinical pharmacy specialists who serve as faculty in the CoEPCE provide pertinent information on individual patients, increasing integration with the PACT. Finally, there is anecdotal evidence that faculty also are learning more about interprofessional education and expanding their own skills.

Clinical Performance

CoEPCE trainees, non-CoEPCE physician residents, and CoEPCE faculty participants regularly receive patient data with which they can proactively develop or amend a treatment plan between visits. PM has resulted in improved data sharing with providers. Instead of once a year, providers and clinic staff now receive patient data monthly on chronic conditions from the clinic director. Trainees on ambulatory rotations are expected to review their panel data at least a half day per week. CoEPCE staff evaluate trainee likelihood to use PM and ability to identify patients who benefit from team-based care.

At the population level of chronic disease management, preliminary evidence demonstrates that primary care clinic patient panels are increasingly within target for DM and blood pressure measures, as assessed by periodic clinical reports to providers. Some of the PM topics have resulted in systems-level improvements, such as reducing unnecessary ED use for nonacute conditions and better opioid prescription monitoring. Moreover, PM supports everyone working at the top of his/her professional capability. For example, the RN care manager has the impetus to initiate DM education with a particular patient.

Since CoEPCE began teaching PM, the Seattle primary care clinic has committed to the regular access and review of data. This has encouraged the alignment of standards of care for chronic disease management so that all care providers are working toward the same benchmark goals.

Patient Outcomes

At the individual level, PM provide a mechanism to systemically review trainee panel patients with out-of-target clinical measures, and develop new care approaches involving interprofessional strategies and problem solving. PM also helps identify patients who have missed follow-up, reducing the risk that patients with chronic care needs will be lost to clinical engagement if they are not reminded or do not pursue appointments. The PM-trained PACT reaches out to patients who might not otherwise get care before the next clinic visit and provides new care plans. Second, patients have the benefit of a team that manages their health needs. For example, including the clinical pharmacists in the PM sessions ensures timely identification of medication interactions and the potential AEs. Additionally, PM contributes to the care coordination model by involving individuals on the primary care team who know the patient. These members review the patient’s data between visits and initiate team-based changes to the care plan to improve care. More team members connect with a patient, resulting in more intense care and quicker follow-up to determine the effectiveness of a treatment plan.

PM topics have spun off QI projects resulting in new clinic processes and programs, including processes for managing wounds in primary care and to assure timely post-ED visit follow-ups. Areas for expansion include a follow-up QI project to reduce nonacute ED visits by patients on the homeless PACT panel and interventions for better management of care for women veterans with mental health needs. PM also has extended to non-Co- EPCE teams and to other clinic activities, such as strengthening huddles of team members specifically related to panel data and addressing selected patient cases between visits. Pharmacy residents and faculty are more involved in reviewing the panel before patients are seen to review medication lists and identify duplications.

The Future

Under stage 2 of the program, the Seattle CoEPCE intends to lead in the creation of a PM toolkit as well as a data access guide that will allow VA facilities with limited data management expertise to access chronic disease metrics. Second, the CoEPCE will continue its dissemination efforts locally to other residents in the internal medicine residency program in all of its continuity clinics. Additionally, there is high interest by DNP training programs to expand and export longitudinal training experience PM curriculum to non-VA based students.

This article is part of a series that illustrates strategies intended to redesign primary care education at the Veterans Health Administration (VHA), using interprofessional workplace learning. All have been implemented in the VA Centers of Excellence in Primary Care Education (CoEPCE). These models embody visionary transformation of clinical and educational environments that have potential for replication and dissemination throughout VA and other primary care clinical educational environments. For an introduction to the series see Klink K. Transforming primary care clinical learning environments to optimize education, outcomes, and satisfaction. Fed Pract. 2018;35(9):8-10.

Background

In 2011, 5 US Department of Veterans Affairs (VA) medical centers were selected by the VA Office of Academic Affiliations (OAA) to establish Centers of Excellence in Primary Care Education (CoEPCE). Part of the New Models of Care initiative, the 5 CoEPCEs use VA primary care settings to develop and test innovative approaches to prepare physician residents, medical students, advanced practice registered nurses, undergraduate nursing students, and other health professions’ trainees, such as social workers, pharmacists, psychologists, and physician assistants, for improved primary care practice. The CoEPCEs are interprofessional Academic PACTs (iAPACTs) with ≥ 2 professions of trainees engaged in learning on the PACT team.

The VA Puget Sound Seattle CoEPCE curriculum is embedded in a well-established academic VA primary care training site.¹ Trainees include doctor of nursing practice (DNP) students in adult, family, and psychiatric mental health nurse practitioner (NP) programs; NP residents; internal medicine physician residents; postgraduate pharmacy residents; and other health professions’ trainees. A Seattle CoEPCE priority is to provide DNP students, DNP residents, and physician residents with a longitudinal experience in team-based care as well as interprofessional education and collaborative practice (IPECP). Learners spend the majority of CoEPCE time in supervised, direct patient care, including primary care, women’s health, deployment health, homeless care, and home care. Formal IPECP activities comprise about 20% of time, supported by 3 educational strategies: (1) Panel management (PM)/quality improvement (QI); (2) Team building/ communications; and (3) Clinical content seminars to expand trainee clinical knowledge and skills and curriculum developed with the CoEPCE enterprise core domains in mind (Table).

Panel Management

Clinicians are increasingly being required to proactively optimize the health of an assigned population of patients in addition to assessing and managing the health of individual patients presenting for care. To address the objectives of increased accountability for population health outcomes and improved face-to-face care, Seattle CoEPCE developed curriculum for trainees to learn PM, a set of tools and processes that can be applied in the primary care setting.

PM clinical providers use data to proactively provide care to their patients between traditional clinic visits. The process is proactive in that gaps are identified whether or not an in-person visit occurs and involves an outreach mechanism to increase continuity of care, such as follow-up communications with the patients.² PM also has been associated with improvements in chronic disease care.^3-5

The Seattle CoEPCE developed an interprofessional team approach to PM that teaches trainees about the tools and resources used to close the gaps in care, including the use of clinical team members as health care systems subject matter experts. CoEPCE trainees are taught to analyze the care they provide to their panel of veterans (eg, identifying patients who have not refilled chronic medications or those who use the emergency department [ED] for nonacute conditions) and take action to improve care. PM yields rich discussions on systems resources and processes and is easily applied to a range of health conditions as well as delivery system issues. PM gives learners the tools they can use to close these gaps, such as the expertise of their peers, clinical team, and specialists.⁶

Planning and Implementation

In addition to completing a literature review to determine the state of PM practice and models, CoEPCE faculty polled recent graduates inquiring about strategies they did not learn prior to graduation. Based on their responses, CoEPCE faculty identified 2 skill deficits: management of chronic diseases and proficiency with data and statistics about performance improvement in panel patient care over time. Addressing these unmet needs became the impetus for developing curriculum for conducting PM. Planning and launching the CoEPCE approach to PM took about 3 months and involved CoEPCE faculty, a data manager, and administrative support. The learning objectives of Seattle’s PM initiative are to:

• Promote preventive health and chronic disease care by use performance data;
• Develop individual- and populationfocused action plans;
• Work collaboratively, strategically, and effectively with an interprofessional care team; and
• Learn how to effectively use system resources.

Curriculum

The PM curriculum is a longitudinal, experiential approach to learning how to manage chronic diseases between visits by using patient data. It is designed for trainees in a continuity clinic to review the care of their patients on a regular basis. Seattle CoEPCE medicine residents are assigned patient panels, which increase from 70 patients in the first year to about 140 patients by the end of the third year. DNP postgraduate trainees are assigned an initial panel of 50 patients that increases incrementally over the year-long residency.

CoEPCE faculty determined the focus of PM sessions to be diabetes mellitus (DM), hypertension, obesity, chronic opioid therapy, and low-acuity ED use. Because PM sessions are designed to allow participants to identify systems issues that may affect multiple patients, some of these topics have expanded into QI projects. PM sessions run 2 to 3 hours per session and are held 4 to 6 times a year. Each session is repeated twice to accommodate diverse trainee schedules. PM participants must have their patient visit time blocked for each session (Appendix).

Faculty Roles and Development

PM faculty involved in any individual session may include a combination of a CoEPCE clinical pharmacy specialist, a registered nurse (RN) care manager, a social worker, a NP, a physician, a clinical psychologist, and a medicine outpatient chief resident (PGY4, termed clinician-teacher fellow at Seattle VA medical center). The chief resident is a medicine residency graduate and takes on teaching responsibilities depending on the topic of the session. The CoEPCE clinical pharmacist role varies depending on the session topic: They may facilitate the session or provide recommendations for medication management for individual cases. The RN care manager often knows the patients and brings a unique perspective that complements that of the primary care providers and ideally participates in every session. The patients of multiple RN care managers may be presented at each session, and it was not feasible to include all RN care managers in every session. After case discussions, trainees often communicated with the RN care managers about the case, using instant messaging, and CoEPCE provides other avenues for patient care discussion through huddles involving the provider, RN care manager, clinical pharmacist, and other clinical professions.

Resources

The primary resource required to support PM is an information technology (IT) system that provides relevant health outcome and health care utilization data on patients assigned to trainees. PM sessions include teaching trainees how to access patient data. Since discussion about the care of panel patients during the learning sessions often results in real-time adjustments in the care plan, modest administrative support required post-PM sessions, such as clerical scheduling of the requested clinic or telephone follow-up with the physician, nurse, or pharmacist.

Monitoring and Assessment

Panel performance is evaluated at each educational session. To assess the CoEPCE PM curriculum, participants provide feedback in 8 questions over 3 domains: trainee perception of curriculum content, confidence in performing PM involving completion of a PM workshop, and likelihood of using PM techniques in the future. CoEPCE faculty use the feedback to improve their instruction of panel management skill and develop new sessions that target additional population groups. Evaluation of the curriculum also includes monitoring of panel patients’ chronic disease measures.

Several partnerships have contributed to the success and integrations of PM into facility activities. First, having the primary care clinic director as a member of the Co- EPCE faculty has encouraged faculty and staff to operationalize and implement PM broadly by distributing data monthly to all clinic staff. Second, high facility staff interest outside the CoEPCE and primary care clinic has facilitated establishing communications outside the CoEPCE regarding clinic data.

Challenges and Solutions

Trainees at earlier academic levels often desire more instruction in clinical knowledge, such as treatment options for DM or goals of therapy in hypertension. In contrast, advanced trainees are able to review patient data, brainstorm, and optimize solutions. Seattle CoEPCE balances these different learning needs via a flexible approach to the 3-hour sessions. For example, advanced trainees progress from structured short lectures to informal sessions, which train them to perform PM on their own. In addition, the flexible design integrates trainees with diverse schedules, particularly among DNP students and residents, pharmacy residents, and physician residents. Some of this work falls on the RN care management team and administrative support staff.

Competing Priorities

The demand for direct patient care points to the importance of indirect patient care activities like PM to demonstrate improved results. Managing chronic conditions and matching appropriate services and resources should improve clinical outcomes and efficiency longterm. In the interim, it is important to note that PM demonstrates the continuous aspect of clinical care, particularly for trainees who have strict guidelines defining clinical care for the experiences to count toward eligibility for licensure. Additionally, PM results in trainees who are making decisions with VA patients and are more efficiently providing and supporting patient care. Therefore, it is critical to secure important resources, such as provider time for conducting PM.

Data Access

No single data system in VA covers the broad range of topics covered in the PM sessions, and not all trainees have their own assigned panels. For example, health professions students are not assigned a panel of patients. While they do not have access to panel data such as those generated by Primary Care Almanac in VSSC (a data source in the VA Support Service Center database),the Seattle CoEPCE data manager pulls a set of patient data from the students’ paired faculty preceptors’ panels for review. Thus they learn PM principles and strategies for improving patient care via PM as part of the unique VA longitudinal clinic experience and the opportunity to learn from a multidisciplinary team that is not available at other clinical sites. Postgraduate NP residents in CoEPCE training have their own panels of patients and thus the ability to directly access their panel performance data.

Success Factors

A key success factor includes CoEPCE faculty’s ability to develop and operationalize a panel management model that simultaneously aligns with the educational goals of an interprofessional education training program and supports VA adoption of the medical home or patient aligned care teams (PACT). The CoEPCE contributes staff expertise in accessing and reporting patient data, accessing appropriate teaching space, managing panels of patients with chronic diseases, and facilitating a team-based approach to care. Additionally, the CoEPCE brand is helpful for getting buy-in from the clinical and academic stakeholders necessary for moving PM forward.

Colocating CoEPCE trainees and faculty in the primary care clinic promotes team identity around the RN care managers and facilitated communications with non-CoEPCE clinical teams that have trainees from other professions. RN care managers serve as the locus of highquality PM since they share patient panels with the trainees and already track admissions, ED visits, and numerous chronic health care metrics. RN care managers offer a level of insight into chronic disease that other providers may not possess, such as the specific details on medication adherence and the impact of adverse effects (AEs) for that particular patient. RN care managers are able to teach about their team role and responsibilities, strengthening the model.

PM is an opportunity to expand CoEPCE interprofessional education capacity by creating colocation of different trainee and faculty professions during the PM sessions; the sharing of data with trainees; and sharing and reflecting on data, strengthening communications between professions and within the PACT. The Seattle CoEPCE now has systems in place that allow the RN care manager to send notes to a physician and DNP resident, and the resident is expected to respond. In addition, the PM approach provides experience with analyzing data to improve care in an interprofessional team setting, which is a requirement of the Accreditation Council for Graduate Medical Education.

Interprofessional Collaboration

PM sessions are intentionally designed to improve communication among team members and foster a team approach to care. PM sessions provide an opportunity for trainees and clinician faculty to be together and learn about each profession’s perspectives. For example, early in the process physician and DNP trainees learn about the importance of clinical pharmacists to the team who prescribe and make medication adjustments within their scope of practice as well as the importance of making appropriate pharmacy referrals. Additionally, the RN care manager and clinical pharmacy specialists who serve as faculty in the CoEPCE provide pertinent information on individual patients, increasing integration with the PACT. Finally, there is anecdotal evidence that faculty also are learning more about interprofessional education and expanding their own skills.

Clinical Performance

CoEPCE trainees, non-CoEPCE physician residents, and CoEPCE faculty participants regularly receive patient data with which they can proactively develop or amend a treatment plan between visits. PM has resulted in improved data sharing with providers. Instead of once a year, providers and clinic staff now receive patient data monthly on chronic conditions from the clinic director. Trainees on ambulatory rotations are expected to review their panel data at least a half day per week. CoEPCE staff evaluate trainee likelihood to use PM and ability to identify patients who benefit from team-based care.

At the population level of chronic disease management, preliminary evidence demonstrates that primary care clinic patient panels are increasingly within target for DM and blood pressure measures, as assessed by periodic clinical reports to providers. Some of the PM topics have resulted in systems-level improvements, such as reducing unnecessary ED use for nonacute conditions and better opioid prescription monitoring. Moreover, PM supports everyone working at the top of his/her professional capability. For example, the RN care manager has the impetus to initiate DM education with a particular patient.

Since CoEPCE began teaching PM, the Seattle primary care clinic has committed to the regular access and review of data. This has encouraged the alignment of standards of care for chronic disease management so that all care providers are working toward the same benchmark goals.

Patient Outcomes

At the individual level, PM provide a mechanism to systemically review trainee panel patients with out-of-target clinical measures, and develop new care approaches involving interprofessional strategies and problem solving. PM also helps identify patients who have missed follow-up, reducing the risk that patients with chronic care needs will be lost to clinical engagement if they are not reminded or do not pursue appointments. The PM-trained PACT reaches out to patients who might not otherwise get care before the next clinic visit and provides new care plans. Second, patients have the benefit of a team that manages their health needs. For example, including the clinical pharmacists in the PM sessions ensures timely identification of medication interactions and the potential AEs. Additionally, PM contributes to the care coordination model by involving individuals on the primary care team who know the patient. These members review the patient’s data between visits and initiate team-based changes to the care plan to improve care. More team members connect with a patient, resulting in more intense care and quicker follow-up to determine the effectiveness of a treatment plan.

PM topics have spun off QI projects resulting in new clinic processes and programs, including processes for managing wounds in primary care and to assure timely post-ED visit follow-ups. Areas for expansion include a follow-up QI project to reduce nonacute ED visits by patients on the homeless PACT panel and interventions for better management of care for women veterans with mental health needs. PM also has extended to non-Co- EPCE teams and to other clinic activities, such as strengthening huddles of team members specifically related to panel data and addressing selected patient cases between visits. Pharmacy residents and faculty are more involved in reviewing the panel before patients are seen to review medication lists and identify duplications.

The Future

Under stage 2 of the program, the Seattle CoEPCE intends to lead in the creation of a PM toolkit as well as a data access guide that will allow VA facilities with limited data management expertise to access chronic disease metrics. Second, the CoEPCE will continue its dissemination efforts locally to other residents in the internal medicine residency program in all of its continuity clinics. Additionally, there is high interest by DNP training programs to expand and export longitudinal training experience PM curriculum to non-VA based students.

This article is part of a series that illustrates strategies intended to redesign primary care education at the Veterans Health Administration (VHA), using interprofessional workplace learning. All have been implemented in the VA Centers of Excellence in Primary Care Education (CoEPCE). These models embody visionary transformation of clinical and educational environments that have potential for replication and dissemination throughout VA and other primary care clinical educational environments. For an introduction to the series see Klink K. Transforming primary care clinical learning environments to optimize education, outcomes, and satisfaction. Fed Pract. 2018;35(9):8-10.

Background

In 2011, 5 US Department of Veterans Affairs (VA) medical centers were selected by the VA Office of Academic Affiliations (OAA) to establish Centers of Excellence in Primary Care Education (CoEPCE). Part of the New Models of Care initiative, the 5 CoEPCEs use VA primary care settings to develop and test innovative approaches to prepare physician residents, medical students, advanced practice registered nurses, undergraduate nursing students, and other health professions’ trainees, such as social workers, pharmacists, psychologists, and physician assistants, for improved primary care practice. The CoEPCEs are interprofessional Academic PACTs (iAPACTs) with ≥ 2 professions of trainees engaged in learning on the PACT team.

The VA Puget Sound Seattle CoEPCE curriculum is embedded in a well-established academic VA primary care training site.¹ Trainees include doctor of nursing practice (DNP) students in adult, family, and psychiatric mental health nurse practitioner (NP) programs; NP residents; internal medicine physician residents; postgraduate pharmacy residents; and other health professions’ trainees. A Seattle CoEPCE priority is to provide DNP students, DNP residents, and physician residents with a longitudinal experience in team-based care as well as interprofessional education and collaborative practice (IPECP). Learners spend the majority of CoEPCE time in supervised, direct patient care, including primary care, women’s health, deployment health, homeless care, and home care. Formal IPECP activities comprise about 20% of time, supported by 3 educational strategies: (1) Panel management (PM)/quality improvement (QI); (2) Team building/ communications; and (3) Clinical content seminars to expand trainee clinical knowledge and skills and curriculum developed with the CoEPCE enterprise core domains in mind (Table).

Panel Management

Clinicians are increasingly being required to proactively optimize the health of an assigned population of patients in addition to assessing and managing the health of individual patients presenting for care. To address the objectives of increased accountability for population health outcomes and improved face-to-face care, Seattle CoEPCE developed curriculum for trainees to learn PM, a set of tools and processes that can be applied in the primary care setting.

PM clinical providers use data to proactively provide care to their patients between traditional clinic visits. The process is proactive in that gaps are identified whether or not an in-person visit occurs and involves an outreach mechanism to increase continuity of care, such as follow-up communications with the patients.² PM also has been associated with improvements in chronic disease care.^3-5

The Seattle CoEPCE developed an interprofessional team approach to PM that teaches trainees about the tools and resources used to close the gaps in care, including the use of clinical team members as health care systems subject matter experts. CoEPCE trainees are taught to analyze the care they provide to their panel of veterans (eg, identifying patients who have not refilled chronic medications or those who use the emergency department [ED] for nonacute conditions) and take action to improve care. PM yields rich discussions on systems resources and processes and is easily applied to a range of health conditions as well as delivery system issues. PM gives learners the tools they can use to close these gaps, such as the expertise of their peers, clinical team, and specialists.⁶

Planning and Implementation

In addition to completing a literature review to determine the state of PM practice and models, CoEPCE faculty polled recent graduates inquiring about strategies they did not learn prior to graduation. Based on their responses, CoEPCE faculty identified 2 skill deficits: management of chronic diseases and proficiency with data and statistics about performance improvement in panel patient care over time. Addressing these unmet needs became the impetus for developing curriculum for conducting PM. Planning and launching the CoEPCE approach to PM took about 3 months and involved CoEPCE faculty, a data manager, and administrative support. The learning objectives of Seattle’s PM initiative are to:

• Promote preventive health and chronic disease care by use performance data;
• Develop individual- and populationfocused action plans;
• Work collaboratively, strategically, and effectively with an interprofessional care team; and
• Learn how to effectively use system resources.

Curriculum

The PM curriculum is a longitudinal, experiential approach to learning how to manage chronic diseases between visits by using patient data. It is designed for trainees in a continuity clinic to review the care of their patients on a regular basis. Seattle CoEPCE medicine residents are assigned patient panels, which increase from 70 patients in the first year to about 140 patients by the end of the third year. DNP postgraduate trainees are assigned an initial panel of 50 patients that increases incrementally over the year-long residency.

CoEPCE faculty determined the focus of PM sessions to be diabetes mellitus (DM), hypertension, obesity, chronic opioid therapy, and low-acuity ED use. Because PM sessions are designed to allow participants to identify systems issues that may affect multiple patients, some of these topics have expanded into QI projects. PM sessions run 2 to 3 hours per session and are held 4 to 6 times a year. Each session is repeated twice to accommodate diverse trainee schedules. PM participants must have their patient visit time blocked for each session (Appendix).

Faculty Roles and Development

PM faculty involved in any individual session may include a combination of a CoEPCE clinical pharmacy specialist, a registered nurse (RN) care manager, a social worker, a NP, a physician, a clinical psychologist, and a medicine outpatient chief resident (PGY4, termed clinician-teacher fellow at Seattle VA medical center). The chief resident is a medicine residency graduate and takes on teaching responsibilities depending on the topic of the session. The CoEPCE clinical pharmacist role varies depending on the session topic: They may facilitate the session or provide recommendations for medication management for individual cases. The RN care manager often knows the patients and brings a unique perspective that complements that of the primary care providers and ideally participates in every session. The patients of multiple RN care managers may be presented at each session, and it was not feasible to include all RN care managers in every session. After case discussions, trainees often communicated with the RN care managers about the case, using instant messaging, and CoEPCE provides other avenues for patient care discussion through huddles involving the provider, RN care manager, clinical pharmacist, and other clinical professions.

Resources

The primary resource required to support PM is an information technology (IT) system that provides relevant health outcome and health care utilization data on patients assigned to trainees. PM sessions include teaching trainees how to access patient data. Since discussion about the care of panel patients during the learning sessions often results in real-time adjustments in the care plan, modest administrative support required post-PM sessions, such as clerical scheduling of the requested clinic or telephone follow-up with the physician, nurse, or pharmacist.

Monitoring and Assessment

Panel performance is evaluated at each educational session. To assess the CoEPCE PM curriculum, participants provide feedback in 8 questions over 3 domains: trainee perception of curriculum content, confidence in performing PM involving completion of a PM workshop, and likelihood of using PM techniques in the future. CoEPCE faculty use the feedback to improve their instruction of panel management skill and develop new sessions that target additional population groups. Evaluation of the curriculum also includes monitoring of panel patients’ chronic disease measures.

Several partnerships have contributed to the success and integrations of PM into facility activities. First, having the primary care clinic director as a member of the Co- EPCE faculty has encouraged faculty and staff to operationalize and implement PM broadly by distributing data monthly to all clinic staff. Second, high facility staff interest outside the CoEPCE and primary care clinic has facilitated establishing communications outside the CoEPCE regarding clinic data.

Challenges and Solutions

Trainees at earlier academic levels often desire more instruction in clinical knowledge, such as treatment options for DM or goals of therapy in hypertension. In contrast, advanced trainees are able to review patient data, brainstorm, and optimize solutions. Seattle CoEPCE balances these different learning needs via a flexible approach to the 3-hour sessions. For example, advanced trainees progress from structured short lectures to informal sessions, which train them to perform PM on their own. In addition, the flexible design integrates trainees with diverse schedules, particularly among DNP students and residents, pharmacy residents, and physician residents. Some of this work falls on the RN care management team and administrative support staff.

Competing Priorities

The demand for direct patient care points to the importance of indirect patient care activities like PM to demonstrate improved results. Managing chronic conditions and matching appropriate services and resources should improve clinical outcomes and efficiency longterm. In the interim, it is important to note that PM demonstrates the continuous aspect of clinical care, particularly for trainees who have strict guidelines defining clinical care for the experiences to count toward eligibility for licensure. Additionally, PM results in trainees who are making decisions with VA patients and are more efficiently providing and supporting patient care. Therefore, it is critical to secure important resources, such as provider time for conducting PM.

Data Access

No single data system in VA covers the broad range of topics covered in the PM sessions, and not all trainees have their own assigned panels. For example, health professions students are not assigned a panel of patients. While they do not have access to panel data such as those generated by Primary Care Almanac in VSSC (a data source in the VA Support Service Center database),the Seattle CoEPCE data manager pulls a set of patient data from the students’ paired faculty preceptors’ panels for review. Thus they learn PM principles and strategies for improving patient care via PM as part of the unique VA longitudinal clinic experience and the opportunity to learn from a multidisciplinary team that is not available at other clinical sites. Postgraduate NP residents in CoEPCE training have their own panels of patients and thus the ability to directly access their panel performance data.

Success Factors

A key success factor includes CoEPCE faculty’s ability to develop and operationalize a panel management model that simultaneously aligns with the educational goals of an interprofessional education training program and supports VA adoption of the medical home or patient aligned care teams (PACT). The CoEPCE contributes staff expertise in accessing and reporting patient data, accessing appropriate teaching space, managing panels of patients with chronic diseases, and facilitating a team-based approach to care. Additionally, the CoEPCE brand is helpful for getting buy-in from the clinical and academic stakeholders necessary for moving PM forward.

Colocating CoEPCE trainees and faculty in the primary care clinic promotes team identity around the RN care managers and facilitated communications with non-CoEPCE clinical teams that have trainees from other professions. RN care managers serve as the locus of highquality PM since they share patient panels with the trainees and already track admissions, ED visits, and numerous chronic health care metrics. RN care managers offer a level of insight into chronic disease that other providers may not possess, such as the specific details on medication adherence and the impact of adverse effects (AEs) for that particular patient. RN care managers are able to teach about their team role and responsibilities, strengthening the model.

PM is an opportunity to expand CoEPCE interprofessional education capacity by creating colocation of different trainee and faculty professions during the PM sessions; the sharing of data with trainees; and sharing and reflecting on data, strengthening communications between professions and within the PACT. The Seattle CoEPCE now has systems in place that allow the RN care manager to send notes to a physician and DNP resident, and the resident is expected to respond. In addition, the PM approach provides experience with analyzing data to improve care in an interprofessional team setting, which is a requirement of the Accreditation Council for Graduate Medical Education.

Interprofessional Collaboration

PM sessions are intentionally designed to improve communication among team members and foster a team approach to care. PM sessions provide an opportunity for trainees and clinician faculty to be together and learn about each profession’s perspectives. For example, early in the process physician and DNP trainees learn about the importance of clinical pharmacists to the team who prescribe and make medication adjustments within their scope of practice as well as the importance of making appropriate pharmacy referrals. Additionally, the RN care manager and clinical pharmacy specialists who serve as faculty in the CoEPCE provide pertinent information on individual patients, increasing integration with the PACT. Finally, there is anecdotal evidence that faculty also are learning more about interprofessional education and expanding their own skills.

Clinical Performance

CoEPCE trainees, non-CoEPCE physician residents, and CoEPCE faculty participants regularly receive patient data with which they can proactively develop or amend a treatment plan between visits. PM has resulted in improved data sharing with providers. Instead of once a year, providers and clinic staff now receive patient data monthly on chronic conditions from the clinic director. Trainees on ambulatory rotations are expected to review their panel data at least a half day per week. CoEPCE staff evaluate trainee likelihood to use PM and ability to identify patients who benefit from team-based care.

At the population level of chronic disease management, preliminary evidence demonstrates that primary care clinic patient panels are increasingly within target for DM and blood pressure measures, as assessed by periodic clinical reports to providers. Some of the PM topics have resulted in systems-level improvements, such as reducing unnecessary ED use for nonacute conditions and better opioid prescription monitoring. Moreover, PM supports everyone working at the top of his/her professional capability. For example, the RN care manager has the impetus to initiate DM education with a particular patient.

Since CoEPCE began teaching PM, the Seattle primary care clinic has committed to the regular access and review of data. This has encouraged the alignment of standards of care for chronic disease management so that all care providers are working toward the same benchmark goals.

Patient Outcomes

At the individual level, PM provide a mechanism to systemically review trainee panel patients with out-of-target clinical measures, and develop new care approaches involving interprofessional strategies and problem solving. PM also helps identify patients who have missed follow-up, reducing the risk that patients with chronic care needs will be lost to clinical engagement if they are not reminded or do not pursue appointments. The PM-trained PACT reaches out to patients who might not otherwise get care before the next clinic visit and provides new care plans. Second, patients have the benefit of a team that manages their health needs. For example, including the clinical pharmacists in the PM sessions ensures timely identification of medication interactions and the potential AEs. Additionally, PM contributes to the care coordination model by involving individuals on the primary care team who know the patient. These members review the patient’s data between visits and initiate team-based changes to the care plan to improve care. More team members connect with a patient, resulting in more intense care and quicker follow-up to determine the effectiveness of a treatment plan.

PM topics have spun off QI projects resulting in new clinic processes and programs, including processes for managing wounds in primary care and to assure timely post-ED visit follow-ups. Areas for expansion include a follow-up QI project to reduce nonacute ED visits by patients on the homeless PACT panel and interventions for better management of care for women veterans with mental health needs. PM also has extended to non-Co- EPCE teams and to other clinic activities, such as strengthening huddles of team members specifically related to panel data and addressing selected patient cases between visits. Pharmacy residents and faculty are more involved in reviewing the panel before patients are seen to review medication lists and identify duplications.

The Future

Under stage 2 of the program, the Seattle CoEPCE intends to lead in the creation of a PM toolkit as well as a data access guide that will allow VA facilities with limited data management expertise to access chronic disease metrics. Second, the CoEPCE will continue its dissemination efforts locally to other residents in the internal medicine residency program in all of its continuity clinics. Additionally, there is high interest by DNP training programs to expand and export longitudinal training experience PM curriculum to non-VA based students.

From Tufts Medical Center, Boston, MA.

Abstract

• Objective: Audits at our academic medical center revealed near 100% compliance with protocols for perioperative venous thromboembolism (VTE) prophylaxis, but recent National Surgical Quality Improvement Program data demonstrated a higher than expected incidence of VTE (observed/expected = 1.32). The objective of this study was to identify potential causes of this discrepancy.
• Design: Retrospective case-control study.
• Setting: Urban academic medical center with high case-mix indices (Medicare approximately 2.4, non-Medicare approximately 2.0).
• Participants: 102 surgical inpatients with VTE (September 2012 to October 2015) matched with controls for age, gender, and type of procedure.
• Measurements: Prevalence of common VTE risk factors, length of stay, number of procedures, index operation times, and postoperative bed rest > 12 hours were assessed. Utilization of and compliance with our VTE risk assessment tool was also investigated.
• Results: Cases underwent more procedures and had longer lengths of stay and index procedures than controls. In addition, cases were more likely to have had > 12 hours of postoperative bed rest and central venous access than controls. Cases had more infections and were more likely to have severe lung disease, thrombophilia, and a history of prior VTE than controls. No differences in body mass index, tobacco use, current or previous malignancy, or VTE risk assessment form use were observed. Overall, care complexity and risk factors were equally important in determining VTE incidence. Our analyses also revealed lack of strict adherence to our VTE risk stratification protocol and frequent use of suboptimal prophylactic regimens.
• Conclusion: Well-accepted risk factors and overall care complexity determine VTE risk. Preventing VTE in high-risk patients requires assiduous attention to detail in VTE risk assessment and in delivery of optimal prophylaxis. Patients at especially high risk may require customized prophylactic regimens.

Keywords: hospital-acquired venous thromboembolic disease; VTE prophylaxis, surgical patients.

Deep vein thrombosis (DVT) and pulmonary embolism (PE) are well-recognized causes of morbidity and mortality in surgical patients. Between 350,000 and 600,000 cases of venous thromboembolism (VTE) occur each year in the United States, and it is responsible for approximately 10% of preventable in-hospital fatalities.^1-3 Given VTE’s impact on patients and the healthcare system and the fact that it is preventable, intense effort has been focused on developing more effective prophylactic measures to decrease its incidence.^2-4 In 2008, the surgeon general issued a “call to action” for increased efforts to prevent VTE.⁵

The American College of Chest Physicians (ACCP) guidelines subcategorize patients based on type of surgery. In addition, the ACCP guidelines support the use of a Caprini-based scoring system to aid in risk stratification and improve clinical decision-making (Table 1).^4,6-9 In general, scores ≥ 5 qualify individuals as high risk. Based on their risk category, patients receive mechanical prophylaxis, chemical prophylaxis, or a combination of the 2. Lower-risk patients who are ambulatory typically receive only mechanical prophylaxis while in bed, whereas higher-risk patients receive a combination of mechanical prophylaxis and chemoprophylaxis measures.⁷ In general, low-molecular-weight heparin (40 mg daily) and low-dose unfractionated heparin (5000 units 3 times daily) have been the standard evidence-based options for chemoprophylaxis in surgical patients. Absolute contraindications for prophylaxis include active bleeding and known increased risk of bleeding based on patient- or procedure-specific factors.

Our hospital, a 350-bed academic medical center in downtown Boston, MA, serving a diverse population with a very high case-mix index (2.4 Medicare and 2.0 non-Medicare), has strict protocols for VTE prophylaxis consistent with the ACCP guidelines and based on the Surgical Care Improvement Project (SCIP) measures published in 2006.¹⁰ The SCIP mandates allow for considerable surgeon discretion in the use of chemoprophylaxis for neurosurgical cases and general and orthopedic surgery cases deemed to be at high risk for bleeding. In addition, SCIP requires only that prophylaxis be initiated within 24 hours of surgical end time. Although recent audits revealed nearly 100% compliance with SCIP-mandated protocols, National Surgical Quality Improvement Program (NSQIP) data showed that the incidence of VTE events at our institution was higher than expected (observed/expected [O/E] = 1.32).

In order to determine the reasons for this mismatch between process and outcome performance, we investigated whether there were characteristics of our patient population that contributed to the higher than expected rates of VTE, and we scrutinized our VTE prophylaxis protocol to determine if there were aspects of our process that were also contributory.

Methods

Study Sample

This is a retrospective case-control study of surgical inpatients at our hospital during the period September 2012 to October 2015. Cases were identified as patients diagnosed with a VTE (DVT or PE). Controls were identified from a pool of surgical patients whose courses were not complicated by VTE during the same time frame as the cases and who were matched as closely as possible by procedure code, age, and gender.

 

 

Variables

Patient and hospital course variables that were analyzed included demographics, comorbidities, length of stay, number of procedures, index operation times, duration of postoperative bed rest, use of mechanical prophylaxis, and type of chemoprophylaxis and time frame within which it was initiated. Data were collected via chart review using International Classification of Diseases-9 and -10 codes to identify surgical cases within the allotted time period who were diagnosed with VTE. Demographic variables included age, sex, and ethnicity. Comorbidities included hypertension, diabetes, coronary artery disease, serious lung disease, previous or current malignancy, documented hypercoagulable state, and previous history of VTE. Body mass index (BMI) was also recorded. The aforementioned disease-specific variables were not matched between the case and control groups, as this data was obtained retrospectively during data collection.

Analysis

Associations between case and matched control were analyzed using the paired t-test for continuous variables and McNemar’s test for categorical variables. P values < 0.05 were considered statistically significant. SAS Enterprise Guide 7.15 (Cary, NC) was used for all statistical analyses.

The requirement for informed consent was waived by our Institutional Review Board, as the study was initially deemed to be a quality improvement project, and all data used for this report were de-identified.

Results

Our retrospective case-control analysis included a sample of 102 surgical patients whose courses were complicated by VTE between September 2012 and October 2015. The cases were distributed among 6 different surgical categories (Figure 1): trauma (20%), cancer (10%), cardiovascular (21%), noncancer neurosurgery (28%), elective orthopedics (11%), and miscellaneous general surgery (10%).

Comparisons between cases and controls in terms of patient demographics and risk factors are shown in Table 2. No statistically significant difference was observed in ethnicity or race between the 2 groups. Overall, cases had more hip/pelvis/leg fractures at presentation (P = 0.0008). The case group also had higher proportions of patients with postoperative bed rest greater than 12 hours (P = 0.009), central venous access (P < 0.0001), infection (P < 0.0001), and lower extremity edema documented during the hospitalization prior to development of DVT (P < 0.0001). Additionally, cases had significantly greater rates of previous VTE (P = 0.0004), inherited or acquired thrombophilia (P = 0.03), history of stroke (P = 0.0003), and severe lung disease, including pneumonia (P = 0.0008). No significant differences were noted between cases and matched controls in BMI (P = 0.43), current tobacco use (P = 0.71), current malignancy (P = 0.80), previous malignancy (P = 0.83), head trauma (P = 0.17), or acute cardiac disease (myocardial infarction or congestive heart failure; P = 0.12).

Variables felt to indicate overall complexity of hospital course for cases as compared to controls are outlined in Table 3. Cases were found to have significantly longer lengths of stay (median, 15.5 days versus 3 days, P < 0.0001). To account for the possibility that the development of VTE contributed to the increased length of stay in the cases, we also looked at the duration between admission date and the date of VTE diagnosis and determined that cases still had a longer length of stay when this was accounted for (median, 7 days versus 3 days, P < 0.0001). A much higher proportion of cases underwent more than 1 procedure compared to controls (P < 0.0001), and cases had significantly longer index operations as compared to controls (P = 0.002).

 

Seventeen cases received heparin on induction during their index procedure, compared to 23 controls (P = 0.24). Additionally, 63 cases began a prophylaxis regimen within 24 hours of surgery end time, compared to 68 controls (P = 0.24). The chemoprophylactic regimens utilized in cases and in controls are summarized in Figure 2. Of note, only 26 cases and 32 controls received standard prophylactic regimens with no missed doses (heparin 5000 units 3 times daily or enoxaparin 40 mg daily). Additionally, in over half of cases and a third of controls, nonstandard regimens were ordered. Examples of nonstandard regimens included nonstandard heparin or enoxaparin doses, low-dose warfarin, or aspirin alone. In most cases, nonstandard regimens were justified on the basis of high risk for bleeding.

Mechanical prophylaxis with pneumatic sequential compression devices (SCDs) was ordered in 93 (91%) cases and 87 (85%) controls; however, we were unable to accurately document uniform compliance in the use of these devices.

With regard to evaluation of our process measures, we found only 17% of cases and controls combined actually had a VTE risk assessment in their chart, and when it was present, it was often incomplete or was completed inaccurately.

Discussion

The goal of this study was to identify factors (patient characteristics and/or processes of care) that may be contributing to the higher than expected incidence of VTE events at our medical center, despite internal audits suggesting near perfect compliance with SCIP-mandated protocols. We found that in addition to usual risk factors for VTE, an overarching theme of our case cohort was their high complexity of illness. At baseline, these patients had significantly greater rates of stroke, thrombophilia, severe lung disease, infection, and history of VTE than controls. Moreover, the hospital courses of cases were significantly more complex than those of controls, as these patients had more procedures, longer lengths of stay and longer index operations, higher rates of postoperative bed rest exceeding 12 hours, and more prevalent central venous access than controls (Table 2). Several of these risk factors have been found to contribute to VTE development despite compliance with prophylaxis protocols.

Cassidy et al reviewed a cohort of nontrauma general surgery patients who developed VTE despite receiving appropriate prophylaxis and found that both multiple operations and emergency procedures contributed to the failure of VTE prophylaxis.¹¹ Similarly, Wang et al identified several independent risk factors for VTE despite thromboprophylaxis, including central venous access and infection, as well as intensive care unit admission, hospitalization for cranial surgery, and admission from a long-term care facility.¹² While our study did not capture some of these additional factors considered by Wang et al, the presence of risk factors not captured in traditional assessment tools suggests that additional consideration for complex patients is warranted.

 

In addition to these nonmodifiable patient characteristics, aspects of our VTE prophylaxis processes likely contributed to the higher than expected rate of VTE. While the electronic medical record at our institution does contain a VTE risk assessment tool based on the Caprini score, we found it often is not used at all or is used incorrectly/incompletely, which likely reflects the fact that physicians are neither prompted nor required to complete the assessment prior to prescribing VTE prophylaxis.

There is a significant body of evidence demonstrating that mandatory computerized VTE risk assessments can effectively reduce VTE rates and that improved outcomes occur shortly after implementation. Cassidy et al demonstrated the benefits of instituting a hospital-wide, mandatory, Caprini-based computerized VTE risk assessment that provides prophylaxis/early ambulation recommendations. Two years after implementing this system, they observed an 84% reduction in DVTs (P < 0.001) and a 55% reduction in PEs (P < 0.001).¹³ Nimeri et al had similarly impressive success, achieving a reduction in their NSQIP O/E for PE/DVT in general surgery from 6.00 in 2010 to 0.82 (for DVTs) and 0.78 (for PEs) 5 years after implementation of mandatory VTE risk assessment (though they noted that the most dramatic reduction occurred 1 year after implementation).¹⁴ Additionally, a recent systematic review and meta-analysis by Borab et al found computerized VTE risk assessments to be associated with a significant decrease in VTE events.¹⁵

The risk assessment tool used at our institution is qualitative in nature, and current literature suggests that employing a more quantitative tool may yield improved outcomes. Numerous studies have highlighted the importance of identifying patients at very high risk for VTE, as higher risk may necessitate more careful consideration of their prophylactic regimens. Obi et al found patients with Caprini scores higher than 8 to be at significantly greater risk of developing VTE compared to patients with scores of 7 or 8. Also, patients with scores of 7 or 8 were significantly more likely to have a VTE compared to those with scores of 5 or 6.¹⁶ In another study, Lobastov et al identified Caprini scores of 11 or higher as representing an extremely high-risk category for which standard prophylaxis regimens may not be effective.¹⁷ Thus, while having mandatory risk assessment has been shown to dramatically decrease VTE incidence, it is important to consider the magnitude of the numerical risk score. This is of particular importance at medical centers with high case-mix indices where patients at the highest risk might need to be managed with different prophylactic guidelines.

Another notable aspect of the process at our hospital was the great variation in the types of prophylactic regimens ordered, and the adherence to what was ordered. Only 25.5% of patients were maintained on a standard prophylactic regimen with no missed doses (heparin 5000 every 8 hours or enoxaparin 40 mg daily). Thus, the vast majority of the patients who went on to develop VTE either were prescribed a nontraditional prophylaxis regimen or missed doses of standard agents. The need for secondary surgical procedures or other invasive interventions may explain many, but not all, of the missed doses.

The timing of prophylaxis initiation for our patients was also found to deviate from accepted standards. Only 16.8% of cases received prophylaxis upon induction of anesthesia, and furthermore, 38% of cases did not receive any anticoagulation within 24 hours of their index operation. While this variability in prophylaxis implementation was acceptable within the SCIP guidelines based on “high risk for bleeding” or other considerations, it likely contributed to our suboptimal outcomes. The variations and interruptions in prophylactic regimens speak to barriers that have previously been reported as contributing factors to noncompliance with VTE prophylaxis.¹⁸

 

Given these known barriers and the observed underutilization and improper use of our risk assessment tool, we have recently changed our surgical admission order sets such that a mandatory quantitative risk assessment must be done for every surgical patient at the time of admission/operation before other orders can be completed. Following completion of the assessment, the physician will be presented with an appropriate standard regimen based on the individual patient’s risk assessment. Early results of our VTE quality improvement project have been satisfying: in the most recent NSQIP semi-annual report, our O/E for VTE was 0.74, placing us in the first decile. Some of these early reports may simply be the product of the Hawthorne effect; however, we are encouraged by the early improvements seen in other research. While we are hopeful that these changes will result in sustainable improvements in outcomes, patients at extremely high risk may require novel weight-based or otherwise customized aggressive prophylactic regimens. Such regimens have already been proposed for arthroplasty and other high-risk patients.

Future research may identify other risk factors not captured by traditional risk assessments. In addition, research should continue to explore the use and efficacy of standard prophylactic regimens in these populations to help determine if they are sufficient. Currently, weight-based low-molecular-weight heparin dosing and alternative regimens employing fondaparinux are under investigation for very-high-risk patients.¹⁹

There were several limitations to the present study. First, due to the retrospective design of our study, we could collect only data that had been uniformly recorded in the charts throughout the study period. Second, we were unable to accurately assess compliance with mechanical prophylaxis. While our chart review showed that the vast majority of cases and controls were ordered to have mechanical prophylaxis, it is impossible to document how often these devices were used appropriately in a retrospective analysis. Anecdotal observation suggests that once patients are out of post-anesthesia or critical care units, SCD use is not standardized. The inability to measure compliance precisely may be leading to an overestimation of our compliance with prophylaxis. Finally, because our study included only patients who underwent surgery at our hospital, our observations may not be generalizable outside our institution.

Conclusion

Our study findings reinforce the importance of attention to detail in VTE risk assessment and in ordering and administering VTE prophylactic regimens, especially in high-risk surgical patients. While we adhered to the SCIP-mandated prophylaxis requirements, the complexity of our patients and our lack of a truly standardized approach to risk assessment and prophylactic regimens resulted in suboptimal outcomes. Stricter and more quantitative mandatory VTE risk assessment, along with highly standardized VTE prophylaxis regimens, are required to achieve optimal outcomes.

Corresponding author: Jason C. DeGiovanni, MS, BA, [email protected].

Financial disclosures: None.

From Tufts Medical Center, Boston, MA.

Abstract

• Objective: Audits at our academic medical center revealed near 100% compliance with protocols for perioperative venous thromboembolism (VTE) prophylaxis, but recent National Surgical Quality Improvement Program data demonstrated a higher than expected incidence of VTE (observed/expected = 1.32). The objective of this study was to identify potential causes of this discrepancy.
• Design: Retrospective case-control study.
• Setting: Urban academic medical center with high case-mix indices (Medicare approximately 2.4, non-Medicare approximately 2.0).
• Participants: 102 surgical inpatients with VTE (September 2012 to October 2015) matched with controls for age, gender, and type of procedure.
• Measurements: Prevalence of common VTE risk factors, length of stay, number of procedures, index operation times, and postoperative bed rest > 12 hours were assessed. Utilization of and compliance with our VTE risk assessment tool was also investigated.
• Results: Cases underwent more procedures and had longer lengths of stay and index procedures than controls. In addition, cases were more likely to have had > 12 hours of postoperative bed rest and central venous access than controls. Cases had more infections and were more likely to have severe lung disease, thrombophilia, and a history of prior VTE than controls. No differences in body mass index, tobacco use, current or previous malignancy, or VTE risk assessment form use were observed. Overall, care complexity and risk factors were equally important in determining VTE incidence. Our analyses also revealed lack of strict adherence to our VTE risk stratification protocol and frequent use of suboptimal prophylactic regimens.
• Conclusion: Well-accepted risk factors and overall care complexity determine VTE risk. Preventing VTE in high-risk patients requires assiduous attention to detail in VTE risk assessment and in delivery of optimal prophylaxis. Patients at especially high risk may require customized prophylactic regimens.

Keywords: hospital-acquired venous thromboembolic disease; VTE prophylaxis, surgical patients.

Deep vein thrombosis (DVT) and pulmonary embolism (PE) are well-recognized causes of morbidity and mortality in surgical patients. Between 350,000 and 600,000 cases of venous thromboembolism (VTE) occur each year in the United States, and it is responsible for approximately 10% of preventable in-hospital fatalities.^1-3 Given VTE’s impact on patients and the healthcare system and the fact that it is preventable, intense effort has been focused on developing more effective prophylactic measures to decrease its incidence.^2-4 In 2008, the surgeon general issued a “call to action” for increased efforts to prevent VTE.⁵

The American College of Chest Physicians (ACCP) guidelines subcategorize patients based on type of surgery. In addition, the ACCP guidelines support the use of a Caprini-based scoring system to aid in risk stratification and improve clinical decision-making (Table 1).^4,6-9 In general, scores ≥ 5 qualify individuals as high risk. Based on their risk category, patients receive mechanical prophylaxis, chemical prophylaxis, or a combination of the 2. Lower-risk patients who are ambulatory typically receive only mechanical prophylaxis while in bed, whereas higher-risk patients receive a combination of mechanical prophylaxis and chemoprophylaxis measures.⁷ In general, low-molecular-weight heparin (40 mg daily) and low-dose unfractionated heparin (5000 units 3 times daily) have been the standard evidence-based options for chemoprophylaxis in surgical patients. Absolute contraindications for prophylaxis include active bleeding and known increased risk of bleeding based on patient- or procedure-specific factors.

Our hospital, a 350-bed academic medical center in downtown Boston, MA, serving a diverse population with a very high case-mix index (2.4 Medicare and 2.0 non-Medicare), has strict protocols for VTE prophylaxis consistent with the ACCP guidelines and based on the Surgical Care Improvement Project (SCIP) measures published in 2006.¹⁰ The SCIP mandates allow for considerable surgeon discretion in the use of chemoprophylaxis for neurosurgical cases and general and orthopedic surgery cases deemed to be at high risk for bleeding. In addition, SCIP requires only that prophylaxis be initiated within 24 hours of surgical end time. Although recent audits revealed nearly 100% compliance with SCIP-mandated protocols, National Surgical Quality Improvement Program (NSQIP) data showed that the incidence of VTE events at our institution was higher than expected (observed/expected [O/E] = 1.32).

In order to determine the reasons for this mismatch between process and outcome performance, we investigated whether there were characteristics of our patient population that contributed to the higher than expected rates of VTE, and we scrutinized our VTE prophylaxis protocol to determine if there were aspects of our process that were also contributory.

Methods

Study Sample

This is a retrospective case-control study of surgical inpatients at our hospital during the period September 2012 to October 2015. Cases were identified as patients diagnosed with a VTE (DVT or PE). Controls were identified from a pool of surgical patients whose courses were not complicated by VTE during the same time frame as the cases and who were matched as closely as possible by procedure code, age, and gender.

 

 

Variables

Patient and hospital course variables that were analyzed included demographics, comorbidities, length of stay, number of procedures, index operation times, duration of postoperative bed rest, use of mechanical prophylaxis, and type of chemoprophylaxis and time frame within which it was initiated. Data were collected via chart review using International Classification of Diseases-9 and -10 codes to identify surgical cases within the allotted time period who were diagnosed with VTE. Demographic variables included age, sex, and ethnicity. Comorbidities included hypertension, diabetes, coronary artery disease, serious lung disease, previous or current malignancy, documented hypercoagulable state, and previous history of VTE. Body mass index (BMI) was also recorded. The aforementioned disease-specific variables were not matched between the case and control groups, as this data was obtained retrospectively during data collection.

Analysis

Associations between case and matched control were analyzed using the paired t-test for continuous variables and McNemar’s test for categorical variables. P values < 0.05 were considered statistically significant. SAS Enterprise Guide 7.15 (Cary, NC) was used for all statistical analyses.

The requirement for informed consent was waived by our Institutional Review Board, as the study was initially deemed to be a quality improvement project, and all data used for this report were de-identified.

Results

Our retrospective case-control analysis included a sample of 102 surgical patients whose courses were complicated by VTE between September 2012 and October 2015. The cases were distributed among 6 different surgical categories (Figure 1): trauma (20%), cancer (10%), cardiovascular (21%), noncancer neurosurgery (28%), elective orthopedics (11%), and miscellaneous general surgery (10%).

Comparisons between cases and controls in terms of patient demographics and risk factors are shown in Table 2. No statistically significant difference was observed in ethnicity or race between the 2 groups. Overall, cases had more hip/pelvis/leg fractures at presentation (P = 0.0008). The case group also had higher proportions of patients with postoperative bed rest greater than 12 hours (P = 0.009), central venous access (P < 0.0001), infection (P < 0.0001), and lower extremity edema documented during the hospitalization prior to development of DVT (P < 0.0001). Additionally, cases had significantly greater rates of previous VTE (P = 0.0004), inherited or acquired thrombophilia (P = 0.03), history of stroke (P = 0.0003), and severe lung disease, including pneumonia (P = 0.0008). No significant differences were noted between cases and matched controls in BMI (P = 0.43), current tobacco use (P = 0.71), current malignancy (P = 0.80), previous malignancy (P = 0.83), head trauma (P = 0.17), or acute cardiac disease (myocardial infarction or congestive heart failure; P = 0.12).

Variables felt to indicate overall complexity of hospital course for cases as compared to controls are outlined in Table 3. Cases were found to have significantly longer lengths of stay (median, 15.5 days versus 3 days, P < 0.0001). To account for the possibility that the development of VTE contributed to the increased length of stay in the cases, we also looked at the duration between admission date and the date of VTE diagnosis and determined that cases still had a longer length of stay when this was accounted for (median, 7 days versus 3 days, P < 0.0001). A much higher proportion of cases underwent more than 1 procedure compared to controls (P < 0.0001), and cases had significantly longer index operations as compared to controls (P = 0.002).

 

Seventeen cases received heparin on induction during their index procedure, compared to 23 controls (P = 0.24). Additionally, 63 cases began a prophylaxis regimen within 24 hours of surgery end time, compared to 68 controls (P = 0.24). The chemoprophylactic regimens utilized in cases and in controls are summarized in Figure 2. Of note, only 26 cases and 32 controls received standard prophylactic regimens with no missed doses (heparin 5000 units 3 times daily or enoxaparin 40 mg daily). Additionally, in over half of cases and a third of controls, nonstandard regimens were ordered. Examples of nonstandard regimens included nonstandard heparin or enoxaparin doses, low-dose warfarin, or aspirin alone. In most cases, nonstandard regimens were justified on the basis of high risk for bleeding.

Mechanical prophylaxis with pneumatic sequential compression devices (SCDs) was ordered in 93 (91%) cases and 87 (85%) controls; however, we were unable to accurately document uniform compliance in the use of these devices.

With regard to evaluation of our process measures, we found only 17% of cases and controls combined actually had a VTE risk assessment in their chart, and when it was present, it was often incomplete or was completed inaccurately.

Discussion

The goal of this study was to identify factors (patient characteristics and/or processes of care) that may be contributing to the higher than expected incidence of VTE events at our medical center, despite internal audits suggesting near perfect compliance with SCIP-mandated protocols. We found that in addition to usual risk factors for VTE, an overarching theme of our case cohort was their high complexity of illness. At baseline, these patients had significantly greater rates of stroke, thrombophilia, severe lung disease, infection, and history of VTE than controls. Moreover, the hospital courses of cases were significantly more complex than those of controls, as these patients had more procedures, longer lengths of stay and longer index operations, higher rates of postoperative bed rest exceeding 12 hours, and more prevalent central venous access than controls (Table 2). Several of these risk factors have been found to contribute to VTE development despite compliance with prophylaxis protocols.

Cassidy et al reviewed a cohort of nontrauma general surgery patients who developed VTE despite receiving appropriate prophylaxis and found that both multiple operations and emergency procedures contributed to the failure of VTE prophylaxis.¹¹ Similarly, Wang et al identified several independent risk factors for VTE despite thromboprophylaxis, including central venous access and infection, as well as intensive care unit admission, hospitalization for cranial surgery, and admission from a long-term care facility.¹² While our study did not capture some of these additional factors considered by Wang et al, the presence of risk factors not captured in traditional assessment tools suggests that additional consideration for complex patients is warranted.

 

In addition to these nonmodifiable patient characteristics, aspects of our VTE prophylaxis processes likely contributed to the higher than expected rate of VTE. While the electronic medical record at our institution does contain a VTE risk assessment tool based on the Caprini score, we found it often is not used at all or is used incorrectly/incompletely, which likely reflects the fact that physicians are neither prompted nor required to complete the assessment prior to prescribing VTE prophylaxis.

There is a significant body of evidence demonstrating that mandatory computerized VTE risk assessments can effectively reduce VTE rates and that improved outcomes occur shortly after implementation. Cassidy et al demonstrated the benefits of instituting a hospital-wide, mandatory, Caprini-based computerized VTE risk assessment that provides prophylaxis/early ambulation recommendations. Two years after implementing this system, they observed an 84% reduction in DVTs (P < 0.001) and a 55% reduction in PEs (P < 0.001).¹³ Nimeri et al had similarly impressive success, achieving a reduction in their NSQIP O/E for PE/DVT in general surgery from 6.00 in 2010 to 0.82 (for DVTs) and 0.78 (for PEs) 5 years after implementation of mandatory VTE risk assessment (though they noted that the most dramatic reduction occurred 1 year after implementation).¹⁴ Additionally, a recent systematic review and meta-analysis by Borab et al found computerized VTE risk assessments to be associated with a significant decrease in VTE events.¹⁵

The risk assessment tool used at our institution is qualitative in nature, and current literature suggests that employing a more quantitative tool may yield improved outcomes. Numerous studies have highlighted the importance of identifying patients at very high risk for VTE, as higher risk may necessitate more careful consideration of their prophylactic regimens. Obi et al found patients with Caprini scores higher than 8 to be at significantly greater risk of developing VTE compared to patients with scores of 7 or 8. Also, patients with scores of 7 or 8 were significantly more likely to have a VTE compared to those with scores of 5 or 6.¹⁶ In another study, Lobastov et al identified Caprini scores of 11 or higher as representing an extremely high-risk category for which standard prophylaxis regimens may not be effective.¹⁷ Thus, while having mandatory risk assessment has been shown to dramatically decrease VTE incidence, it is important to consider the magnitude of the numerical risk score. This is of particular importance at medical centers with high case-mix indices where patients at the highest risk might need to be managed with different prophylactic guidelines.

Another notable aspect of the process at our hospital was the great variation in the types of prophylactic regimens ordered, and the adherence to what was ordered. Only 25.5% of patients were maintained on a standard prophylactic regimen with no missed doses (heparin 5000 every 8 hours or enoxaparin 40 mg daily). Thus, the vast majority of the patients who went on to develop VTE either were prescribed a nontraditional prophylaxis regimen or missed doses of standard agents. The need for secondary surgical procedures or other invasive interventions may explain many, but not all, of the missed doses.

The timing of prophylaxis initiation for our patients was also found to deviate from accepted standards. Only 16.8% of cases received prophylaxis upon induction of anesthesia, and furthermore, 38% of cases did not receive any anticoagulation within 24 hours of their index operation. While this variability in prophylaxis implementation was acceptable within the SCIP guidelines based on “high risk for bleeding” or other considerations, it likely contributed to our suboptimal outcomes. The variations and interruptions in prophylactic regimens speak to barriers that have previously been reported as contributing factors to noncompliance with VTE prophylaxis.¹⁸

 

Given these known barriers and the observed underutilization and improper use of our risk assessment tool, we have recently changed our surgical admission order sets such that a mandatory quantitative risk assessment must be done for every surgical patient at the time of admission/operation before other orders can be completed. Following completion of the assessment, the physician will be presented with an appropriate standard regimen based on the individual patient’s risk assessment. Early results of our VTE quality improvement project have been satisfying: in the most recent NSQIP semi-annual report, our O/E for VTE was 0.74, placing us in the first decile. Some of these early reports may simply be the product of the Hawthorne effect; however, we are encouraged by the early improvements seen in other research. While we are hopeful that these changes will result in sustainable improvements in outcomes, patients at extremely high risk may require novel weight-based or otherwise customized aggressive prophylactic regimens. Such regimens have already been proposed for arthroplasty and other high-risk patients.

Future research may identify other risk factors not captured by traditional risk assessments. In addition, research should continue to explore the use and efficacy of standard prophylactic regimens in these populations to help determine if they are sufficient. Currently, weight-based low-molecular-weight heparin dosing and alternative regimens employing fondaparinux are under investigation for very-high-risk patients.¹⁹

There were several limitations to the present study. First, due to the retrospective design of our study, we could collect only data that had been uniformly recorded in the charts throughout the study period. Second, we were unable to accurately assess compliance with mechanical prophylaxis. While our chart review showed that the vast majority of cases and controls were ordered to have mechanical prophylaxis, it is impossible to document how often these devices were used appropriately in a retrospective analysis. Anecdotal observation suggests that once patients are out of post-anesthesia or critical care units, SCD use is not standardized. The inability to measure compliance precisely may be leading to an overestimation of our compliance with prophylaxis. Finally, because our study included only patients who underwent surgery at our hospital, our observations may not be generalizable outside our institution.

Conclusion

Our study findings reinforce the importance of attention to detail in VTE risk assessment and in ordering and administering VTE prophylactic regimens, especially in high-risk surgical patients. While we adhered to the SCIP-mandated prophylaxis requirements, the complexity of our patients and our lack of a truly standardized approach to risk assessment and prophylactic regimens resulted in suboptimal outcomes. Stricter and more quantitative mandatory VTE risk assessment, along with highly standardized VTE prophylaxis regimens, are required to achieve optimal outcomes.

Corresponding author: Jason C. DeGiovanni, MS, BA, [email protected].

Financial disclosures: None.

From Tufts Medical Center, Boston, MA.

Abstract

• Objective: Audits at our academic medical center revealed near 100% compliance with protocols for perioperative venous thromboembolism (VTE) prophylaxis, but recent National Surgical Quality Improvement Program data demonstrated a higher than expected incidence of VTE (observed/expected = 1.32). The objective of this study was to identify potential causes of this discrepancy.
• Design: Retrospective case-control study.
• Setting: Urban academic medical center with high case-mix indices (Medicare approximately 2.4, non-Medicare approximately 2.0).
• Participants: 102 surgical inpatients with VTE (September 2012 to October 2015) matched with controls for age, gender, and type of procedure.
• Measurements: Prevalence of common VTE risk factors, length of stay, number of procedures, index operation times, and postoperative bed rest > 12 hours were assessed. Utilization of and compliance with our VTE risk assessment tool was also investigated.
• Results: Cases underwent more procedures and had longer lengths of stay and index procedures than controls. In addition, cases were more likely to have had > 12 hours of postoperative bed rest and central venous access than controls. Cases had more infections and were more likely to have severe lung disease, thrombophilia, and a history of prior VTE than controls. No differences in body mass index, tobacco use, current or previous malignancy, or VTE risk assessment form use were observed. Overall, care complexity and risk factors were equally important in determining VTE incidence. Our analyses also revealed lack of strict adherence to our VTE risk stratification protocol and frequent use of suboptimal prophylactic regimens.
• Conclusion: Well-accepted risk factors and overall care complexity determine VTE risk. Preventing VTE in high-risk patients requires assiduous attention to detail in VTE risk assessment and in delivery of optimal prophylaxis. Patients at especially high risk may require customized prophylactic regimens.

Keywords: hospital-acquired venous thromboembolic disease; VTE prophylaxis, surgical patients.

Deep vein thrombosis (DVT) and pulmonary embolism (PE) are well-recognized causes of morbidity and mortality in surgical patients. Between 350,000 and 600,000 cases of venous thromboembolism (VTE) occur each year in the United States, and it is responsible for approximately 10% of preventable in-hospital fatalities.^1-3 Given VTE’s impact on patients and the healthcare system and the fact that it is preventable, intense effort has been focused on developing more effective prophylactic measures to decrease its incidence.^2-4 In 2008, the surgeon general issued a “call to action” for increased efforts to prevent VTE.⁵

The American College of Chest Physicians (ACCP) guidelines subcategorize patients based on type of surgery. In addition, the ACCP guidelines support the use of a Caprini-based scoring system to aid in risk stratification and improve clinical decision-making (Table 1).^4,6-9 In general, scores ≥ 5 qualify individuals as high risk. Based on their risk category, patients receive mechanical prophylaxis, chemical prophylaxis, or a combination of the 2. Lower-risk patients who are ambulatory typically receive only mechanical prophylaxis while in bed, whereas higher-risk patients receive a combination of mechanical prophylaxis and chemoprophylaxis measures.⁷ In general, low-molecular-weight heparin (40 mg daily) and low-dose unfractionated heparin (5000 units 3 times daily) have been the standard evidence-based options for chemoprophylaxis in surgical patients. Absolute contraindications for prophylaxis include active bleeding and known increased risk of bleeding based on patient- or procedure-specific factors.

Our hospital, a 350-bed academic medical center in downtown Boston, MA, serving a diverse population with a very high case-mix index (2.4 Medicare and 2.0 non-Medicare), has strict protocols for VTE prophylaxis consistent with the ACCP guidelines and based on the Surgical Care Improvement Project (SCIP) measures published in 2006.¹⁰ The SCIP mandates allow for considerable surgeon discretion in the use of chemoprophylaxis for neurosurgical cases and general and orthopedic surgery cases deemed to be at high risk for bleeding. In addition, SCIP requires only that prophylaxis be initiated within 24 hours of surgical end time. Although recent audits revealed nearly 100% compliance with SCIP-mandated protocols, National Surgical Quality Improvement Program (NSQIP) data showed that the incidence of VTE events at our institution was higher than expected (observed/expected [O/E] = 1.32).

In order to determine the reasons for this mismatch between process and outcome performance, we investigated whether there were characteristics of our patient population that contributed to the higher than expected rates of VTE, and we scrutinized our VTE prophylaxis protocol to determine if there were aspects of our process that were also contributory.

Methods

Study Sample

This is a retrospective case-control study of surgical inpatients at our hospital during the period September 2012 to October 2015. Cases were identified as patients diagnosed with a VTE (DVT or PE). Controls were identified from a pool of surgical patients whose courses were not complicated by VTE during the same time frame as the cases and who were matched as closely as possible by procedure code, age, and gender.

 

 

Variables

Patient and hospital course variables that were analyzed included demographics, comorbidities, length of stay, number of procedures, index operation times, duration of postoperative bed rest, use of mechanical prophylaxis, and type of chemoprophylaxis and time frame within which it was initiated. Data were collected via chart review using International Classification of Diseases-9 and -10 codes to identify surgical cases within the allotted time period who were diagnosed with VTE. Demographic variables included age, sex, and ethnicity. Comorbidities included hypertension, diabetes, coronary artery disease, serious lung disease, previous or current malignancy, documented hypercoagulable state, and previous history of VTE. Body mass index (BMI) was also recorded. The aforementioned disease-specific variables were not matched between the case and control groups, as this data was obtained retrospectively during data collection.

Analysis

Associations between case and matched control were analyzed using the paired t-test for continuous variables and McNemar’s test for categorical variables. P values < 0.05 were considered statistically significant. SAS Enterprise Guide 7.15 (Cary, NC) was used for all statistical analyses.

The requirement for informed consent was waived by our Institutional Review Board, as the study was initially deemed to be a quality improvement project, and all data used for this report were de-identified.

Results

Our retrospective case-control analysis included a sample of 102 surgical patients whose courses were complicated by VTE between September 2012 and October 2015. The cases were distributed among 6 different surgical categories (Figure 1): trauma (20%), cancer (10%), cardiovascular (21%), noncancer neurosurgery (28%), elective orthopedics (11%), and miscellaneous general surgery (10%).

Comparisons between cases and controls in terms of patient demographics and risk factors are shown in Table 2. No statistically significant difference was observed in ethnicity or race between the 2 groups. Overall, cases had more hip/pelvis/leg fractures at presentation (P = 0.0008). The case group also had higher proportions of patients with postoperative bed rest greater than 12 hours (P = 0.009), central venous access (P < 0.0001), infection (P < 0.0001), and lower extremity edema documented during the hospitalization prior to development of DVT (P < 0.0001). Additionally, cases had significantly greater rates of previous VTE (P = 0.0004), inherited or acquired thrombophilia (P = 0.03), history of stroke (P = 0.0003), and severe lung disease, including pneumonia (P = 0.0008). No significant differences were noted between cases and matched controls in BMI (P = 0.43), current tobacco use (P = 0.71), current malignancy (P = 0.80), previous malignancy (P = 0.83), head trauma (P = 0.17), or acute cardiac disease (myocardial infarction or congestive heart failure; P = 0.12).

Variables felt to indicate overall complexity of hospital course for cases as compared to controls are outlined in Table 3. Cases were found to have significantly longer lengths of stay (median, 15.5 days versus 3 days, P < 0.0001). To account for the possibility that the development of VTE contributed to the increased length of stay in the cases, we also looked at the duration between admission date and the date of VTE diagnosis and determined that cases still had a longer length of stay when this was accounted for (median, 7 days versus 3 days, P < 0.0001). A much higher proportion of cases underwent more than 1 procedure compared to controls (P < 0.0001), and cases had significantly longer index operations as compared to controls (P = 0.002).

 

Seventeen cases received heparin on induction during their index procedure, compared to 23 controls (P = 0.24). Additionally, 63 cases began a prophylaxis regimen within 24 hours of surgery end time, compared to 68 controls (P = 0.24). The chemoprophylactic regimens utilized in cases and in controls are summarized in Figure 2. Of note, only 26 cases and 32 controls received standard prophylactic regimens with no missed doses (heparin 5000 units 3 times daily or enoxaparin 40 mg daily). Additionally, in over half of cases and a third of controls, nonstandard regimens were ordered. Examples of nonstandard regimens included nonstandard heparin or enoxaparin doses, low-dose warfarin, or aspirin alone. In most cases, nonstandard regimens were justified on the basis of high risk for bleeding.

Mechanical prophylaxis with pneumatic sequential compression devices (SCDs) was ordered in 93 (91%) cases and 87 (85%) controls; however, we were unable to accurately document uniform compliance in the use of these devices.

With regard to evaluation of our process measures, we found only 17% of cases and controls combined actually had a VTE risk assessment in their chart, and when it was present, it was often incomplete or was completed inaccurately.

Discussion

The goal of this study was to identify factors (patient characteristics and/or processes of care) that may be contributing to the higher than expected incidence of VTE events at our medical center, despite internal audits suggesting near perfect compliance with SCIP-mandated protocols. We found that in addition to usual risk factors for VTE, an overarching theme of our case cohort was their high complexity of illness. At baseline, these patients had significantly greater rates of stroke, thrombophilia, severe lung disease, infection, and history of VTE than controls. Moreover, the hospital courses of cases were significantly more complex than those of controls, as these patients had more procedures, longer lengths of stay and longer index operations, higher rates of postoperative bed rest exceeding 12 hours, and more prevalent central venous access than controls (Table 2). Several of these risk factors have been found to contribute to VTE development despite compliance with prophylaxis protocols.

Cassidy et al reviewed a cohort of nontrauma general surgery patients who developed VTE despite receiving appropriate prophylaxis and found that both multiple operations and emergency procedures contributed to the failure of VTE prophylaxis.¹¹ Similarly, Wang et al identified several independent risk factors for VTE despite thromboprophylaxis, including central venous access and infection, as well as intensive care unit admission, hospitalization for cranial surgery, and admission from a long-term care facility.¹² While our study did not capture some of these additional factors considered by Wang et al, the presence of risk factors not captured in traditional assessment tools suggests that additional consideration for complex patients is warranted.

 

In addition to these nonmodifiable patient characteristics, aspects of our VTE prophylaxis processes likely contributed to the higher than expected rate of VTE. While the electronic medical record at our institution does contain a VTE risk assessment tool based on the Caprini score, we found it often is not used at all or is used incorrectly/incompletely, which likely reflects the fact that physicians are neither prompted nor required to complete the assessment prior to prescribing VTE prophylaxis.

There is a significant body of evidence demonstrating that mandatory computerized VTE risk assessments can effectively reduce VTE rates and that improved outcomes occur shortly after implementation. Cassidy et al demonstrated the benefits of instituting a hospital-wide, mandatory, Caprini-based computerized VTE risk assessment that provides prophylaxis/early ambulation recommendations. Two years after implementing this system, they observed an 84% reduction in DVTs (P < 0.001) and a 55% reduction in PEs (P < 0.001).¹³ Nimeri et al had similarly impressive success, achieving a reduction in their NSQIP O/E for PE/DVT in general surgery from 6.00 in 2010 to 0.82 (for DVTs) and 0.78 (for PEs) 5 years after implementation of mandatory VTE risk assessment (though they noted that the most dramatic reduction occurred 1 year after implementation).¹⁴ Additionally, a recent systematic review and meta-analysis by Borab et al found computerized VTE risk assessments to be associated with a significant decrease in VTE events.¹⁵

The risk assessment tool used at our institution is qualitative in nature, and current literature suggests that employing a more quantitative tool may yield improved outcomes. Numerous studies have highlighted the importance of identifying patients at very high risk for VTE, as higher risk may necessitate more careful consideration of their prophylactic regimens. Obi et al found patients with Caprini scores higher than 8 to be at significantly greater risk of developing VTE compared to patients with scores of 7 or 8. Also, patients with scores of 7 or 8 were significantly more likely to have a VTE compared to those with scores of 5 or 6.¹⁶ In another study, Lobastov et al identified Caprini scores of 11 or higher as representing an extremely high-risk category for which standard prophylaxis regimens may not be effective.¹⁷ Thus, while having mandatory risk assessment has been shown to dramatically decrease VTE incidence, it is important to consider the magnitude of the numerical risk score. This is of particular importance at medical centers with high case-mix indices where patients at the highest risk might need to be managed with different prophylactic guidelines.

Another notable aspect of the process at our hospital was the great variation in the types of prophylactic regimens ordered, and the adherence to what was ordered. Only 25.5% of patients were maintained on a standard prophylactic regimen with no missed doses (heparin 5000 every 8 hours or enoxaparin 40 mg daily). Thus, the vast majority of the patients who went on to develop VTE either were prescribed a nontraditional prophylaxis regimen or missed doses of standard agents. The need for secondary surgical procedures or other invasive interventions may explain many, but not all, of the missed doses.

The timing of prophylaxis initiation for our patients was also found to deviate from accepted standards. Only 16.8% of cases received prophylaxis upon induction of anesthesia, and furthermore, 38% of cases did not receive any anticoagulation within 24 hours of their index operation. While this variability in prophylaxis implementation was acceptable within the SCIP guidelines based on “high risk for bleeding” or other considerations, it likely contributed to our suboptimal outcomes. The variations and interruptions in prophylactic regimens speak to barriers that have previously been reported as contributing factors to noncompliance with VTE prophylaxis.¹⁸

 

Given these known barriers and the observed underutilization and improper use of our risk assessment tool, we have recently changed our surgical admission order sets such that a mandatory quantitative risk assessment must be done for every surgical patient at the time of admission/operation before other orders can be completed. Following completion of the assessment, the physician will be presented with an appropriate standard regimen based on the individual patient’s risk assessment. Early results of our VTE quality improvement project have been satisfying: in the most recent NSQIP semi-annual report, our O/E for VTE was 0.74, placing us in the first decile. Some of these early reports may simply be the product of the Hawthorne effect; however, we are encouraged by the early improvements seen in other research. While we are hopeful that these changes will result in sustainable improvements in outcomes, patients at extremely high risk may require novel weight-based or otherwise customized aggressive prophylactic regimens. Such regimens have already been proposed for arthroplasty and other high-risk patients.

Future research may identify other risk factors not captured by traditional risk assessments. In addition, research should continue to explore the use and efficacy of standard prophylactic regimens in these populations to help determine if they are sufficient. Currently, weight-based low-molecular-weight heparin dosing and alternative regimens employing fondaparinux are under investigation for very-high-risk patients.¹⁹

There were several limitations to the present study. First, due to the retrospective design of our study, we could collect only data that had been uniformly recorded in the charts throughout the study period. Second, we were unable to accurately assess compliance with mechanical prophylaxis. While our chart review showed that the vast majority of cases and controls were ordered to have mechanical prophylaxis, it is impossible to document how often these devices were used appropriately in a retrospective analysis. Anecdotal observation suggests that once patients are out of post-anesthesia or critical care units, SCD use is not standardized. The inability to measure compliance precisely may be leading to an overestimation of our compliance with prophylaxis. Finally, because our study included only patients who underwent surgery at our hospital, our observations may not be generalizable outside our institution.

Conclusion

Our study findings reinforce the importance of attention to detail in VTE risk assessment and in ordering and administering VTE prophylactic regimens, especially in high-risk surgical patients. While we adhered to the SCIP-mandated prophylaxis requirements, the complexity of our patients and our lack of a truly standardized approach to risk assessment and prophylactic regimens resulted in suboptimal outcomes. Stricter and more quantitative mandatory VTE risk assessment, along with highly standardized VTE prophylaxis regimens, are required to achieve optimal outcomes.

Corresponding author: Jason C. DeGiovanni, MS, BA, [email protected].

Financial disclosures: None.

Studies have shown that a majority of patients remain confused about their health care plans after being discharged from the hospital.^1,2 Furthermore, most patients do not recognize their lack of comprehension.² A substantial proportion of medical information is forgotten immediately after discharge. Kessels found that when larger amounts of information were presented, less was recalled, and almost half of the recalled information was incorrect.³ Researchers also have found that health information that was focused on individual needs not only increased patients’ understanding of their health needs and improved their health literacy, but supported self-management and promoted health outcomes for adults with chronic illness.^4,5

Health literacy is the “capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”⁶ To read instructions on a prescription bottle, patients need an intermediate level of health literacy. Even for patients with such a level of health literacy, comprehending and managing a health care plan for a chronic disease can be challenging. About 35% of Americans had lower than an intermediate level of health literacy.⁷ Insufficient health literacy is associated with increased health system use and costs, health disparities, and poor health outcomes.⁸ As a result, it is crucial to gear oral instructions to patients’ health literacy levels to ensure that patients understand health information and instructions and perform self-care at home. The teach-back method, a technique for verifying patients’ understanding of their health information, has been recommended by the Agency for Healthcare Research and Quality (AHRQ) and the Institute for Healthcare Improvement (IHI) as a strategy for taking universal precautions for health literacy. Patients are asked to repeat the instructions they receive from their health care professionals (HCPs). HCPs should use caring and plain language in a shame-free environment during patient education. By using the teach-back method, HCPs can assess patients’ understanding, and reteach or modify teaching if comprehension is not demonstrated. Patients have an important role in their health and their ability to understand health information has a significant impact on their health behavior and outcomes.

In our systematic research, we examined the effectiveness of using the teach-back method to understand health education as well as the impact of this method on patients’ disease self-management and health outcomes.

Methods

In the teach-back method, patients explain health information in their own words.⁹ To gauge the use and effectiveness of this method, investigators have studied patient perceptions and acknowledgments of the method as well as the effects of the method on health interventions. According to Dorothea Orem’s self-care deficit nursing theory, disease self-management is an “executive ability” to “control, handle, direct or govern” self-care activities.¹⁰ We define disease self-management as disease knowledge and disease management changes that promote self-care activities. In addition, we define health outcomes as health changes that result from the teach-back method, such as changes in postdischarge readmission rates, patient satisfaction, and health behavior.

 

Inclusion Criteria

We systematically reviewed evidence regarding the teach-back method as an educational intervention for patients aged ≥ 18 years. We included articles if they reported the process and outcomes of using the method alone or in combination with other educational strategies. The literature search focused on English-language articles published in peer-reviewed journals. Included in the review were qualitative, randomized controlled trials (RCTs); quasi-experimental studies; cohort studies; and pretest–posttest studies on the effects of the teach-back method. As the method can be applied in any health care setting, we used studies conducted in a variety of settings, including primary care, inpatient, outpatient, emergency department (ED), and community, in any time frame. Study participants had heart failure, diabetes mellitus (DM), hypertension, asthma, or other chronic diseases.

Exclusion Criteria

Studies that used the teach-back method as an outcome measurement but not an intervention were excluded. For example, those that used the method to measure patients’ postintervention understanding were excluded. Also excluded were those that used the method to examine HCP training or to measure HCP outcomes (ie, studies that did not use the method for patient education or outcomes).

Literature Search

In September 2017, we searched 4 databases: Ovid Medline, PubMed, EBSCO (Elton B. Stephens Co), CINAHL (Cumulative Index to Nursing and Allied Health Literature), and ProQuest. Also included were relevant studies from cited reference searching (Figure).

This systematic review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline for searches and formatting results. The literature search was performed with the term teach-back and terms from the structured PICO (population, intervention, comparison, outcomes) statement. The study population consisted of patients who received the teach-back intervention as part of the patient education process in a medical care setting, and the comparator population consisted of patients who did not receive the intervention in their patient education. Target outcomes were disease self-management, self-care, patient satisfaction, patient perception and acknowledgment of the teach-back method, and other health outcomes.

Data Collection

Data collected included authors, publication date, and journal; purpose; study design; setting, sample, and population; intervention; and outcomes.

The methodologic quality of papers retrieved for review was determined with Critical Appraisals Skills Programme (CASP) guidelines (casp-uk.net/casp-tools-checklists). CASP randomised controlled trial, cohort study, case control study, and qualitative checklists were used. The authors assessed the full texts for eligibility. Disagreements were resolved through discussion.

The initial literature search found 112, 135, and 161 articles from EBSCO CINAHL, Ovid Medline, and PubMed, respectively. Five articles from ProQuest were identified through the EBSCO CINAHL search. After inclusion and exclusion criteria were applied, duplicate articles removed, a cited reference added, and CASP criteria assessed, 26 articles remained in the review. The 26 studies consisted of 15 cohort studies, 5 case–control studies, 5 RCTs, and 1 qualitative interview. Twenty-two of the articles were published in the US, the other 4 in Australia and Iran (2 each).^11-14 All 26 studies used the teach-back method with other educational interventions to reinforce learning (eg, the method was used after heart failure or DM education). Of the 26 studies, 10 used a pretest–posttest intervention design,^15-24 and 10 used a quasi-experimental or experimental design.^{11,13,14,25-31}

 

Results

The common outcome measures used in the 26 studies fall into 5 categories: patient satisfaction; postdischarge readmission; patient perception of teach-back method effectiveness; disease knowledge and disease management improvements; and intervention effects on health-related quality of life (HR-QOL). A summary of included articles, study setting, design, outcomes, and details is available from the author.

Patient Satisfaction

Ten studies examined the impact of the teach-back method on patient satisfaction.^{15,17,19,21,23,26,27,29,31,32} Of these 10 studies, 6 explored the influence of the method on Hospital Consumer Assessment of Healthcare Providers and Systems survey scores.^{15,17,19,21,22,26} All included studies indicated improved satisfaction with medication education, discharge information, and health management—except for the Silva study, who found an upward trend but not a statistically significant improvement in patient understanding of the purpose of a medication.²³

Grice and colleagues also found that community-dwelling seniors expressed satisfaction with using the teach-back method while being evaluated and assessed for health services at home.³² Improvement or a positive trend in teach-back groups was reported in a majority of the studies except for those by Hyrkas and Wiggins, and Griffey and colleagues.^27,29 Hyrkas and Wiggins found the method slightly improved patients’ medication confidence after hospital discharge, though patient satisfaction scores were associated with patient–nurse relationships, not with use of the teach-back method and a motivational interview.²⁷ Similarly, Griffey and colleagues found that patients who had limited health literacy and received a standard discharge with teach-back scored higher on medication comprehension, compared with patients who received only a standard discharge, but there was no difference in patient satisfaction after an ED visit.²⁹

Postdischarge Readmission

Results emphasized the importance of teach-back in reinforcing discharge instructions and improving postdischarge readmission rates. Of the 6 studies on the effect that teach-back with discharge summary had on readmission rates, 2 found statistically significant improvement for patients with heart failure at 12 months (teach-back, 59%; non-teach-back, 44%; P = .005) and patients with coronary artery bypass grafting (CABG) at 30 days (preintervention, 25%; postintervention, 12%; P = .02).^11,16 In addition, 3 of the 6 studies reported improvement but did not provide P values.^18,20,22 One study indicated improvement in other measured outcomes but found no significant difference for patients who received teach-back with their discharge summaries.²⁷ In all studies, teach-back was added to an intervention and used to confirm and promote knowledge and self-care management.

Patient Perception of Teach-Back Effectiveness

In 2 qualitative studies, patients indicated teach-back was an effective educational method.^16,33 For patients with CABG, Bates and colleagues added a scheduled cardiology follow-up appointment and teach-back patient education to their State Action on Avoidable Rehospitalizations interventions; 96% of participants rated teach-back effective or highly effective.¹⁶ In the other study, Samuels-Kalow and colleagues interviewed 51 patients and parents who received teach-back as part of the discharge process in 2 EDs; participants indicated teach-back helped them remember what they learned from their HCPs, and gave them the opportunity to connect with their HCPs, though some with lower health literacy expressed concerns about perceived judgment by HCPs.³³

 

Disease Knowledge and Management

Thirteen studies examined knowledge improvement after interventions that included teach-back. Study participants answered most questions correctly after receiving teach-back.^20,32,34,35 Slater and colleagues found ED patients who received discharge instructions with teach-back had significantly higher scores measuring knowledge of diagnosis (P < .001), signs and symptoms indicating a need to return to the ED (P < .001), and follow-up instructions (P = .03); scores measuring knowledge of medication were higher as well, but were not statistically different (P = .14).²⁴ In multiple studies, improvement was not always statistically significant in terms of knowledge retention.^{12,25,29-31,36} Studies that compared medication adherence found teach-back was more effective than motivational interviews (P = .56).²⁷

Teach-back has been widely used in primary care, inpatient, and ED settings. Two studies on the effect of teach-back in primary care sampled patients with DM.^28,36 Kandula and colleagues found that participants who answered questions incorrectly after watching a multimedia DM education program could significantly improve their DM knowledge by engaging in teach-back immediately after the intervention; however, knowledge retention was not improved at 2-week follow-up (phone call).²⁸ In contrast, Swavely and colleagues compared patients who completed a 13-hour DM education program with or without teach-back and found that teach-back patients demonstrated significantly improved DM knowledge and self-care activities at 3 months.³⁶

Effects of Interventions on HR-QOL

The teach-back method had been used with QOL improvement programs and other interventions. Ahmadidarrehsima and colleagues incorporated teach-back into their medical self-management program (8 to 11 sessions, each lasting 1.5 to 2 hours) for women with breast cancer and found that the mean happiness score increased to 62.9 from 37.2 (P < .001) in the intervention group, whereas the score for the usual-care group decreased from 41.4 to 29.8.¹³ Ghiasvand and colleagues compared QOL of postpartum mothers who received routine care with QOL of those who received routine care plus 2 sessions of postpartum self-care with teach-back; mean QOL scores were significantly (P < .001) higher for the teach-back group (124.73) than for the no teach-back group (115.03).¹⁴

Discussion

This review examined the use and effectiveness of the teach-back method in health education and its influence in patients’ disease self-management and health outcomes. Results showed positive effects of teach-back on patient satisfaction, patient perceptions and acknowledgments, postdischarge readmissions, disease self-management and knowledge, and HR-QOL.

The teach-back method has been widely used in inpatient, outpatient, ED, and community settings as part of health education programs and interventions. It has been paired with educational interventions ranging from short instructions to 20-hour programs. These differences reflect the broad application of the method in patient education. Many studies have found that teach-back improves disease knowledge and self-management, though their results are not always statistically significant. In an RCT of patients with low health literacy, Griffey and colleagues studied the effect of ED discharge education with and without teach-back and found teach-back did not increase post-ED comprehension of diagnoses, medical examinations, and treatments or perceived comprehension of treatment and care; however, compared with the no teach-back group, the teach-back group had significantly higher scores on comprehension of post-ED self-care (P < .02), follow-up (P < .0001), and medication (P = .054).²⁹ This finding indicates teach-back is an effective method for helping patients understand self-care and disease self-management at home.

Comprehending medical diagnoses, examinations, and treatments involves acquiring, analyzing, and comparing multiple pieces of health information. Because comprehension requires a level of abstract thinking usually present in patients with intermediate and proficient health literacy,improvements might be more difficult to see in patients with low health literacy.⁸ Press and colleagues found that asthma patients who repeated respiratory inhaler instructions with teach-back during discharge education had less misuse of (P = .01) metered-dose and Diskus (P = .05) inhalers and lower 30-day readmission rates (P = .02) compared with the misuse of patients who received only 1 set of oral and written instructions.³¹ Even though the Diskus result was not statistically significant, it demonstrated teach-back can be used to improve patient self-care and education.³¹

Most participants in the reviewed studies improved their disease knowledge with teach-back, though the evidence regarding improved health care knowledge retention was limited. For example, the 2 studies on use of teach-back in primary care clinics had contradictory knowledge retention results.^28,36 As both studies incorporated teach-back into existing interventions, these results could be associated with those interventions and not with the teach-back method.

Health literacy is achieved through a complicated process of obtaining, analyzing, choosing, and communicating health information. Even though its knowledge retention results are inconsistent, the teach-back method is recommended by the American Academy of Family Physicians at strength of recommendation taxonomy level C.⁸ Such a designation indicates that the recommendation is based on expert opinion, bench research, consensus guideline, usual practice, clinical experience, or a case series and is appropriate for assessment of patient comprehension.³⁷ Teach-back is also suggested by AHRQ and IHI for university precautions regarding health literacy and as such should remain a standard of practice. More study is needed to understand the inconsistent results of knowledge retention and the long-term effects of the teach-back method.

 

Limitations

Although this review did not limit the publication years of its articles, no pre-2011 articles were found. The teach-back method has been used to measure patients’ postintervention understanding and to educate HCPs on ways to improve patient communication. As this review did not include studies of teach-back as an outcome measurement or studies of training and adaptation of teach-back in HCP or nurse education, other study results may have a bearing on the current findings. Teach-back has been used to close communication gaps between patients and HCPs.

All articles included in this review used the teach-back method with other educational or organizational interventions. The outcomes found in this review may be associated with those interventions and not with teach-back itself. Data reported here have not demonstrated a definite association between teach-back and the measured outcomes; therefore, caution should be exercised when drawing conclusions based on these data. In addition, most of the studies considered in this review were cohort or case–control studies; only 5 RCTs were included. Other confounding factors, including patient health literacy levels, HCP types, HCP competencies in use of teach-back, and type and duration of interventions used before teach-back, may have contributed to this review’s findings.

Conclusion

Findings of this systematic review support use of the teach-back method as effective in reinforcing or confirming patient education. As none of the included studies reported harmful outcomes, the teach-back method poses little risk with respect to increasing patients’ understanding of their education. The findings emphasize the importance of conducting more studies to try to understand the inconsistent results of knowledge retention and determine ways to preserve the long-term effects of teach-back.

Studies have shown that a majority of patients remain confused about their health care plans after being discharged from the hospital.^1,2 Furthermore, most patients do not recognize their lack of comprehension.² A substantial proportion of medical information is forgotten immediately after discharge. Kessels found that when larger amounts of information were presented, less was recalled, and almost half of the recalled information was incorrect.³ Researchers also have found that health information that was focused on individual needs not only increased patients’ understanding of their health needs and improved their health literacy, but supported self-management and promoted health outcomes for adults with chronic illness.^4,5

Health literacy is the “capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”⁶ To read instructions on a prescription bottle, patients need an intermediate level of health literacy. Even for patients with such a level of health literacy, comprehending and managing a health care plan for a chronic disease can be challenging. About 35% of Americans had lower than an intermediate level of health literacy.⁷ Insufficient health literacy is associated with increased health system use and costs, health disparities, and poor health outcomes.⁸ As a result, it is crucial to gear oral instructions to patients’ health literacy levels to ensure that patients understand health information and instructions and perform self-care at home. The teach-back method, a technique for verifying patients’ understanding of their health information, has been recommended by the Agency for Healthcare Research and Quality (AHRQ) and the Institute for Healthcare Improvement (IHI) as a strategy for taking universal precautions for health literacy. Patients are asked to repeat the instructions they receive from their health care professionals (HCPs). HCPs should use caring and plain language in a shame-free environment during patient education. By using the teach-back method, HCPs can assess patients’ understanding, and reteach or modify teaching if comprehension is not demonstrated. Patients have an important role in their health and their ability to understand health information has a significant impact on their health behavior and outcomes.

In our systematic research, we examined the effectiveness of using the teach-back method to understand health education as well as the impact of this method on patients’ disease self-management and health outcomes.

Methods

In the teach-back method, patients explain health information in their own words.⁹ To gauge the use and effectiveness of this method, investigators have studied patient perceptions and acknowledgments of the method as well as the effects of the method on health interventions. According to Dorothea Orem’s self-care deficit nursing theory, disease self-management is an “executive ability” to “control, handle, direct or govern” self-care activities.¹⁰ We define disease self-management as disease knowledge and disease management changes that promote self-care activities. In addition, we define health outcomes as health changes that result from the teach-back method, such as changes in postdischarge readmission rates, patient satisfaction, and health behavior.

 

Inclusion Criteria

We systematically reviewed evidence regarding the teach-back method as an educational intervention for patients aged ≥ 18 years. We included articles if they reported the process and outcomes of using the method alone or in combination with other educational strategies. The literature search focused on English-language articles published in peer-reviewed journals. Included in the review were qualitative, randomized controlled trials (RCTs); quasi-experimental studies; cohort studies; and pretest–posttest studies on the effects of the teach-back method. As the method can be applied in any health care setting, we used studies conducted in a variety of settings, including primary care, inpatient, outpatient, emergency department (ED), and community, in any time frame. Study participants had heart failure, diabetes mellitus (DM), hypertension, asthma, or other chronic diseases.

Exclusion Criteria

Studies that used the teach-back method as an outcome measurement but not an intervention were excluded. For example, those that used the method to measure patients’ postintervention understanding were excluded. Also excluded were those that used the method to examine HCP training or to measure HCP outcomes (ie, studies that did not use the method for patient education or outcomes).

Literature Search

In September 2017, we searched 4 databases: Ovid Medline, PubMed, EBSCO (Elton B. Stephens Co), CINAHL (Cumulative Index to Nursing and Allied Health Literature), and ProQuest. Also included were relevant studies from cited reference searching (Figure).

This systematic review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline for searches and formatting results. The literature search was performed with the term teach-back and terms from the structured PICO (population, intervention, comparison, outcomes) statement. The study population consisted of patients who received the teach-back intervention as part of the patient education process in a medical care setting, and the comparator population consisted of patients who did not receive the intervention in their patient education. Target outcomes were disease self-management, self-care, patient satisfaction, patient perception and acknowledgment of the teach-back method, and other health outcomes.

Data Collection

Data collected included authors, publication date, and journal; purpose; study design; setting, sample, and population; intervention; and outcomes.

The methodologic quality of papers retrieved for review was determined with Critical Appraisals Skills Programme (CASP) guidelines (casp-uk.net/casp-tools-checklists). CASP randomised controlled trial, cohort study, case control study, and qualitative checklists were used. The authors assessed the full texts for eligibility. Disagreements were resolved through discussion.

The initial literature search found 112, 135, and 161 articles from EBSCO CINAHL, Ovid Medline, and PubMed, respectively. Five articles from ProQuest were identified through the EBSCO CINAHL search. After inclusion and exclusion criteria were applied, duplicate articles removed, a cited reference added, and CASP criteria assessed, 26 articles remained in the review. The 26 studies consisted of 15 cohort studies, 5 case–control studies, 5 RCTs, and 1 qualitative interview. Twenty-two of the articles were published in the US, the other 4 in Australia and Iran (2 each).^11-14 All 26 studies used the teach-back method with other educational interventions to reinforce learning (eg, the method was used after heart failure or DM education). Of the 26 studies, 10 used a pretest–posttest intervention design,^15-24 and 10 used a quasi-experimental or experimental design.^{11,13,14,25-31}

 

Results

The common outcome measures used in the 26 studies fall into 5 categories: patient satisfaction; postdischarge readmission; patient perception of teach-back method effectiveness; disease knowledge and disease management improvements; and intervention effects on health-related quality of life (HR-QOL). A summary of included articles, study setting, design, outcomes, and details is available from the author.

Patient Satisfaction

Ten studies examined the impact of the teach-back method on patient satisfaction.^{15,17,19,21,23,26,27,29,31,32} Of these 10 studies, 6 explored the influence of the method on Hospital Consumer Assessment of Healthcare Providers and Systems survey scores.^{15,17,19,21,22,26} All included studies indicated improved satisfaction with medication education, discharge information, and health management—except for the Silva study, who found an upward trend but not a statistically significant improvement in patient understanding of the purpose of a medication.²³

Grice and colleagues also found that community-dwelling seniors expressed satisfaction with using the teach-back method while being evaluated and assessed for health services at home.³² Improvement or a positive trend in teach-back groups was reported in a majority of the studies except for those by Hyrkas and Wiggins, and Griffey and colleagues.^27,29 Hyrkas and Wiggins found the method slightly improved patients’ medication confidence after hospital discharge, though patient satisfaction scores were associated with patient–nurse relationships, not with use of the teach-back method and a motivational interview.²⁷ Similarly, Griffey and colleagues found that patients who had limited health literacy and received a standard discharge with teach-back scored higher on medication comprehension, compared with patients who received only a standard discharge, but there was no difference in patient satisfaction after an ED visit.²⁹

Postdischarge Readmission

Results emphasized the importance of teach-back in reinforcing discharge instructions and improving postdischarge readmission rates. Of the 6 studies on the effect that teach-back with discharge summary had on readmission rates, 2 found statistically significant improvement for patients with heart failure at 12 months (teach-back, 59%; non-teach-back, 44%; P = .005) and patients with coronary artery bypass grafting (CABG) at 30 days (preintervention, 25%; postintervention, 12%; P = .02).^11,16 In addition, 3 of the 6 studies reported improvement but did not provide P values.^18,20,22 One study indicated improvement in other measured outcomes but found no significant difference for patients who received teach-back with their discharge summaries.²⁷ In all studies, teach-back was added to an intervention and used to confirm and promote knowledge and self-care management.

Patient Perception of Teach-Back Effectiveness

In 2 qualitative studies, patients indicated teach-back was an effective educational method.^16,33 For patients with CABG, Bates and colleagues added a scheduled cardiology follow-up appointment and teach-back patient education to their State Action on Avoidable Rehospitalizations interventions; 96% of participants rated teach-back effective or highly effective.¹⁶ In the other study, Samuels-Kalow and colleagues interviewed 51 patients and parents who received teach-back as part of the discharge process in 2 EDs; participants indicated teach-back helped them remember what they learned from their HCPs, and gave them the opportunity to connect with their HCPs, though some with lower health literacy expressed concerns about perceived judgment by HCPs.³³

 

Disease Knowledge and Management

Thirteen studies examined knowledge improvement after interventions that included teach-back. Study participants answered most questions correctly after receiving teach-back.^20,32,34,35 Slater and colleagues found ED patients who received discharge instructions with teach-back had significantly higher scores measuring knowledge of diagnosis (P < .001), signs and symptoms indicating a need to return to the ED (P < .001), and follow-up instructions (P = .03); scores measuring knowledge of medication were higher as well, but were not statistically different (P = .14).²⁴ In multiple studies, improvement was not always statistically significant in terms of knowledge retention.^{12,25,29-31,36} Studies that compared medication adherence found teach-back was more effective than motivational interviews (P = .56).²⁷

Teach-back has been widely used in primary care, inpatient, and ED settings. Two studies on the effect of teach-back in primary care sampled patients with DM.^28,36 Kandula and colleagues found that participants who answered questions incorrectly after watching a multimedia DM education program could significantly improve their DM knowledge by engaging in teach-back immediately after the intervention; however, knowledge retention was not improved at 2-week follow-up (phone call).²⁸ In contrast, Swavely and colleagues compared patients who completed a 13-hour DM education program with or without teach-back and found that teach-back patients demonstrated significantly improved DM knowledge and self-care activities at 3 months.³⁶

Effects of Interventions on HR-QOL

The teach-back method had been used with QOL improvement programs and other interventions. Ahmadidarrehsima and colleagues incorporated teach-back into their medical self-management program (8 to 11 sessions, each lasting 1.5 to 2 hours) for women with breast cancer and found that the mean happiness score increased to 62.9 from 37.2 (P < .001) in the intervention group, whereas the score for the usual-care group decreased from 41.4 to 29.8.¹³ Ghiasvand and colleagues compared QOL of postpartum mothers who received routine care with QOL of those who received routine care plus 2 sessions of postpartum self-care with teach-back; mean QOL scores were significantly (P < .001) higher for the teach-back group (124.73) than for the no teach-back group (115.03).¹⁴

Discussion

This review examined the use and effectiveness of the teach-back method in health education and its influence in patients’ disease self-management and health outcomes. Results showed positive effects of teach-back on patient satisfaction, patient perceptions and acknowledgments, postdischarge readmissions, disease self-management and knowledge, and HR-QOL.

The teach-back method has been widely used in inpatient, outpatient, ED, and community settings as part of health education programs and interventions. It has been paired with educational interventions ranging from short instructions to 20-hour programs. These differences reflect the broad application of the method in patient education. Many studies have found that teach-back improves disease knowledge and self-management, though their results are not always statistically significant. In an RCT of patients with low health literacy, Griffey and colleagues studied the effect of ED discharge education with and without teach-back and found teach-back did not increase post-ED comprehension of diagnoses, medical examinations, and treatments or perceived comprehension of treatment and care; however, compared with the no teach-back group, the teach-back group had significantly higher scores on comprehension of post-ED self-care (P < .02), follow-up (P < .0001), and medication (P = .054).²⁹ This finding indicates teach-back is an effective method for helping patients understand self-care and disease self-management at home.

Comprehending medical diagnoses, examinations, and treatments involves acquiring, analyzing, and comparing multiple pieces of health information. Because comprehension requires a level of abstract thinking usually present in patients with intermediate and proficient health literacy,improvements might be more difficult to see in patients with low health literacy.⁸ Press and colleagues found that asthma patients who repeated respiratory inhaler instructions with teach-back during discharge education had less misuse of (P = .01) metered-dose and Diskus (P = .05) inhalers and lower 30-day readmission rates (P = .02) compared with the misuse of patients who received only 1 set of oral and written instructions.³¹ Even though the Diskus result was not statistically significant, it demonstrated teach-back can be used to improve patient self-care and education.³¹

Most participants in the reviewed studies improved their disease knowledge with teach-back, though the evidence regarding improved health care knowledge retention was limited. For example, the 2 studies on use of teach-back in primary care clinics had contradictory knowledge retention results.^28,36 As both studies incorporated teach-back into existing interventions, these results could be associated with those interventions and not with the teach-back method.

Health literacy is achieved through a complicated process of obtaining, analyzing, choosing, and communicating health information. Even though its knowledge retention results are inconsistent, the teach-back method is recommended by the American Academy of Family Physicians at strength of recommendation taxonomy level C.⁸ Such a designation indicates that the recommendation is based on expert opinion, bench research, consensus guideline, usual practice, clinical experience, or a case series and is appropriate for assessment of patient comprehension.³⁷ Teach-back is also suggested by AHRQ and IHI for university precautions regarding health literacy and as such should remain a standard of practice. More study is needed to understand the inconsistent results of knowledge retention and the long-term effects of the teach-back method.

 

Limitations

Although this review did not limit the publication years of its articles, no pre-2011 articles were found. The teach-back method has been used to measure patients’ postintervention understanding and to educate HCPs on ways to improve patient communication. As this review did not include studies of teach-back as an outcome measurement or studies of training and adaptation of teach-back in HCP or nurse education, other study results may have a bearing on the current findings. Teach-back has been used to close communication gaps between patients and HCPs.

All articles included in this review used the teach-back method with other educational or organizational interventions. The outcomes found in this review may be associated with those interventions and not with teach-back itself. Data reported here have not demonstrated a definite association between teach-back and the measured outcomes; therefore, caution should be exercised when drawing conclusions based on these data. In addition, most of the studies considered in this review were cohort or case–control studies; only 5 RCTs were included. Other confounding factors, including patient health literacy levels, HCP types, HCP competencies in use of teach-back, and type and duration of interventions used before teach-back, may have contributed to this review’s findings.

Conclusion

Findings of this systematic review support use of the teach-back method as effective in reinforcing or confirming patient education. As none of the included studies reported harmful outcomes, the teach-back method poses little risk with respect to increasing patients’ understanding of their education. The findings emphasize the importance of conducting more studies to try to understand the inconsistent results of knowledge retention and determine ways to preserve the long-term effects of teach-back.

Studies have shown that a majority of patients remain confused about their health care plans after being discharged from the hospital.^1,2 Furthermore, most patients do not recognize their lack of comprehension.² A substantial proportion of medical information is forgotten immediately after discharge. Kessels found that when larger amounts of information were presented, less was recalled, and almost half of the recalled information was incorrect.³ Researchers also have found that health information that was focused on individual needs not only increased patients’ understanding of their health needs and improved their health literacy, but supported self-management and promoted health outcomes for adults with chronic illness.^4,5

Health literacy is the “capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”⁶ To read instructions on a prescription bottle, patients need an intermediate level of health literacy. Even for patients with such a level of health literacy, comprehending and managing a health care plan for a chronic disease can be challenging. About 35% of Americans had lower than an intermediate level of health literacy.⁷ Insufficient health literacy is associated with increased health system use and costs, health disparities, and poor health outcomes.⁸ As a result, it is crucial to gear oral instructions to patients’ health literacy levels to ensure that patients understand health information and instructions and perform self-care at home. The teach-back method, a technique for verifying patients’ understanding of their health information, has been recommended by the Agency for Healthcare Research and Quality (AHRQ) and the Institute for Healthcare Improvement (IHI) as a strategy for taking universal precautions for health literacy. Patients are asked to repeat the instructions they receive from their health care professionals (HCPs). HCPs should use caring and plain language in a shame-free environment during patient education. By using the teach-back method, HCPs can assess patients’ understanding, and reteach or modify teaching if comprehension is not demonstrated. Patients have an important role in their health and their ability to understand health information has a significant impact on their health behavior and outcomes.

In our systematic research, we examined the effectiveness of using the teach-back method to understand health education as well as the impact of this method on patients’ disease self-management and health outcomes.

Methods

In the teach-back method, patients explain health information in their own words.⁹ To gauge the use and effectiveness of this method, investigators have studied patient perceptions and acknowledgments of the method as well as the effects of the method on health interventions. According to Dorothea Orem’s self-care deficit nursing theory, disease self-management is an “executive ability” to “control, handle, direct or govern” self-care activities.¹⁰ We define disease self-management as disease knowledge and disease management changes that promote self-care activities. In addition, we define health outcomes as health changes that result from the teach-back method, such as changes in postdischarge readmission rates, patient satisfaction, and health behavior.

 

Inclusion Criteria

We systematically reviewed evidence regarding the teach-back method as an educational intervention for patients aged ≥ 18 years. We included articles if they reported the process and outcomes of using the method alone or in combination with other educational strategies. The literature search focused on English-language articles published in peer-reviewed journals. Included in the review were qualitative, randomized controlled trials (RCTs); quasi-experimental studies; cohort studies; and pretest–posttest studies on the effects of the teach-back method. As the method can be applied in any health care setting, we used studies conducted in a variety of settings, including primary care, inpatient, outpatient, emergency department (ED), and community, in any time frame. Study participants had heart failure, diabetes mellitus (DM), hypertension, asthma, or other chronic diseases.

Exclusion Criteria

Studies that used the teach-back method as an outcome measurement but not an intervention were excluded. For example, those that used the method to measure patients’ postintervention understanding were excluded. Also excluded were those that used the method to examine HCP training or to measure HCP outcomes (ie, studies that did not use the method for patient education or outcomes).

Literature Search

In September 2017, we searched 4 databases: Ovid Medline, PubMed, EBSCO (Elton B. Stephens Co), CINAHL (Cumulative Index to Nursing and Allied Health Literature), and ProQuest. Also included were relevant studies from cited reference searching (Figure).

This systematic review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline for searches and formatting results. The literature search was performed with the term teach-back and terms from the structured PICO (population, intervention, comparison, outcomes) statement. The study population consisted of patients who received the teach-back intervention as part of the patient education process in a medical care setting, and the comparator population consisted of patients who did not receive the intervention in their patient education. Target outcomes were disease self-management, self-care, patient satisfaction, patient perception and acknowledgment of the teach-back method, and other health outcomes.

Data Collection

Data collected included authors, publication date, and journal; purpose; study design; setting, sample, and population; intervention; and outcomes.

The methodologic quality of papers retrieved for review was determined with Critical Appraisals Skills Programme (CASP) guidelines (casp-uk.net/casp-tools-checklists). CASP randomised controlled trial, cohort study, case control study, and qualitative checklists were used. The authors assessed the full texts for eligibility. Disagreements were resolved through discussion.

The initial literature search found 112, 135, and 161 articles from EBSCO CINAHL, Ovid Medline, and PubMed, respectively. Five articles from ProQuest were identified through the EBSCO CINAHL search. After inclusion and exclusion criteria were applied, duplicate articles removed, a cited reference added, and CASP criteria assessed, 26 articles remained in the review. The 26 studies consisted of 15 cohort studies, 5 case–control studies, 5 RCTs, and 1 qualitative interview. Twenty-two of the articles were published in the US, the other 4 in Australia and Iran (2 each).^11-14 All 26 studies used the teach-back method with other educational interventions to reinforce learning (eg, the method was used after heart failure or DM education). Of the 26 studies, 10 used a pretest–posttest intervention design,^15-24 and 10 used a quasi-experimental or experimental design.^{11,13,14,25-31}

 

Results

The common outcome measures used in the 26 studies fall into 5 categories: patient satisfaction; postdischarge readmission; patient perception of teach-back method effectiveness; disease knowledge and disease management improvements; and intervention effects on health-related quality of life (HR-QOL). A summary of included articles, study setting, design, outcomes, and details is available from the author.

Patient Satisfaction

Ten studies examined the impact of the teach-back method on patient satisfaction.^{15,17,19,21,23,26,27,29,31,32} Of these 10 studies, 6 explored the influence of the method on Hospital Consumer Assessment of Healthcare Providers and Systems survey scores.^{15,17,19,21,22,26} All included studies indicated improved satisfaction with medication education, discharge information, and health management—except for the Silva study, who found an upward trend but not a statistically significant improvement in patient understanding of the purpose of a medication.²³

Grice and colleagues also found that community-dwelling seniors expressed satisfaction with using the teach-back method while being evaluated and assessed for health services at home.³² Improvement or a positive trend in teach-back groups was reported in a majority of the studies except for those by Hyrkas and Wiggins, and Griffey and colleagues.^27,29 Hyrkas and Wiggins found the method slightly improved patients’ medication confidence after hospital discharge, though patient satisfaction scores were associated with patient–nurse relationships, not with use of the teach-back method and a motivational interview.²⁷ Similarly, Griffey and colleagues found that patients who had limited health literacy and received a standard discharge with teach-back scored higher on medication comprehension, compared with patients who received only a standard discharge, but there was no difference in patient satisfaction after an ED visit.²⁹

Postdischarge Readmission

Results emphasized the importance of teach-back in reinforcing discharge instructions and improving postdischarge readmission rates. Of the 6 studies on the effect that teach-back with discharge summary had on readmission rates, 2 found statistically significant improvement for patients with heart failure at 12 months (teach-back, 59%; non-teach-back, 44%; P = .005) and patients with coronary artery bypass grafting (CABG) at 30 days (preintervention, 25%; postintervention, 12%; P = .02).^11,16 In addition, 3 of the 6 studies reported improvement but did not provide P values.^18,20,22 One study indicated improvement in other measured outcomes but found no significant difference for patients who received teach-back with their discharge summaries.²⁷ In all studies, teach-back was added to an intervention and used to confirm and promote knowledge and self-care management.

Patient Perception of Teach-Back Effectiveness

In 2 qualitative studies, patients indicated teach-back was an effective educational method.^16,33 For patients with CABG, Bates and colleagues added a scheduled cardiology follow-up appointment and teach-back patient education to their State Action on Avoidable Rehospitalizations interventions; 96% of participants rated teach-back effective or highly effective.¹⁶ In the other study, Samuels-Kalow and colleagues interviewed 51 patients and parents who received teach-back as part of the discharge process in 2 EDs; participants indicated teach-back helped them remember what they learned from their HCPs, and gave them the opportunity to connect with their HCPs, though some with lower health literacy expressed concerns about perceived judgment by HCPs.³³

 

Disease Knowledge and Management

Thirteen studies examined knowledge improvement after interventions that included teach-back. Study participants answered most questions correctly after receiving teach-back.^20,32,34,35 Slater and colleagues found ED patients who received discharge instructions with teach-back had significantly higher scores measuring knowledge of diagnosis (P < .001), signs and symptoms indicating a need to return to the ED (P < .001), and follow-up instructions (P = .03); scores measuring knowledge of medication were higher as well, but were not statistically different (P = .14).²⁴ In multiple studies, improvement was not always statistically significant in terms of knowledge retention.^{12,25,29-31,36} Studies that compared medication adherence found teach-back was more effective than motivational interviews (P = .56).²⁷

Teach-back has been widely used in primary care, inpatient, and ED settings. Two studies on the effect of teach-back in primary care sampled patients with DM.^28,36 Kandula and colleagues found that participants who answered questions incorrectly after watching a multimedia DM education program could significantly improve their DM knowledge by engaging in teach-back immediately after the intervention; however, knowledge retention was not improved at 2-week follow-up (phone call).²⁸ In contrast, Swavely and colleagues compared patients who completed a 13-hour DM education program with or without teach-back and found that teach-back patients demonstrated significantly improved DM knowledge and self-care activities at 3 months.³⁶

Effects of Interventions on HR-QOL

The teach-back method had been used with QOL improvement programs and other interventions. Ahmadidarrehsima and colleagues incorporated teach-back into their medical self-management program (8 to 11 sessions, each lasting 1.5 to 2 hours) for women with breast cancer and found that the mean happiness score increased to 62.9 from 37.2 (P < .001) in the intervention group, whereas the score for the usual-care group decreased from 41.4 to 29.8.¹³ Ghiasvand and colleagues compared QOL of postpartum mothers who received routine care with QOL of those who received routine care plus 2 sessions of postpartum self-care with teach-back; mean QOL scores were significantly (P < .001) higher for the teach-back group (124.73) than for the no teach-back group (115.03).¹⁴

Discussion

This review examined the use and effectiveness of the teach-back method in health education and its influence in patients’ disease self-management and health outcomes. Results showed positive effects of teach-back on patient satisfaction, patient perceptions and acknowledgments, postdischarge readmissions, disease self-management and knowledge, and HR-QOL.

The teach-back method has been widely used in inpatient, outpatient, ED, and community settings as part of health education programs and interventions. It has been paired with educational interventions ranging from short instructions to 20-hour programs. These differences reflect the broad application of the method in patient education. Many studies have found that teach-back improves disease knowledge and self-management, though their results are not always statistically significant. In an RCT of patients with low health literacy, Griffey and colleagues studied the effect of ED discharge education with and without teach-back and found teach-back did not increase post-ED comprehension of diagnoses, medical examinations, and treatments or perceived comprehension of treatment and care; however, compared with the no teach-back group, the teach-back group had significantly higher scores on comprehension of post-ED self-care (P < .02), follow-up (P < .0001), and medication (P = .054).²⁹ This finding indicates teach-back is an effective method for helping patients understand self-care and disease self-management at home.

Comprehending medical diagnoses, examinations, and treatments involves acquiring, analyzing, and comparing multiple pieces of health information. Because comprehension requires a level of abstract thinking usually present in patients with intermediate and proficient health literacy,improvements might be more difficult to see in patients with low health literacy.⁸ Press and colleagues found that asthma patients who repeated respiratory inhaler instructions with teach-back during discharge education had less misuse of (P = .01) metered-dose and Diskus (P = .05) inhalers and lower 30-day readmission rates (P = .02) compared with the misuse of patients who received only 1 set of oral and written instructions.³¹ Even though the Diskus result was not statistically significant, it demonstrated teach-back can be used to improve patient self-care and education.³¹

Most participants in the reviewed studies improved their disease knowledge with teach-back, though the evidence regarding improved health care knowledge retention was limited. For example, the 2 studies on use of teach-back in primary care clinics had contradictory knowledge retention results.^28,36 As both studies incorporated teach-back into existing interventions, these results could be associated with those interventions and not with the teach-back method.

Health literacy is achieved through a complicated process of obtaining, analyzing, choosing, and communicating health information. Even though its knowledge retention results are inconsistent, the teach-back method is recommended by the American Academy of Family Physicians at strength of recommendation taxonomy level C.⁸ Such a designation indicates that the recommendation is based on expert opinion, bench research, consensus guideline, usual practice, clinical experience, or a case series and is appropriate for assessment of patient comprehension.³⁷ Teach-back is also suggested by AHRQ and IHI for university precautions regarding health literacy and as such should remain a standard of practice. More study is needed to understand the inconsistent results of knowledge retention and the long-term effects of the teach-back method.

 

Limitations

Although this review did not limit the publication years of its articles, no pre-2011 articles were found. The teach-back method has been used to measure patients’ postintervention understanding and to educate HCPs on ways to improve patient communication. As this review did not include studies of teach-back as an outcome measurement or studies of training and adaptation of teach-back in HCP or nurse education, other study results may have a bearing on the current findings. Teach-back has been used to close communication gaps between patients and HCPs.

All articles included in this review used the teach-back method with other educational or organizational interventions. The outcomes found in this review may be associated with those interventions and not with teach-back itself. Data reported here have not demonstrated a definite association between teach-back and the measured outcomes; therefore, caution should be exercised when drawing conclusions based on these data. In addition, most of the studies considered in this review were cohort or case–control studies; only 5 RCTs were included. Other confounding factors, including patient health literacy levels, HCP types, HCP competencies in use of teach-back, and type and duration of interventions used before teach-back, may have contributed to this review’s findings.

Conclusion

Findings of this systematic review support use of the teach-back method as effective in reinforcing or confirming patient education. As none of the included studies reported harmful outcomes, the teach-back method poses little risk with respect to increasing patients’ understanding of their education. The findings emphasize the importance of conducting more studies to try to understand the inconsistent results of knowledge retention and determine ways to preserve the long-term effects of teach-back.

Providing consistent and high-quality services to gender diverse patients is a top priority for health care systems, including the Veterans Health Administration (VHA).¹ Over the past decade, awareness of transgender and gender nonconforming (TGNC) people in the US has increased. Gender identity refers to a person’s inner sense of where that person belongs on a continuum of masculine to androgynous to feminine traits. This identity range can additionally include nonbinary identifications, such as “gender fluid” or “genderqueer.” A goal of patient-centered care is for health care providers (HCPs) to refer to TGNC individuals, like their cisgender counterparts, according to their gender identity. Gender identity for TGNC individuals may be different from their birth sex. Birth sex, commonly referred to as “sex assigned at birth,” is the biologic and physiologic characteristics that are reflected on a person’s original birth certificate and described as male or female.

Background

In the electronic health record (EHR), birth sex is an important, structured variable that is used to facilitate effective patient care that is efficient, equitable, and patient-centered. Birth sex in an EHR often is used to cue automatic timely generation of health screens (eg, pap smears, prostate exams) and calculation of medication dosages and laboratory test ranges by adjusting for a person’s typical hormonal history and anatomy.

Gender identity fields are independently helpful to include in the EHR, because clinicians can use this information to ensure proper pronoun use and avoid misgendering a patient. Additionally, the gender identity field informs HCPs who may conduct more frequent or different health screenings to evaluate specific health risks that are more prevalent in gender minority (ie, lesbian, gay, bisexual) patients.^2,3

EHRs rely on structured data elements to standardize data about patients for clinical care, quality improvement, data sharing, and patient safety.^4,5 However, health care organizations are grappling with how to incorporate gender identity and birth sex information into EHRs.³ A 2011 Veterans Health Administration (VHA) directive required staff and providers to address and provide care to veterans based on their gender identity. Like other health systems, VHA had 1 demographic data field in the EHR to indicate birth sex, with no field for gender identity. A HCP could enter gender identity information into a progress note, but this addition might not be noticed by other HCPs. Consequently, staff and providers had no effective way of knowing a veteran’s gender identity from the EHR, which contributed to misgendering TGNC veterans.

With the singular demographic field of sex representing both birth sex and gender identity, some TGNC veterans chose to change their birth sex information to align with their gender identity. This change assured TGNC veterans that staff and providers would not misgender them because the birth sex field is easily observed and would allow providers to use respectful, gender-consistent pronouns when speaking with them. However, changing the birth sex field can misalign natal sex–based clinical reminders, medication dosages, and laboratory test values, which created potential patient safety risks. Thus, birth sex created potential hazards to quality and safety when used as a marker even with other variables—such as current anatomy, height, and weight—for health screenings, medication dosing, and other medical decisions.

In this article, we: (1) outline several patient safety issues that can arise with the birth sex field serving as an indicator for both birth sex and gender identity; (2) present case examples that illustrate the benefits of self-identified gender identity (SIGI) in an EHR; (3) describe the process of work-group development of patient-provider communication tools to improve patient safety; and (4) provide a brief overview of resources rolled out as a part of SIGI. This report serves as a guide for other federal organizations that wish to increase affirmative care and safe practices for transgender consumers. We will provide an overview of the tasks leading up to SIGI implementation, deliverables from the project, and lessons learned.

 

Veterans Affairs SIGI EHR Field

In 2016, the US Department of Veterans Affairs (VA) began implementing a SIGI demographic field across all EHRs, requiring administrative staff to ask enrolled and new veterans their gender identity (full implementation of SIGI has not yet occurred and will occur when a later EHR upgrade displays SIGI in the EHR). The initiation of SIGI did not change any information in the birth sex field, meaning that some veterans continue to have birth sex field information that results in problematic automatic medical reminders and dosing values. Consequently, the National Center for Patient Safety (NCPS) noted that this discrepancy may be a pertinent patient safety issue. The NCPS and Lesbian, Gay Bisexual, and Transgender (LGBT) Health national program offices worked to provide documentation to TGNC veterans to inform them of the clinical health care implications of having their birth sex demographic field reflect gender identity that is inconsistent with their natal sex (ie, original birth certificate record of sex).

Patient Safety Issues

Conversations between transgender patients and their HCPs about transition goals, necessary medical tests, and laboratory ranges based on their current anatomy and physiology can improve patient safety and satisfaction with medical care. Prior to the availability of the SIGI field, VA facilities varied in their documentation of gender identity in the patient chart. LGBT veteran care coordinators discussed diverse suggestions that ranged from informally documenting SIGI in each progress note to using flags to draw attention to use certain sections of local EHRs. These suggestions, though well intentioned, were not adequate for documenting gender identity at the national level because of regional variations in EHR customization options. Furthermore, the use of flags for drawing clinical attention to gender identity posed a potential for stigma toward patients, given that flags are typically reserved for behavioral or other risk concerns.

Several problems can emerge when HCPs are not equipped with accurate information about patient birth sex and SIGI. For instance, TGNC patients lack a way of being known from clinic to clinic by proper pronouns or self-labels. Providers may misgender veterans, which is a negative experience for TGNC veterans linked with increased barriers to care and decreased frequency of health care visits.⁴ Moreover, the quality and personalization of care across clinic locations in the facility’s system is variable without a consistent method of documenting birth sex and SIGI. For example, in clinics where the veteran is well known (eg, primary care), staff may be more affirming of the veteran’s gender identity than those in specialty care clinics that lack prior patient contact.

Furthermore, depending on hormone and surgical interventions, some health screenings may be irrelevant for TGNC patients. To determine appropriate health screens and assess potential risks associated with hormone therapy, providers must have access to current information regarding a patient’s physiologic anatomy.⁶ Health screenings and laboratory results in sophisticated EHRs (ie, EHRs that might autodetermine normative values) may populate incorrect treatment recommendations, such as sex-based medication dosages. Furthermore, laboratory test results could be incorrectly paired with a different assumed hormonal history, potentially putting the patient at risk.

 

Case Examples

An important element of EHRs facilitating the goal of patient-centered care is that patients have their EHR validate their sense of self, and their providers can use names and pronouns that correspond to the patient’s SIGI. Some patients have spent a great amount of effort altering their name and sex in legal records and may want their birth sex field to conform to their gender identity. To that end, patients may seek to alter their birth sex information so that it is congruent with how they see themselves to affirm their identity, despite patient safety risks. Several scenarios below demonstrate the potential costs and benefits to patients altering birth sex and SIGI in the EHR.

Case 1 Presentation

A young transman is working with his therapist on engaging in self-validating behaviors. This veteran has met with his PCP and informed the provider of his decision to alter the birth sex field in his EHR from female to male.

Ideally, the patient would begin to have regular conversations with his HCPs about his birth sex and gender identity, so that medical professionals can provide relevant screenings and affirm the patient’s gender identity while acknowledging his right to list his birth sex as he chooses. However, particular attention will need to be paid to assuring that natal sex–based health screenings (eg, pap smears, mammograms) are conducted on an appropriate schedule and that the veteran continues to discuss his current anatomy with providers.

Case 2 Presentation

A veteran has a male birth sex, identifies as a transwoman, and uses nongendered plural pronouns “they/them/theirs.” The word “they,” used as a singular pronoun may feel uncomfortable to some providers, but it validates the veteran’s sense of self and helps them feel welcome in the treatment environment. This patient communicated proactively with their HCPs about their transition goals and current hormone use.

They opted to have their birth sex field continue to indicate “male” because they, after a discussion with their PCP, are aware of the health implications of receiving an incorrect dose for their diabetes medication. They understand that having open communication and receiving input from their HCPs is part of good health care.

Case 3 Presentation

A patient with a sexual development disorder (intersex condition) identifies as a man (indicated as “male” in the SIGI field) and had his birth sex field changed to match his gender identity. He now seeks to change his birth sex field back to female, as he has complicated health considerations due to breast cancer.

The veteran thinks it is important that providers know about his intersex condition so that his breast cancer care is as seamless as possible. In particular, although this veteran is comfortable talking about his intersex condition and his identity with his PCP and oncologist, he wants to ensure that all people involved in his care (eg, pharmacists, radiologists) use the correct values in interpreting his medical data. Providers will need to use the female birth sex field for interpreting his medical data but use male pronouns when interacting with the veteran and documenting his care.

These case examples illustrate the need for HCPs to have patient-affirming education and appropriate clinical tools available when speaking to patients about birth sex, SIGI, and the implications of changing birth sex in the EHR. Moreover, these cases highlight that patient health needs may vary over time, due to factors such as perceived costs/benefits of a change in the sex field of the EHR as well as patient comfort with providers.

 

Current Status of SIGI and EHR

Although having separate fields for birth sex and SIGI in the EHR is ideal, the VHA does not yet have a fully functional SIGI field, and several TGNC veterans have changed their birth sex field to align with their gender identity. Roughly 9,700 patients have diagnostic codes related to transgender care in the VHA, meaning thousands of current patients would potentially benefit from SIGI implementation (John Blosnich, written communication, March 2018). A possible action that the VHA could take with the goal of enhancing patient safety would be to revert the birth sex field of patients who had previously changed the field back to the patient’s original birth sex. However, if this alteration to the EHR were done without the patient’s consent, numerous additional problems would result—including invalidating a veteran’s wishes—potentially driving patients away from receiving health care.

Moreover, in the absence of updated SIGI information (which only the veteran can provide), making a change in the EHR would perpetuate the misgendering of TGNC veterans who have already sought an administrative fix for this problem. Thus, the agency decided to engage patients in a discussion about their decision to keep the birth sex field consistent with their original birth certificate. In cases in which the field had been changed previously, the recommendation is for HCPs to gain patient consent to change the birth sex field back to what was on their original birth certificate. Thus, decisions about what should be listed in the EHR are made by the veteran using an informed decision-making model.

Patient Safety Education Workgroup

To begin the process of disentangling birth sex and SIGI fields in the EHR, 2 work groups were created: a technical work group (coding the patches for SIGI implementation) and a SIGI patient safety education work group. The patient safety education work group was committed to promoting affirmative VA policies that require validation of the gender identity of all veterans and pursuing best practices through clinical guidelines to promote effective, efficient, equitable, and safe veteran care. The patient safety education work group included representatives from all 3 branches of the VA (VHA, Veterans Benefits Administration, and National Cemetery Administration), including clinical media, patient safety, information technology, and education specialists. The group developed trainings for administrative staff about the appropriate ways to ask birth sex and SIGI questions, and how to record veteran-driven responses.

SIGI Fact Sheet

The patient safety education work group examined clinical literature and developed tools for staff and veterans to facilitate effective discussions about the importance and utility of documenting both birth sex and SIGI in the EHR. The patient safety education work group along with media and educational experts created basic key term definition documents to address the importance, purpose, and use of the SIGI field. The patient safety education work group developed 2 documents to facilitate communication between patients and providers.

 

A 1-page veteran-facing fact sheet was developed that described the differences between birth sex and SIGI fields and how these fields are used in the VA EHR system (Figure 1). In addition, a 1-page HCP-facing fact sheet was designed to inform HCPs that patients may have changed their birth sex in their EHR or might still wish to change their birth sex field, and to inform HCPs of the importance of patient-centered, gender-affirmative care (Figure 2). An additional goal of both documents was to educate veterans and HCPs on how the EHR automatically calculates laboratory results and screening notifications based on birth sex.

 

Review Process

As part of reviewing and finalizing the SIGI patient fact sheet, the patient safety education work group previewed the document’s content with veterans who provided feedback on drafts to improve comprehension, patient-centered care, and clinical accuracy. For instance, several patients commented that the document should address many gender identities, including intersex identities. As noted in one of the case presentations earlier, individuals who identify as intersex may have changed their birth sex to be consistent with their gender and might benefit from being informed about the EHR’s autocalculation feature. The patient safety education work group adjusted the SIGI patient fact sheet to include individuals who identify as intersex and instructed them to have a conversation with their HCP regarding potential birth sex changes in the EHR.

Much of the veteran feedback to the patient safety education work group reflected veteran concerns, more broadly, about implementation of SIGI. Many veterans were interested in how federal policy changes might affect their benefits package or clinical care within the VA. The SIGI patient fact sheet was a tool for communicating that Department of Defense (DoD) policies, specifically, do not have a bearing on VA care for LGBT veterans. Therefore, SIGI information does not affect service connection or benefits eligibility and is not shared with the DoD. Veterans found this information helpful to see reflected in the SIGI patient fact sheet.

The patient safety education work group also shared the SIGI provider fact sheet with VHA providers before finalizing the content. PCPs gave feedback to improve the specification of patient safety concerns and appropriate readership language. The patient safety education work group adjusted the SIGI provider fact sheet to be inclusive of relevant literature and an e-consultation link for assisting HCPs who are unsure how to proceed with a patient.

Implementation

The patient safety education work group also developed several materials to provide information about the birth sex and SIGI fields in the EHR. Because the SIGI demographic field is new and collected by clerical staff, training was necessary to explain the difference between birth sex and SIGI before implementation in the EHR. The training sessions educated staff about the difference between birth sex and SIGI, how to ask and respond to questions respectfully, and how to update these fields in the EHR. These trainings included a 20-minute video demonstrating best practices for asking about SIGI, a frequently asked questions document responding to 7 common questions about the new fields, and a quick reference guide for administrative staff to have handy at their desks.

Dissemination of the SIGI patient and provider fact sheets is planned to occur, ideally, several weeks before implementation of the new patches updating the EHR fields in spring 2020. Building on existing resources, the patient safety education work group plans to disseminate the patient fact sheets via e-mail lists for the national mental health facility leaders as well as through e-mail lists for VA PCPs, nursing and clerical staff, privacy officers, facility LGBT veteran care coordinators, VISN leads, transgender e-consultation, the Office of Connected Care, the LGBT external homepage for the VA, and the training website for VA employees. The goal is to target potential points of contact for veterans who may have already changed their birth sex and might benefit medically from altering birth sex to be consistent with their original birth certificate.

The SIGI provider fact sheet will be disseminated using internal e-mails, announcements on routine LGBT veteran care coordinator calls, weekly Ask LGBT Health teleconferences, and announcements at LGBT health training events both internally and externally. Several dissemination tools have already ensured that VA employees are aware of the SIGI field in the EHR. Leadership throughout the VA will be encouraged to share SIGI trainings with clerical staff. Additionally, broad-based e-mails summarizing changes to the EHR will be provided concurrent to the SIGI patch implementation to VA staff as well as links to the resources and training materials.

 

Challenges

One difficulty in the development process for both SIGI fact sheets was addressing the issue of patient safety for veterans who may be at different points in their gender transition process. It was challenging for the patient safety education work group to not sound alarmist in discussing the safety implications of birth sex changes in the EHR, as this is just one factor in clinical decision making. The goal was to educate veterans from a patient safety perspective about the implications of having a state-of-the-art, automated EHR. However, text can be perceived differently by different people, which is why the patient safety education work group asked veterans to preview the patient document and clinical providers to preview the provider document.

Both work groups encountered technologic challenges, including a delay in the implementation of the SIGI field due to a systemwide delay of EHR updates. Although it released training and educational materials to the VHA, the patient safety education work group understood that at some point in the future, VA programmers will update the EHR to change the information clerks and HCPs can see in the EHR. Coordination of the fact sheet release alongside information technology has been an important part of the SIGI rollout process.

Conclusion

HCPs have a complex role in providing treatment to TGNC patients in the VHA: They must affirm a patient’s gender identity through how they address them, while openly communicating the health risks inherent in having their birth sex field be incongruent with the sex recorded on their original birth certificate. Accomplishing these tasks simultaneously is difficult without invalidating the veteran’s identity or right to choose their EHR demographic birth sex label. Furthermore, patients may ask HCPs to write letters of support for either medical or surgical intervention or other documentation changes (eg, changes to a patient’s legal name, passport changes, or a safe passage letter for TGNC patients). Navigating the dialectic of safety and validation requires strong rapport, trust, and effective communication in the patient-provider relationship and great empathy by the provider.

A future task for the SIGI patient safety education work group is to continue to communicate with the technical work group and providers in the field about how demographic fields in the EHR are utilized to enable future EHR changes. This hurdle is not easy because EHR updates change the infrastructure through which demographic content is delivered and incorporated into a patient’s treatment. The VA HCPs are tasked with thoroughly examining the results that automated systems produce to ensure safe and accurate medical services are always provided to all patients. An integral part of patient-centered care is balancing any computer-guided recommendations with an understanding that actual patient needs may differ due to presence/absence of anatomy and other factors (eg, weight, current medications).

From a systems perspective, a benefit of adding the SIGI demographic field is systemic improvement in calculating the number of transgender veterans under VA care and evaluating health outcomes for this population. SIGI is particularly important for signaling gender pronouns for veterans, regardless of whether they are receiving care for a gender-related diagnosis. In terms of scope, the SIGI project potentially will apply to > 9 million enrolled veterans and nearly 400,000 VA employees.

Improvements could be made in the SIGI field of the new EHR, such as expanding the options for self-labels. Additionally, a text field could be used to enhance the quality of personalization provided to veterans self-identifying in the EHR, including pronoun specification. Moreover, adding new fields such as “preferred name” could improve the health care experience of not only TGNC veterans but all veterans who use something other than their full legal name (eg, a nickname). It will be good practice to notify providers and staff of a veteran’s requested name and pronouns when the patient checks in at an electronic kiosk so that all staff immediately know how to address the patient. The VHA can continue to adjust the options for the SIGI field once the new EHR system is operational. Ideally, this new EHR will display birth sex and SIGI to clinicians or clerks engaged in patient interactions.

Technology will continue to automate medical care, meaning that HCPs must be vigilant about how computer programming and the accuracy of prepopulated information affect patient care. The concerns discussed in this report relating to patient safety are relatively absent in the medical literature, even though substantial health risks exist to patients who have birth sex listed incorrectly for any reason.^6,7 Additionally, administrative burden can be reduced if patients who do not need certain screenings based on their current anatomy are not contacted for unnecessary screenings. Future EHR systems might incorporate anatomical considerations from an inventory to assist in automating patient care in safe and accessible ways.

Providing consistent and high-quality services to gender diverse patients is a top priority for health care systems, including the Veterans Health Administration (VHA).¹ Over the past decade, awareness of transgender and gender nonconforming (TGNC) people in the US has increased. Gender identity refers to a person’s inner sense of where that person belongs on a continuum of masculine to androgynous to feminine traits. This identity range can additionally include nonbinary identifications, such as “gender fluid” or “genderqueer.” A goal of patient-centered care is for health care providers (HCPs) to refer to TGNC individuals, like their cisgender counterparts, according to their gender identity. Gender identity for TGNC individuals may be different from their birth sex. Birth sex, commonly referred to as “sex assigned at birth,” is the biologic and physiologic characteristics that are reflected on a person’s original birth certificate and described as male or female.

Background

In the electronic health record (EHR), birth sex is an important, structured variable that is used to facilitate effective patient care that is efficient, equitable, and patient-centered. Birth sex in an EHR often is used to cue automatic timely generation of health screens (eg, pap smears, prostate exams) and calculation of medication dosages and laboratory test ranges by adjusting for a person’s typical hormonal history and anatomy.

Gender identity fields are independently helpful to include in the EHR, because clinicians can use this information to ensure proper pronoun use and avoid misgendering a patient. Additionally, the gender identity field informs HCPs who may conduct more frequent or different health screenings to evaluate specific health risks that are more prevalent in gender minority (ie, lesbian, gay, bisexual) patients.^2,3

EHRs rely on structured data elements to standardize data about patients for clinical care, quality improvement, data sharing, and patient safety.^4,5 However, health care organizations are grappling with how to incorporate gender identity and birth sex information into EHRs.³ A 2011 Veterans Health Administration (VHA) directive required staff and providers to address and provide care to veterans based on their gender identity. Like other health systems, VHA had 1 demographic data field in the EHR to indicate birth sex, with no field for gender identity. A HCP could enter gender identity information into a progress note, but this addition might not be noticed by other HCPs. Consequently, staff and providers had no effective way of knowing a veteran’s gender identity from the EHR, which contributed to misgendering TGNC veterans.

With the singular demographic field of sex representing both birth sex and gender identity, some TGNC veterans chose to change their birth sex information to align with their gender identity. This change assured TGNC veterans that staff and providers would not misgender them because the birth sex field is easily observed and would allow providers to use respectful, gender-consistent pronouns when speaking with them. However, changing the birth sex field can misalign natal sex–based clinical reminders, medication dosages, and laboratory test values, which created potential patient safety risks. Thus, birth sex created potential hazards to quality and safety when used as a marker even with other variables—such as current anatomy, height, and weight—for health screenings, medication dosing, and other medical decisions.

In this article, we: (1) outline several patient safety issues that can arise with the birth sex field serving as an indicator for both birth sex and gender identity; (2) present case examples that illustrate the benefits of self-identified gender identity (SIGI) in an EHR; (3) describe the process of work-group development of patient-provider communication tools to improve patient safety; and (4) provide a brief overview of resources rolled out as a part of SIGI. This report serves as a guide for other federal organizations that wish to increase affirmative care and safe practices for transgender consumers. We will provide an overview of the tasks leading up to SIGI implementation, deliverables from the project, and lessons learned.

 

Veterans Affairs SIGI EHR Field

In 2016, the US Department of Veterans Affairs (VA) began implementing a SIGI demographic field across all EHRs, requiring administrative staff to ask enrolled and new veterans their gender identity (full implementation of SIGI has not yet occurred and will occur when a later EHR upgrade displays SIGI in the EHR). The initiation of SIGI did not change any information in the birth sex field, meaning that some veterans continue to have birth sex field information that results in problematic automatic medical reminders and dosing values. Consequently, the National Center for Patient Safety (NCPS) noted that this discrepancy may be a pertinent patient safety issue. The NCPS and Lesbian, Gay Bisexual, and Transgender (LGBT) Health national program offices worked to provide documentation to TGNC veterans to inform them of the clinical health care implications of having their birth sex demographic field reflect gender identity that is inconsistent with their natal sex (ie, original birth certificate record of sex).

Patient Safety Issues

Conversations between transgender patients and their HCPs about transition goals, necessary medical tests, and laboratory ranges based on their current anatomy and physiology can improve patient safety and satisfaction with medical care. Prior to the availability of the SIGI field, VA facilities varied in their documentation of gender identity in the patient chart. LGBT veteran care coordinators discussed diverse suggestions that ranged from informally documenting SIGI in each progress note to using flags to draw attention to use certain sections of local EHRs. These suggestions, though well intentioned, were not adequate for documenting gender identity at the national level because of regional variations in EHR customization options. Furthermore, the use of flags for drawing clinical attention to gender identity posed a potential for stigma toward patients, given that flags are typically reserved for behavioral or other risk concerns.

Several problems can emerge when HCPs are not equipped with accurate information about patient birth sex and SIGI. For instance, TGNC patients lack a way of being known from clinic to clinic by proper pronouns or self-labels. Providers may misgender veterans, which is a negative experience for TGNC veterans linked with increased barriers to care and decreased frequency of health care visits.⁴ Moreover, the quality and personalization of care across clinic locations in the facility’s system is variable without a consistent method of documenting birth sex and SIGI. For example, in clinics where the veteran is well known (eg, primary care), staff may be more affirming of the veteran’s gender identity than those in specialty care clinics that lack prior patient contact.

Furthermore, depending on hormone and surgical interventions, some health screenings may be irrelevant for TGNC patients. To determine appropriate health screens and assess potential risks associated with hormone therapy, providers must have access to current information regarding a patient’s physiologic anatomy.⁶ Health screenings and laboratory results in sophisticated EHRs (ie, EHRs that might autodetermine normative values) may populate incorrect treatment recommendations, such as sex-based medication dosages. Furthermore, laboratory test results could be incorrectly paired with a different assumed hormonal history, potentially putting the patient at risk.

 

Case Examples

An important element of EHRs facilitating the goal of patient-centered care is that patients have their EHR validate their sense of self, and their providers can use names and pronouns that correspond to the patient’s SIGI. Some patients have spent a great amount of effort altering their name and sex in legal records and may want their birth sex field to conform to their gender identity. To that end, patients may seek to alter their birth sex information so that it is congruent with how they see themselves to affirm their identity, despite patient safety risks. Several scenarios below demonstrate the potential costs and benefits to patients altering birth sex and SIGI in the EHR.

Case 1 Presentation

A young transman is working with his therapist on engaging in self-validating behaviors. This veteran has met with his PCP and informed the provider of his decision to alter the birth sex field in his EHR from female to male.

Ideally, the patient would begin to have regular conversations with his HCPs about his birth sex and gender identity, so that medical professionals can provide relevant screenings and affirm the patient’s gender identity while acknowledging his right to list his birth sex as he chooses. However, particular attention will need to be paid to assuring that natal sex–based health screenings (eg, pap smears, mammograms) are conducted on an appropriate schedule and that the veteran continues to discuss his current anatomy with providers.

Case 2 Presentation

A veteran has a male birth sex, identifies as a transwoman, and uses nongendered plural pronouns “they/them/theirs.” The word “they,” used as a singular pronoun may feel uncomfortable to some providers, but it validates the veteran’s sense of self and helps them feel welcome in the treatment environment. This patient communicated proactively with their HCPs about their transition goals and current hormone use.

They opted to have their birth sex field continue to indicate “male” because they, after a discussion with their PCP, are aware of the health implications of receiving an incorrect dose for their diabetes medication. They understand that having open communication and receiving input from their HCPs is part of good health care.

Case 3 Presentation

A patient with a sexual development disorder (intersex condition) identifies as a man (indicated as “male” in the SIGI field) and had his birth sex field changed to match his gender identity. He now seeks to change his birth sex field back to female, as he has complicated health considerations due to breast cancer.

The veteran thinks it is important that providers know about his intersex condition so that his breast cancer care is as seamless as possible. In particular, although this veteran is comfortable talking about his intersex condition and his identity with his PCP and oncologist, he wants to ensure that all people involved in his care (eg, pharmacists, radiologists) use the correct values in interpreting his medical data. Providers will need to use the female birth sex field for interpreting his medical data but use male pronouns when interacting with the veteran and documenting his care.

These case examples illustrate the need for HCPs to have patient-affirming education and appropriate clinical tools available when speaking to patients about birth sex, SIGI, and the implications of changing birth sex in the EHR. Moreover, these cases highlight that patient health needs may vary over time, due to factors such as perceived costs/benefits of a change in the sex field of the EHR as well as patient comfort with providers.

 

Current Status of SIGI and EHR

Although having separate fields for birth sex and SIGI in the EHR is ideal, the VHA does not yet have a fully functional SIGI field, and several TGNC veterans have changed their birth sex field to align with their gender identity. Roughly 9,700 patients have diagnostic codes related to transgender care in the VHA, meaning thousands of current patients would potentially benefit from SIGI implementation (John Blosnich, written communication, March 2018). A possible action that the VHA could take with the goal of enhancing patient safety would be to revert the birth sex field of patients who had previously changed the field back to the patient’s original birth sex. However, if this alteration to the EHR were done without the patient’s consent, numerous additional problems would result—including invalidating a veteran’s wishes—potentially driving patients away from receiving health care.

Moreover, in the absence of updated SIGI information (which only the veteran can provide), making a change in the EHR would perpetuate the misgendering of TGNC veterans who have already sought an administrative fix for this problem. Thus, the agency decided to engage patients in a discussion about their decision to keep the birth sex field consistent with their original birth certificate. In cases in which the field had been changed previously, the recommendation is for HCPs to gain patient consent to change the birth sex field back to what was on their original birth certificate. Thus, decisions about what should be listed in the EHR are made by the veteran using an informed decision-making model.

Patient Safety Education Workgroup

To begin the process of disentangling birth sex and SIGI fields in the EHR, 2 work groups were created: a technical work group (coding the patches for SIGI implementation) and a SIGI patient safety education work group. The patient safety education work group was committed to promoting affirmative VA policies that require validation of the gender identity of all veterans and pursuing best practices through clinical guidelines to promote effective, efficient, equitable, and safe veteran care. The patient safety education work group included representatives from all 3 branches of the VA (VHA, Veterans Benefits Administration, and National Cemetery Administration), including clinical media, patient safety, information technology, and education specialists. The group developed trainings for administrative staff about the appropriate ways to ask birth sex and SIGI questions, and how to record veteran-driven responses.

SIGI Fact Sheet

The patient safety education work group examined clinical literature and developed tools for staff and veterans to facilitate effective discussions about the importance and utility of documenting both birth sex and SIGI in the EHR. The patient safety education work group along with media and educational experts created basic key term definition documents to address the importance, purpose, and use of the SIGI field. The patient safety education work group developed 2 documents to facilitate communication between patients and providers.

 

A 1-page veteran-facing fact sheet was developed that described the differences between birth sex and SIGI fields and how these fields are used in the VA EHR system (Figure 1). In addition, a 1-page HCP-facing fact sheet was designed to inform HCPs that patients may have changed their birth sex in their EHR or might still wish to change their birth sex field, and to inform HCPs of the importance of patient-centered, gender-affirmative care (Figure 2). An additional goal of both documents was to educate veterans and HCPs on how the EHR automatically calculates laboratory results and screening notifications based on birth sex.

 

Review Process

As part of reviewing and finalizing the SIGI patient fact sheet, the patient safety education work group previewed the document’s content with veterans who provided feedback on drafts to improve comprehension, patient-centered care, and clinical accuracy. For instance, several patients commented that the document should address many gender identities, including intersex identities. As noted in one of the case presentations earlier, individuals who identify as intersex may have changed their birth sex to be consistent with their gender and might benefit from being informed about the EHR’s autocalculation feature. The patient safety education work group adjusted the SIGI patient fact sheet to include individuals who identify as intersex and instructed them to have a conversation with their HCP regarding potential birth sex changes in the EHR.

Much of the veteran feedback to the patient safety education work group reflected veteran concerns, more broadly, about implementation of SIGI. Many veterans were interested in how federal policy changes might affect their benefits package or clinical care within the VA. The SIGI patient fact sheet was a tool for communicating that Department of Defense (DoD) policies, specifically, do not have a bearing on VA care for LGBT veterans. Therefore, SIGI information does not affect service connection or benefits eligibility and is not shared with the DoD. Veterans found this information helpful to see reflected in the SIGI patient fact sheet.

The patient safety education work group also shared the SIGI provider fact sheet with VHA providers before finalizing the content. PCPs gave feedback to improve the specification of patient safety concerns and appropriate readership language. The patient safety education work group adjusted the SIGI provider fact sheet to be inclusive of relevant literature and an e-consultation link for assisting HCPs who are unsure how to proceed with a patient.

Implementation

The patient safety education work group also developed several materials to provide information about the birth sex and SIGI fields in the EHR. Because the SIGI demographic field is new and collected by clerical staff, training was necessary to explain the difference between birth sex and SIGI before implementation in the EHR. The training sessions educated staff about the difference between birth sex and SIGI, how to ask and respond to questions respectfully, and how to update these fields in the EHR. These trainings included a 20-minute video demonstrating best practices for asking about SIGI, a frequently asked questions document responding to 7 common questions about the new fields, and a quick reference guide for administrative staff to have handy at their desks.

Dissemination of the SIGI patient and provider fact sheets is planned to occur, ideally, several weeks before implementation of the new patches updating the EHR fields in spring 2020. Building on existing resources, the patient safety education work group plans to disseminate the patient fact sheets via e-mail lists for the national mental health facility leaders as well as through e-mail lists for VA PCPs, nursing and clerical staff, privacy officers, facility LGBT veteran care coordinators, VISN leads, transgender e-consultation, the Office of Connected Care, the LGBT external homepage for the VA, and the training website for VA employees. The goal is to target potential points of contact for veterans who may have already changed their birth sex and might benefit medically from altering birth sex to be consistent with their original birth certificate.

The SIGI provider fact sheet will be disseminated using internal e-mails, announcements on routine LGBT veteran care coordinator calls, weekly Ask LGBT Health teleconferences, and announcements at LGBT health training events both internally and externally. Several dissemination tools have already ensured that VA employees are aware of the SIGI field in the EHR. Leadership throughout the VA will be encouraged to share SIGI trainings with clerical staff. Additionally, broad-based e-mails summarizing changes to the EHR will be provided concurrent to the SIGI patch implementation to VA staff as well as links to the resources and training materials.

 

Challenges

One difficulty in the development process for both SIGI fact sheets was addressing the issue of patient safety for veterans who may be at different points in their gender transition process. It was challenging for the patient safety education work group to not sound alarmist in discussing the safety implications of birth sex changes in the EHR, as this is just one factor in clinical decision making. The goal was to educate veterans from a patient safety perspective about the implications of having a state-of-the-art, automated EHR. However, text can be perceived differently by different people, which is why the patient safety education work group asked veterans to preview the patient document and clinical providers to preview the provider document.

Both work groups encountered technologic challenges, including a delay in the implementation of the SIGI field due to a systemwide delay of EHR updates. Although it released training and educational materials to the VHA, the patient safety education work group understood that at some point in the future, VA programmers will update the EHR to change the information clerks and HCPs can see in the EHR. Coordination of the fact sheet release alongside information technology has been an important part of the SIGI rollout process.

Conclusion

HCPs have a complex role in providing treatment to TGNC patients in the VHA: They must affirm a patient’s gender identity through how they address them, while openly communicating the health risks inherent in having their birth sex field be incongruent with the sex recorded on their original birth certificate. Accomplishing these tasks simultaneously is difficult without invalidating the veteran’s identity or right to choose their EHR demographic birth sex label. Furthermore, patients may ask HCPs to write letters of support for either medical or surgical intervention or other documentation changes (eg, changes to a patient’s legal name, passport changes, or a safe passage letter for TGNC patients). Navigating the dialectic of safety and validation requires strong rapport, trust, and effective communication in the patient-provider relationship and great empathy by the provider.

A future task for the SIGI patient safety education work group is to continue to communicate with the technical work group and providers in the field about how demographic fields in the EHR are utilized to enable future EHR changes. This hurdle is not easy because EHR updates change the infrastructure through which demographic content is delivered and incorporated into a patient’s treatment. The VA HCPs are tasked with thoroughly examining the results that automated systems produce to ensure safe and accurate medical services are always provided to all patients. An integral part of patient-centered care is balancing any computer-guided recommendations with an understanding that actual patient needs may differ due to presence/absence of anatomy and other factors (eg, weight, current medications).

From a systems perspective, a benefit of adding the SIGI demographic field is systemic improvement in calculating the number of transgender veterans under VA care and evaluating health outcomes for this population. SIGI is particularly important for signaling gender pronouns for veterans, regardless of whether they are receiving care for a gender-related diagnosis. In terms of scope, the SIGI project potentially will apply to > 9 million enrolled veterans and nearly 400,000 VA employees.

Improvements could be made in the SIGI field of the new EHR, such as expanding the options for self-labels. Additionally, a text field could be used to enhance the quality of personalization provided to veterans self-identifying in the EHR, including pronoun specification. Moreover, adding new fields such as “preferred name” could improve the health care experience of not only TGNC veterans but all veterans who use something other than their full legal name (eg, a nickname). It will be good practice to notify providers and staff of a veteran’s requested name and pronouns when the patient checks in at an electronic kiosk so that all staff immediately know how to address the patient. The VHA can continue to adjust the options for the SIGI field once the new EHR system is operational. Ideally, this new EHR will display birth sex and SIGI to clinicians or clerks engaged in patient interactions.

Technology will continue to automate medical care, meaning that HCPs must be vigilant about how computer programming and the accuracy of prepopulated information affect patient care. The concerns discussed in this report relating to patient safety are relatively absent in the medical literature, even though substantial health risks exist to patients who have birth sex listed incorrectly for any reason.^6,7 Additionally, administrative burden can be reduced if patients who do not need certain screenings based on their current anatomy are not contacted for unnecessary screenings. Future EHR systems might incorporate anatomical considerations from an inventory to assist in automating patient care in safe and accessible ways.

Providing consistent and high-quality services to gender diverse patients is a top priority for health care systems, including the Veterans Health Administration (VHA).¹ Over the past decade, awareness of transgender and gender nonconforming (TGNC) people in the US has increased. Gender identity refers to a person’s inner sense of where that person belongs on a continuum of masculine to androgynous to feminine traits. This identity range can additionally include nonbinary identifications, such as “gender fluid” or “genderqueer.” A goal of patient-centered care is for health care providers (HCPs) to refer to TGNC individuals, like their cisgender counterparts, according to their gender identity. Gender identity for TGNC individuals may be different from their birth sex. Birth sex, commonly referred to as “sex assigned at birth,” is the biologic and physiologic characteristics that are reflected on a person’s original birth certificate and described as male or female.

Background

In the electronic health record (EHR), birth sex is an important, structured variable that is used to facilitate effective patient care that is efficient, equitable, and patient-centered. Birth sex in an EHR often is used to cue automatic timely generation of health screens (eg, pap smears, prostate exams) and calculation of medication dosages and laboratory test ranges by adjusting for a person’s typical hormonal history and anatomy.

Gender identity fields are independently helpful to include in the EHR, because clinicians can use this information to ensure proper pronoun use and avoid misgendering a patient. Additionally, the gender identity field informs HCPs who may conduct more frequent or different health screenings to evaluate specific health risks that are more prevalent in gender minority (ie, lesbian, gay, bisexual) patients.^2,3

EHRs rely on structured data elements to standardize data about patients for clinical care, quality improvement, data sharing, and patient safety.^4,5 However, health care organizations are grappling with how to incorporate gender identity and birth sex information into EHRs.³ A 2011 Veterans Health Administration (VHA) directive required staff and providers to address and provide care to veterans based on their gender identity. Like other health systems, VHA had 1 demographic data field in the EHR to indicate birth sex, with no field for gender identity. A HCP could enter gender identity information into a progress note, but this addition might not be noticed by other HCPs. Consequently, staff and providers had no effective way of knowing a veteran’s gender identity from the EHR, which contributed to misgendering TGNC veterans.

With the singular demographic field of sex representing both birth sex and gender identity, some TGNC veterans chose to change their birth sex information to align with their gender identity. This change assured TGNC veterans that staff and providers would not misgender them because the birth sex field is easily observed and would allow providers to use respectful, gender-consistent pronouns when speaking with them. However, changing the birth sex field can misalign natal sex–based clinical reminders, medication dosages, and laboratory test values, which created potential patient safety risks. Thus, birth sex created potential hazards to quality and safety when used as a marker even with other variables—such as current anatomy, height, and weight—for health screenings, medication dosing, and other medical decisions.

In this article, we: (1) outline several patient safety issues that can arise with the birth sex field serving as an indicator for both birth sex and gender identity; (2) present case examples that illustrate the benefits of self-identified gender identity (SIGI) in an EHR; (3) describe the process of work-group development of patient-provider communication tools to improve patient safety; and (4) provide a brief overview of resources rolled out as a part of SIGI. This report serves as a guide for other federal organizations that wish to increase affirmative care and safe practices for transgender consumers. We will provide an overview of the tasks leading up to SIGI implementation, deliverables from the project, and lessons learned.

 

Veterans Affairs SIGI EHR Field

In 2016, the US Department of Veterans Affairs (VA) began implementing a SIGI demographic field across all EHRs, requiring administrative staff to ask enrolled and new veterans their gender identity (full implementation of SIGI has not yet occurred and will occur when a later EHR upgrade displays SIGI in the EHR). The initiation of SIGI did not change any information in the birth sex field, meaning that some veterans continue to have birth sex field information that results in problematic automatic medical reminders and dosing values. Consequently, the National Center for Patient Safety (NCPS) noted that this discrepancy may be a pertinent patient safety issue. The NCPS and Lesbian, Gay Bisexual, and Transgender (LGBT) Health national program offices worked to provide documentation to TGNC veterans to inform them of the clinical health care implications of having their birth sex demographic field reflect gender identity that is inconsistent with their natal sex (ie, original birth certificate record of sex).

Patient Safety Issues

Conversations between transgender patients and their HCPs about transition goals, necessary medical tests, and laboratory ranges based on their current anatomy and physiology can improve patient safety and satisfaction with medical care. Prior to the availability of the SIGI field, VA facilities varied in their documentation of gender identity in the patient chart. LGBT veteran care coordinators discussed diverse suggestions that ranged from informally documenting SIGI in each progress note to using flags to draw attention to use certain sections of local EHRs. These suggestions, though well intentioned, were not adequate for documenting gender identity at the national level because of regional variations in EHR customization options. Furthermore, the use of flags for drawing clinical attention to gender identity posed a potential for stigma toward patients, given that flags are typically reserved for behavioral or other risk concerns.

Several problems can emerge when HCPs are not equipped with accurate information about patient birth sex and SIGI. For instance, TGNC patients lack a way of being known from clinic to clinic by proper pronouns or self-labels. Providers may misgender veterans, which is a negative experience for TGNC veterans linked with increased barriers to care and decreased frequency of health care visits.⁴ Moreover, the quality and personalization of care across clinic locations in the facility’s system is variable without a consistent method of documenting birth sex and SIGI. For example, in clinics where the veteran is well known (eg, primary care), staff may be more affirming of the veteran’s gender identity than those in specialty care clinics that lack prior patient contact.

Furthermore, depending on hormone and surgical interventions, some health screenings may be irrelevant for TGNC patients. To determine appropriate health screens and assess potential risks associated with hormone therapy, providers must have access to current information regarding a patient’s physiologic anatomy.⁶ Health screenings and laboratory results in sophisticated EHRs (ie, EHRs that might autodetermine normative values) may populate incorrect treatment recommendations, such as sex-based medication dosages. Furthermore, laboratory test results could be incorrectly paired with a different assumed hormonal history, potentially putting the patient at risk.

 

Case Examples

An important element of EHRs facilitating the goal of patient-centered care is that patients have their EHR validate their sense of self, and their providers can use names and pronouns that correspond to the patient’s SIGI. Some patients have spent a great amount of effort altering their name and sex in legal records and may want their birth sex field to conform to their gender identity. To that end, patients may seek to alter their birth sex information so that it is congruent with how they see themselves to affirm their identity, despite patient safety risks. Several scenarios below demonstrate the potential costs and benefits to patients altering birth sex and SIGI in the EHR.

Case 1 Presentation

A young transman is working with his therapist on engaging in self-validating behaviors. This veteran has met with his PCP and informed the provider of his decision to alter the birth sex field in his EHR from female to male.

Ideally, the patient would begin to have regular conversations with his HCPs about his birth sex and gender identity, so that medical professionals can provide relevant screenings and affirm the patient’s gender identity while acknowledging his right to list his birth sex as he chooses. However, particular attention will need to be paid to assuring that natal sex–based health screenings (eg, pap smears, mammograms) are conducted on an appropriate schedule and that the veteran continues to discuss his current anatomy with providers.

Case 2 Presentation

A veteran has a male birth sex, identifies as a transwoman, and uses nongendered plural pronouns “they/them/theirs.” The word “they,” used as a singular pronoun may feel uncomfortable to some providers, but it validates the veteran’s sense of self and helps them feel welcome in the treatment environment. This patient communicated proactively with their HCPs about their transition goals and current hormone use.

They opted to have their birth sex field continue to indicate “male” because they, after a discussion with their PCP, are aware of the health implications of receiving an incorrect dose for their diabetes medication. They understand that having open communication and receiving input from their HCPs is part of good health care.

Case 3 Presentation

A patient with a sexual development disorder (intersex condition) identifies as a man (indicated as “male” in the SIGI field) and had his birth sex field changed to match his gender identity. He now seeks to change his birth sex field back to female, as he has complicated health considerations due to breast cancer.

The veteran thinks it is important that providers know about his intersex condition so that his breast cancer care is as seamless as possible. In particular, although this veteran is comfortable talking about his intersex condition and his identity with his PCP and oncologist, he wants to ensure that all people involved in his care (eg, pharmacists, radiologists) use the correct values in interpreting his medical data. Providers will need to use the female birth sex field for interpreting his medical data but use male pronouns when interacting with the veteran and documenting his care.

These case examples illustrate the need for HCPs to have patient-affirming education and appropriate clinical tools available when speaking to patients about birth sex, SIGI, and the implications of changing birth sex in the EHR. Moreover, these cases highlight that patient health needs may vary over time, due to factors such as perceived costs/benefits of a change in the sex field of the EHR as well as patient comfort with providers.

 

Current Status of SIGI and EHR

Although having separate fields for birth sex and SIGI in the EHR is ideal, the VHA does not yet have a fully functional SIGI field, and several TGNC veterans have changed their birth sex field to align with their gender identity. Roughly 9,700 patients have diagnostic codes related to transgender care in the VHA, meaning thousands of current patients would potentially benefit from SIGI implementation (John Blosnich, written communication, March 2018). A possible action that the VHA could take with the goal of enhancing patient safety would be to revert the birth sex field of patients who had previously changed the field back to the patient’s original birth sex. However, if this alteration to the EHR were done without the patient’s consent, numerous additional problems would result—including invalidating a veteran’s wishes—potentially driving patients away from receiving health care.

Moreover, in the absence of updated SIGI information (which only the veteran can provide), making a change in the EHR would perpetuate the misgendering of TGNC veterans who have already sought an administrative fix for this problem. Thus, the agency decided to engage patients in a discussion about their decision to keep the birth sex field consistent with their original birth certificate. In cases in which the field had been changed previously, the recommendation is for HCPs to gain patient consent to change the birth sex field back to what was on their original birth certificate. Thus, decisions about what should be listed in the EHR are made by the veteran using an informed decision-making model.

Patient Safety Education Workgroup

To begin the process of disentangling birth sex and SIGI fields in the EHR, 2 work groups were created: a technical work group (coding the patches for SIGI implementation) and a SIGI patient safety education work group. The patient safety education work group was committed to promoting affirmative VA policies that require validation of the gender identity of all veterans and pursuing best practices through clinical guidelines to promote effective, efficient, equitable, and safe veteran care. The patient safety education work group included representatives from all 3 branches of the VA (VHA, Veterans Benefits Administration, and National Cemetery Administration), including clinical media, patient safety, information technology, and education specialists. The group developed trainings for administrative staff about the appropriate ways to ask birth sex and SIGI questions, and how to record veteran-driven responses.

SIGI Fact Sheet

The patient safety education work group examined clinical literature and developed tools for staff and veterans to facilitate effective discussions about the importance and utility of documenting both birth sex and SIGI in the EHR. The patient safety education work group along with media and educational experts created basic key term definition documents to address the importance, purpose, and use of the SIGI field. The patient safety education work group developed 2 documents to facilitate communication between patients and providers.

 

A 1-page veteran-facing fact sheet was developed that described the differences between birth sex and SIGI fields and how these fields are used in the VA EHR system (Figure 1). In addition, a 1-page HCP-facing fact sheet was designed to inform HCPs that patients may have changed their birth sex in their EHR or might still wish to change their birth sex field, and to inform HCPs of the importance of patient-centered, gender-affirmative care (Figure 2). An additional goal of both documents was to educate veterans and HCPs on how the EHR automatically calculates laboratory results and screening notifications based on birth sex.

 

Review Process

As part of reviewing and finalizing the SIGI patient fact sheet, the patient safety education work group previewed the document’s content with veterans who provided feedback on drafts to improve comprehension, patient-centered care, and clinical accuracy. For instance, several patients commented that the document should address many gender identities, including intersex identities. As noted in one of the case presentations earlier, individuals who identify as intersex may have changed their birth sex to be consistent with their gender and might benefit from being informed about the EHR’s autocalculation feature. The patient safety education work group adjusted the SIGI patient fact sheet to include individuals who identify as intersex and instructed them to have a conversation with their HCP regarding potential birth sex changes in the EHR.

Much of the veteran feedback to the patient safety education work group reflected veteran concerns, more broadly, about implementation of SIGI. Many veterans were interested in how federal policy changes might affect their benefits package or clinical care within the VA. The SIGI patient fact sheet was a tool for communicating that Department of Defense (DoD) policies, specifically, do not have a bearing on VA care for LGBT veterans. Therefore, SIGI information does not affect service connection or benefits eligibility and is not shared with the DoD. Veterans found this information helpful to see reflected in the SIGI patient fact sheet.

The patient safety education work group also shared the SIGI provider fact sheet with VHA providers before finalizing the content. PCPs gave feedback to improve the specification of patient safety concerns and appropriate readership language. The patient safety education work group adjusted the SIGI provider fact sheet to be inclusive of relevant literature and an e-consultation link for assisting HCPs who are unsure how to proceed with a patient.

Implementation

The patient safety education work group also developed several materials to provide information about the birth sex and SIGI fields in the EHR. Because the SIGI demographic field is new and collected by clerical staff, training was necessary to explain the difference between birth sex and SIGI before implementation in the EHR. The training sessions educated staff about the difference between birth sex and SIGI, how to ask and respond to questions respectfully, and how to update these fields in the EHR. These trainings included a 20-minute video demonstrating best practices for asking about SIGI, a frequently asked questions document responding to 7 common questions about the new fields, and a quick reference guide for administrative staff to have handy at their desks.

Dissemination of the SIGI patient and provider fact sheets is planned to occur, ideally, several weeks before implementation of the new patches updating the EHR fields in spring 2020. Building on existing resources, the patient safety education work group plans to disseminate the patient fact sheets via e-mail lists for the national mental health facility leaders as well as through e-mail lists for VA PCPs, nursing and clerical staff, privacy officers, facility LGBT veteran care coordinators, VISN leads, transgender e-consultation, the Office of Connected Care, the LGBT external homepage for the VA, and the training website for VA employees. The goal is to target potential points of contact for veterans who may have already changed their birth sex and might benefit medically from altering birth sex to be consistent with their original birth certificate.

The SIGI provider fact sheet will be disseminated using internal e-mails, announcements on routine LGBT veteran care coordinator calls, weekly Ask LGBT Health teleconferences, and announcements at LGBT health training events both internally and externally. Several dissemination tools have already ensured that VA employees are aware of the SIGI field in the EHR. Leadership throughout the VA will be encouraged to share SIGI trainings with clerical staff. Additionally, broad-based e-mails summarizing changes to the EHR will be provided concurrent to the SIGI patch implementation to VA staff as well as links to the resources and training materials.

 

Challenges

One difficulty in the development process for both SIGI fact sheets was addressing the issue of patient safety for veterans who may be at different points in their gender transition process. It was challenging for the patient safety education work group to not sound alarmist in discussing the safety implications of birth sex changes in the EHR, as this is just one factor in clinical decision making. The goal was to educate veterans from a patient safety perspective about the implications of having a state-of-the-art, automated EHR. However, text can be perceived differently by different people, which is why the patient safety education work group asked veterans to preview the patient document and clinical providers to preview the provider document.

Both work groups encountered technologic challenges, including a delay in the implementation of the SIGI field due to a systemwide delay of EHR updates. Although it released training and educational materials to the VHA, the patient safety education work group understood that at some point in the future, VA programmers will update the EHR to change the information clerks and HCPs can see in the EHR. Coordination of the fact sheet release alongside information technology has been an important part of the SIGI rollout process.

Conclusion

HCPs have a complex role in providing treatment to TGNC patients in the VHA: They must affirm a patient’s gender identity through how they address them, while openly communicating the health risks inherent in having their birth sex field be incongruent with the sex recorded on their original birth certificate. Accomplishing these tasks simultaneously is difficult without invalidating the veteran’s identity or right to choose their EHR demographic birth sex label. Furthermore, patients may ask HCPs to write letters of support for either medical or surgical intervention or other documentation changes (eg, changes to a patient’s legal name, passport changes, or a safe passage letter for TGNC patients). Navigating the dialectic of safety and validation requires strong rapport, trust, and effective communication in the patient-provider relationship and great empathy by the provider.

A future task for the SIGI patient safety education work group is to continue to communicate with the technical work group and providers in the field about how demographic fields in the EHR are utilized to enable future EHR changes. This hurdle is not easy because EHR updates change the infrastructure through which demographic content is delivered and incorporated into a patient’s treatment. The VA HCPs are tasked with thoroughly examining the results that automated systems produce to ensure safe and accurate medical services are always provided to all patients. An integral part of patient-centered care is balancing any computer-guided recommendations with an understanding that actual patient needs may differ due to presence/absence of anatomy and other factors (eg, weight, current medications).

From a systems perspective, a benefit of adding the SIGI demographic field is systemic improvement in calculating the number of transgender veterans under VA care and evaluating health outcomes for this population. SIGI is particularly important for signaling gender pronouns for veterans, regardless of whether they are receiving care for a gender-related diagnosis. In terms of scope, the SIGI project potentially will apply to > 9 million enrolled veterans and nearly 400,000 VA employees.

Improvements could be made in the SIGI field of the new EHR, such as expanding the options for self-labels. Additionally, a text field could be used to enhance the quality of personalization provided to veterans self-identifying in the EHR, including pronoun specification. Moreover, adding new fields such as “preferred name” could improve the health care experience of not only TGNC veterans but all veterans who use something other than their full legal name (eg, a nickname). It will be good practice to notify providers and staff of a veteran’s requested name and pronouns when the patient checks in at an electronic kiosk so that all staff immediately know how to address the patient. The VHA can continue to adjust the options for the SIGI field once the new EHR system is operational. Ideally, this new EHR will display birth sex and SIGI to clinicians or clerks engaged in patient interactions.

Technology will continue to automate medical care, meaning that HCPs must be vigilant about how computer programming and the accuracy of prepopulated information affect patient care. The concerns discussed in this report relating to patient safety are relatively absent in the medical literature, even though substantial health risks exist to patients who have birth sex listed incorrectly for any reason.^6,7 Additionally, administrative burden can be reduced if patients who do not need certain screenings based on their current anatomy are not contacted for unnecessary screenings. Future EHR systems might incorporate anatomical considerations from an inventory to assist in automating patient care in safe and accessible ways.