Original Research

Approximately 60,000 hospitalists were working in the United States in 2018.¹ Hospitalist groups work collaboratively because of the shiftwork required for 24/7 patient coverage, and first-rate clinical documentation is essential for quality care.² Thoughtful clinical documentation not only transmits one provider’s clinical reasoning to other providers but is a professional responsibility.³ Hospitalists spend two-thirds of their time in indirect patient-care activities and approximately one quarter of their time on documentation in electronic health records (EHRs).⁴ Despite documentation occupying a substantial portion of the clinician’s time, published literature on the best practices for the documentation of clinical reasoning in hospital medicine or its assessment remains scant.^5-7

Clinical reasoning involves establishing a diagnosis and developing a therapeutic plan that fits the unique circumstances and needs of the patient.⁸ Inpatient providers who admit patients to the hospital end the admission note with their assessment and plan (A&P) after reflecting about a patient’s presenting illness. The A&P generally represents the interpretations, deductions, and clinical reasoning of the inpatient providers; this is the section of the note that fellow physicians concentrate on over others.⁹ The documentation of clinical reasoning in the A&P allows for many to consider how the recorded interpretations relate to their own elucidations resulting in distributed cognition.¹⁰

Disorganized documentation can contribute to cognitive overload and impede thoughtful consideration about the clinical presentation.³ The assessment of clinical documentation may translate into reduced medical errors and improved note quality.^11,12 Studies that have formally evaluated the documentation of clinical reasoning have focused exclusively on medical students.^13-15 The nonexistence of a detailed rubric for evaluating clinical reasoning in the A&Ps of hospitalists represents a missed opportunity for evaluating what hospitalists “do”; if this evolves into a mechanism for offering formative feedback, such professional development would impact the highest level of Miller’s assessment pyramid.¹⁶ We therefore undertook this study to establish a metric to assess the hospitalist providers’ documentation of clinical reasoning in the A&P of an admission note.

Study Design, Setting, and Subjects

This was a retrospective study that reviewed the admission notes of hospitalists for patients admitted over the period of January 2014 and October 2017 at three hospitals in Maryland. One is a community hospital (Hospital A) and two are academic medical centers (Hospital B and Hospital C). Even though these three hospitals are part of one health system, they have distinct cultures and leadership, serve different populations, and are staffed by different provider teams.

The notes of physicians working for the hospitalist groups at each of the three hospitals were the focus of the analysis in this study.

Development of the Documentation Assessment Rubric

A team was assembled to develop the Clinical Reasoning in Admission Note Assessment & PLan (CRANAPL) tool. The CRANAPL was designed to assess the comprehensiveness and thoughtfulness of the clinical reasoning documented in the A&P sections of the notes of patients who were admitted to the hospital with an acute illness. Validity evidence for CRANAPL was summarized on the basis of Messick’s unified validity framework by using four of the five sources of validity: content, response process, internal structure, and relations to other variables.¹⁷

Content Validity

The development team consisted of members who have an average of 10 years of clinical experience in hospital medicine; have studied clinical excellence and clinical reasoning; and have expertise in feedback, assessment, and professional development.^18-22 The development of the CRANAPL tool by the team was informed by a review of the clinical reasoning literature, with particular attention paid to the standards and competencies outlined by the Liaison Committee on Medical Education, the Association of American Medical Colleges, the Accreditation Council on Graduate Medical Education, the Internal Medicine Milestone Project, and the Society of Hospital Medicine.^23-26 For each of these parties, diagnostic reasoning and its impact on clinical decision-making are considered to be a core competency. Several works that heavily influenced the CRANAPL tool’s development were Baker’s Interpretive Summary, Differential Diagnosis, Explanation of Reasoning, And Alternatives (IDEA) assessment tool;¹⁴ King’s Pediatric History and Physical Exam Evaluation (P-HAPEE) rubric;¹⁵ and three other studies related to diagnostic reasoning.^16,27,28 These manuscripts and other works substantively informed the preliminary behavioral-based anchors that formed the initial foundation for the tool under development. The CRANAPL tool was shown to colleagues at other institutions who are leaders on clinical reasoning and was presented at academic conferences in the Division of General Internal Medicine and the Division of Hospital Medicine of our institution. Feedback resulted in iterative revisions. The aforementioned methods established content validity evidence for the CRANAPL tool.

Response Process Validity

Several of the authors pilot-tested earlier iterations on admission notes that were excluded from the sample when refining the CRANAPL tool. The weaknesses and sources of confusion with specific items were addressed by scoring 10 A&Ps individually and then comparing data captured on the tool. This cycle was repeated three times for the iterative enhancement and finalization of the CRANAPL tool. On several occasions when two authors were piloting the near-final CRANAPL tool, a third author interviewed each of the two authors about reactivity while assessing individual items and exploring with probes how their own clinical documentation practices were being considered when scoring the notes. The reasonable and thoughtful answers provided by the two authors as they explained and justified the scores they were selecting during the pilot testing served to confer response process validity evidence.

Finalizing the CRANAPL Tool

The nine-item CRANAPL tool includes elements for problem representation, leading diagnosis, uncertainty, differential diagnosis, plans for diagnosis and treatment, estimated length of stay (LOS), potential for upgrade in status to a higher level of care, and consideration of disposition. Although the final three items are not core clinical reasoning domains in the medical education literature, they represent clinical judgments that are especially relevant for the delivery of the high-quality and cost-effective care of hospitalized patients. Given that the probabilities and estimations of these three elements evolve over the course of any hospitalization on the basis of test results and response to therapy, the documentation of initial expectations on these fronts can facilitate distributed cognition with all individuals becoming wiser from shared insights.¹⁰ The tool uses two- and three-point rating scales, with each number score being clearly defined by specific written criteria (total score range: 0-14; Appendix).

Data Collection

Hospitalists’ admission notes from the three hospitals were used to validate the CRANAPL tool. Admission notes from patients hospitalized to the general medical floors with an admission diagnosis of either fever, syncope/dizziness, or abdominal pain were used. These diagnoses were purposefully examined because they (1) have a wide differential diagnosis, (2) are common presenting symptoms, and (3) are prone to diagnostic errors.^29-32

The centralized EHR system across the three hospitals identified admission notes with one of these primary diagnoses of patients admitted over the period of January 2014 to October 2017. We submitted a request for 650 admission notes to be randomly selected from the centralized institutional records system. The notes were stratified by hospital and diagnosis. The sample size of our study was comparable with that of prior psychometric validation studies.^33,34 Upon reviewing the A&Ps associated with these admissions, 365 notes were excluded for one of three reasons: (1) the note was written by a nurse practitioner, physician assistant, resident, or medical student; (2) the admission diagnosis had been definitively confirmed in the emergency department (eg, abdominal pain due to diverticulitis seen on CT); and (3) the note represented the fourth or more note by any single provider (to sample notes of many providers, no more than three notes written by any single provider were analyzed). A total of 285 admission notes were ultimately included in the sample.

Data were deidentified, and the A&P sections of the admission notes were each copied from the EHR into a unique Word document. Patient and hospital demographic data (including age, gender, race, number of comorbid conditions, LOS, hospital charges, and readmission to the same health system within 30 days) were collected separately from the EHR. Select physician characteristics were also collected from the hospitalist groups at each of the three hospitals, as was the length (word count) of each A&P.

The study was approved by our institutional review board.

Data Analysis

Two authors scored all deidentified A&Ps by using the finalized version of the CRANAPL tool. Prior to using the CRANAPL tool on each of the notes, these raters read each A&P and scored them by using two single-item rating scales: a global clinical reasoning and a global readability/clarity measure. Both of these global scales used three-item Likert scales (below average, average, and above average). These global rating scales collected the reviewers’ gestalt about the quality and clarity of the A&P. The use of gestalt ratings as comparators is supported by other research.³⁵

Descriptive statistics were computed for all variables. Each rater rescored a sample of 48 records (one month after the initial scoring) and intraclass correlations (ICCs) were computed for intrarater reliability. ICCs were calculated for each item and for the CRANAPL total to determine interrater reliability.

The averaged ratings from the two raters were used for all other analyses. For CRANAPL’s internal structure validity evidence, Cronbach’s alpha was calculated as a measure of internal consistency. For relations to other variables validity evidence, CRANAPL total scores were compared with the two global assessment variables with linear regressions.

Bivariate analyses were performed by applying parametric and nonparametric tests as appropriate. A series of multivariate linear regressions, controlling for diagnosis and clustered variance by hospital site, were performed using CRANAPL total as the dependent variable and patient variables as predictors.

All data were analyzed using Stata (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, Texas: StataCorp LP.)

The admission notes of 120 hospitalists were evaluated (Table 1). A total of 39 (33%) physicians were moonlighters with primary appointments outside of the hospitalist division, and 81 (68%) were full-time hospitalists. Among the 120 hospitalists, 48 (40%) were female, 60 (50%) were international medical graduates, and 90 (75%) were of nonwhite race. Most hospitalist physicians (n = 47, 58%) had worked in our health system for less than five years, and 64 hospitalists (53%) devoted greater than 50% of their time to patient care.

Approximately equal numbers of patient admission notes were pulled from each of the three hospitals. The average age of patients was 67.2 (SD 13.6) years, 145 (51%) were female, and 120 (42%) were of nonwhite race. The mean LOS for all patients was 4.0 (SD 3.4) days. A total of 44 (15%) patients were readmitted to the same health system within 30 days of discharge. None of the patients died during the incident hospitalization. The average charge for each of the hospitalizations was $10,646 (SD $9,964).

CRANAPL Data

Figure 1 shows the distribution of the scores given by each rater for each of the nine items. The mean of the total CRANAPL score given by both raters was 6.4 (SD 2.2). Scoring for some items were high (eg, summary statement: 1.5/2), whereas performance on others were low (eg, estimating LOS: 0.1/1 and describing the potential need for upgrade in care: 0.0/1).

Validity of the CRANAPL Tool’s Internal Structure

Cronbach’s alpha, which was used to measure internal consistency within the CRANAPL tool, was 0.43. The ICC, which was applied to measure the interrater reliability for both raters for the total CRANAPL score, was 0.83 (95% CI:  0.76-0.87). The ICC values for intrarater reliability for raters 1 and 2 were 0.73 (95% CI: 0.60-0.83) and 0.73 (95% CI: 0.45-0.86), respectively.

Relations to Other Variables Validity

Associations between CRANAPL total scores, global clinical reasoning, and global scores for note readability/clarity were statistically significant (P < .001), Figure 2.

Eight out of nine CRANAPL variables were statistically significantly different across the three hospitals (P <. 01) when data were analyzed by hospital site. Hospital C had the highest mean score of 7.4 (SD 2.0), followed by Hospital B with a score of 6.6 (SD 2.1), and Hospital A had the lowest total CRANAPL score of 5.2 (SD 1.9). This difference was statistically significant (P < .001). Five variables with respect to admission diagnoses (uncertainty acknowledged, differential diagnosis, plan for diagnosis, plan for treatment, and upgrade plan) were statistically significantly different across notes. Notes for syncope/dizziness generally yielded higher scores than those for abdominal pain and fever.

Factors Associated with High CRANAPL Scores

Table 2 shows the associations between CRANAPL scores and several covariates. Before adjustment, high CRANAPL scores were associated with high word counts of A&Ps (P < .001) and high hospital charges (P < .05). These associations were no longer significant after adjusting for hospital site and admitting diagnoses.

We reviewed the documentation of clinical reasoning in 285 admission notes at three different hospitals written by hospitalist physicians during routine clinical care. To our knowledge, this is the first study that assessed the documentation of hospitalists’ clinical reasoning with real patient notes. Wide variability exists in the documentation of clinical reasoning within the A&Ps of hospitalists’ admission notes. We have provided validity evidence to support the use of the user-friendly CRANAPL tool.

Prior studies have described rubrics for evaluating the clinical reasoning skills of medical students.^14,15 The ICCs for the IDEA rubric used to assess medical students’ documentation of clinical reasoning were fair to moderate (0.29-0.67), whereas the ICC for the CRANAPL tool was high at 0.83. This measure of reliability is similar to that for the P-HAPEE rubric used to assess medical students’ documentation of pediatric history and physical notes.¹⁵ These data are markedly different from the data in previous studies that have found low interrater reliability for psychometric evaluations related to judgment and decision-making.^36-39 CRANAPL was also found to have high intrarater reliability, which shows the reproducibility of an individual’s assessment over time. The strong association between the total CRANAPL score and global clinical reasoning assessment found in the present study is similar to that found in previous studies that have also embedded global rating scales as comparators when assessing clinical reasoning.^13,,15,40,41 Global rating scales represent an overarching structure for comparison given the absence of an accepted method or gold standard for assessing clinical reasoning documentation. High-quality provider notes are defined by clarity, thoroughness, and accuracy;³⁵ and effective documentation promotes communication and the coordination of care among the members of the care team.³

The total CRANAPL scores varied by hospital site with academic hospitals (B and C) scoring higher than the community hospital (A) in our study. Similarly, lengthy A&Ps were associated with high CRANAPL scores (P < .001) prior to adjustment for hospital site. Healthcare providers consider that the thoroughness of documentation denotes quality and attention to detail.^35,42 Comprehensive documentation takes time; the longer notes by academic hospitalists than those by community hospitalists may be attributed to the fewer number of patients generally carried by hospitalists at academic centers than that by hospitalists at community hospitals.⁴³

The documentation of the estimations of LOS, possibility of potential upgrade, and thoughts about disposition were consistently poorly described across all hospital sites and diagnoses. In contrast to CRANAPL, other clinical reasoning rubrics have excluded these items or discussed uncertainty.^14,15,44 These elements represent the forward thinking that may be essential for high-quality progressive care by hospitalists. Physicians’s difficulty in acknowledging uncertainty has been associated with resource overuse, including the excessive ordering of tests, iatrogenic injury, and heavy financial burden on the healthcare system.^45,46 The lack of thoughtful clinical and management reasoning at the time of admission is believed to be associated with medical errors.⁴⁷ If used as a guide, the CRANAPL tool may promote reflection on the part of the admitting physician. The estimations of LOS, potential for upgrade to a higher level of care, and disposition are markers of optimal inpatient care, especially for hospitalists who work in shifts with embedded handoffs. When shared with colleagues (through documentation), there is the potential for distributed cognition¹⁰ to extend throughout the social network of the hospitalist group. The fact that so few providers are currently including these items in their A&P’s show that the providers are either not performing or documenting the ‘reasoning’. Either way, this is an opportunity that has been highlighted by the CRANAPL tool.

Several limitations of this study should be considered. First, the CRANAPL tool may not have captured elements of optimal clinical reasoning documentation. The reliance on multiple methods and an iterative process in the refinement of the CRANAPL tool should have minimized this. Second, this study was conducted across a single healthcare system that uses the same EHR; this EHR or institutional culture may influence documentation practices and behaviors. Given that using the CRANAPL tool to score an A&P is quick and easy, the benefit of giving providers feedback on their notes remains to be seen—here and at other hospitals. Third, our sample size could limit the generalizability of the results and the significance of the associations. However, the sample assessed in our study was significantly larger than that assessed in other studies that have validated clinical reasoning rubrics.^14,15 Fourth, clinical reasoning is a broad and multidimensional construct. The CRANAPL tool focuses exclusively on hospitalists’ documentation of clinical reasoning and therefore does not assess aspects of clinical reasoning occurring in the physicians’ minds. Finally, given our goal to optimally validate the CRANAPL tool, we chose to test the tool on specific presentations that are known to be associated with diagnostic practice variation and errors. We may have observed different results had we chosen a different set of diagnoses from each hospital. Further validity evidence will be established when applying the CRANPL tool to different diagnoses and to notes from other clinical settings.

In conclusion, this study focuses on the development and validation of the CRANAPL tool that assesses how hospitalists document their clinical reasoning in the A&P section of admission notes. Our results show that wide variability exists in the documentation of clinical reasoning by hospitalists within and across hospitals. Given the CRANAPL tool’s ease-of-use and its versatility, hospitalist divisions in academic and nonacademic settings may use the CRANAPL tool to assess and provide feedback on the documentation of hospitalists’ clinical reasoning. Beyond studying whether physicians can be taught to improve their notes with feedback based on the CRANAPL tool, future studies may explore whether enhancing clinical reasoning documentation may be associated with improvements in patient care and clinical outcomes.

Acknowledgments

Dr. Wright is the Anne Gaines and G. Thomas Miller Professor of Medicine which is supported through Hopkins’ Center for Innovative Medicine.

The authors thank Christine Caufield-Noll, MLIS, AHIP (Johns Hopkins Bayview Medical Center, Baltimore, Maryland) for her assistance with this project.

Disclosures

The authors have nothing to disclose.

Approximately 60,000 hospitalists were working in the United States in 2018.¹ Hospitalist groups work collaboratively because of the shiftwork required for 24/7 patient coverage, and first-rate clinical documentation is essential for quality care.² Thoughtful clinical documentation not only transmits one provider’s clinical reasoning to other providers but is a professional responsibility.³ Hospitalists spend two-thirds of their time in indirect patient-care activities and approximately one quarter of their time on documentation in electronic health records (EHRs).⁴ Despite documentation occupying a substantial portion of the clinician’s time, published literature on the best practices for the documentation of clinical reasoning in hospital medicine or its assessment remains scant.^5-7

Clinical reasoning involves establishing a diagnosis and developing a therapeutic plan that fits the unique circumstances and needs of the patient.⁸ Inpatient providers who admit patients to the hospital end the admission note with their assessment and plan (A&P) after reflecting about a patient’s presenting illness. The A&P generally represents the interpretations, deductions, and clinical reasoning of the inpatient providers; this is the section of the note that fellow physicians concentrate on over others.⁹ The documentation of clinical reasoning in the A&P allows for many to consider how the recorded interpretations relate to their own elucidations resulting in distributed cognition.¹⁰

Disorganized documentation can contribute to cognitive overload and impede thoughtful consideration about the clinical presentation.³ The assessment of clinical documentation may translate into reduced medical errors and improved note quality.^11,12 Studies that have formally evaluated the documentation of clinical reasoning have focused exclusively on medical students.^13-15 The nonexistence of a detailed rubric for evaluating clinical reasoning in the A&Ps of hospitalists represents a missed opportunity for evaluating what hospitalists “do”; if this evolves into a mechanism for offering formative feedback, such professional development would impact the highest level of Miller’s assessment pyramid.¹⁶ We therefore undertook this study to establish a metric to assess the hospitalist providers’ documentation of clinical reasoning in the A&P of an admission note.

Study Design, Setting, and Subjects

This was a retrospective study that reviewed the admission notes of hospitalists for patients admitted over the period of January 2014 and October 2017 at three hospitals in Maryland. One is a community hospital (Hospital A) and two are academic medical centers (Hospital B and Hospital C). Even though these three hospitals are part of one health system, they have distinct cultures and leadership, serve different populations, and are staffed by different provider teams.

The notes of physicians working for the hospitalist groups at each of the three hospitals were the focus of the analysis in this study.

Development of the Documentation Assessment Rubric

A team was assembled to develop the Clinical Reasoning in Admission Note Assessment & PLan (CRANAPL) tool. The CRANAPL was designed to assess the comprehensiveness and thoughtfulness of the clinical reasoning documented in the A&P sections of the notes of patients who were admitted to the hospital with an acute illness. Validity evidence for CRANAPL was summarized on the basis of Messick’s unified validity framework by using four of the five sources of validity: content, response process, internal structure, and relations to other variables.¹⁷

Content Validity

The development team consisted of members who have an average of 10 years of clinical experience in hospital medicine; have studied clinical excellence and clinical reasoning; and have expertise in feedback, assessment, and professional development.^18-22 The development of the CRANAPL tool by the team was informed by a review of the clinical reasoning literature, with particular attention paid to the standards and competencies outlined by the Liaison Committee on Medical Education, the Association of American Medical Colleges, the Accreditation Council on Graduate Medical Education, the Internal Medicine Milestone Project, and the Society of Hospital Medicine.^23-26 For each of these parties, diagnostic reasoning and its impact on clinical decision-making are considered to be a core competency. Several works that heavily influenced the CRANAPL tool’s development were Baker’s Interpretive Summary, Differential Diagnosis, Explanation of Reasoning, And Alternatives (IDEA) assessment tool;¹⁴ King’s Pediatric History and Physical Exam Evaluation (P-HAPEE) rubric;¹⁵ and three other studies related to diagnostic reasoning.^16,27,28 These manuscripts and other works substantively informed the preliminary behavioral-based anchors that formed the initial foundation for the tool under development. The CRANAPL tool was shown to colleagues at other institutions who are leaders on clinical reasoning and was presented at academic conferences in the Division of General Internal Medicine and the Division of Hospital Medicine of our institution. Feedback resulted in iterative revisions. The aforementioned methods established content validity evidence for the CRANAPL tool.

Response Process Validity

Several of the authors pilot-tested earlier iterations on admission notes that were excluded from the sample when refining the CRANAPL tool. The weaknesses and sources of confusion with specific items were addressed by scoring 10 A&Ps individually and then comparing data captured on the tool. This cycle was repeated three times for the iterative enhancement and finalization of the CRANAPL tool. On several occasions when two authors were piloting the near-final CRANAPL tool, a third author interviewed each of the two authors about reactivity while assessing individual items and exploring with probes how their own clinical documentation practices were being considered when scoring the notes. The reasonable and thoughtful answers provided by the two authors as they explained and justified the scores they were selecting during the pilot testing served to confer response process validity evidence.

Finalizing the CRANAPL Tool

The nine-item CRANAPL tool includes elements for problem representation, leading diagnosis, uncertainty, differential diagnosis, plans for diagnosis and treatment, estimated length of stay (LOS), potential for upgrade in status to a higher level of care, and consideration of disposition. Although the final three items are not core clinical reasoning domains in the medical education literature, they represent clinical judgments that are especially relevant for the delivery of the high-quality and cost-effective care of hospitalized patients. Given that the probabilities and estimations of these three elements evolve over the course of any hospitalization on the basis of test results and response to therapy, the documentation of initial expectations on these fronts can facilitate distributed cognition with all individuals becoming wiser from shared insights.¹⁰ The tool uses two- and three-point rating scales, with each number score being clearly defined by specific written criteria (total score range: 0-14; Appendix).

Data Collection

Hospitalists’ admission notes from the three hospitals were used to validate the CRANAPL tool. Admission notes from patients hospitalized to the general medical floors with an admission diagnosis of either fever, syncope/dizziness, or abdominal pain were used. These diagnoses were purposefully examined because they (1) have a wide differential diagnosis, (2) are common presenting symptoms, and (3) are prone to diagnostic errors.^29-32

The centralized EHR system across the three hospitals identified admission notes with one of these primary diagnoses of patients admitted over the period of January 2014 to October 2017. We submitted a request for 650 admission notes to be randomly selected from the centralized institutional records system. The notes were stratified by hospital and diagnosis. The sample size of our study was comparable with that of prior psychometric validation studies.^33,34 Upon reviewing the A&Ps associated with these admissions, 365 notes were excluded for one of three reasons: (1) the note was written by a nurse practitioner, physician assistant, resident, or medical student; (2) the admission diagnosis had been definitively confirmed in the emergency department (eg, abdominal pain due to diverticulitis seen on CT); and (3) the note represented the fourth or more note by any single provider (to sample notes of many providers, no more than three notes written by any single provider were analyzed). A total of 285 admission notes were ultimately included in the sample.

Data were deidentified, and the A&P sections of the admission notes were each copied from the EHR into a unique Word document. Patient and hospital demographic data (including age, gender, race, number of comorbid conditions, LOS, hospital charges, and readmission to the same health system within 30 days) were collected separately from the EHR. Select physician characteristics were also collected from the hospitalist groups at each of the three hospitals, as was the length (word count) of each A&P.

The study was approved by our institutional review board.

Data Analysis

Two authors scored all deidentified A&Ps by using the finalized version of the CRANAPL tool. Prior to using the CRANAPL tool on each of the notes, these raters read each A&P and scored them by using two single-item rating scales: a global clinical reasoning and a global readability/clarity measure. Both of these global scales used three-item Likert scales (below average, average, and above average). These global rating scales collected the reviewers’ gestalt about the quality and clarity of the A&P. The use of gestalt ratings as comparators is supported by other research.³⁵

Descriptive statistics were computed for all variables. Each rater rescored a sample of 48 records (one month after the initial scoring) and intraclass correlations (ICCs) were computed for intrarater reliability. ICCs were calculated for each item and for the CRANAPL total to determine interrater reliability.

The averaged ratings from the two raters were used for all other analyses. For CRANAPL’s internal structure validity evidence, Cronbach’s alpha was calculated as a measure of internal consistency. For relations to other variables validity evidence, CRANAPL total scores were compared with the two global assessment variables with linear regressions.

Bivariate analyses were performed by applying parametric and nonparametric tests as appropriate. A series of multivariate linear regressions, controlling for diagnosis and clustered variance by hospital site, were performed using CRANAPL total as the dependent variable and patient variables as predictors.

All data were analyzed using Stata (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, Texas: StataCorp LP.)

The admission notes of 120 hospitalists were evaluated (Table 1). A total of 39 (33%) physicians were moonlighters with primary appointments outside of the hospitalist division, and 81 (68%) were full-time hospitalists. Among the 120 hospitalists, 48 (40%) were female, 60 (50%) were international medical graduates, and 90 (75%) were of nonwhite race. Most hospitalist physicians (n = 47, 58%) had worked in our health system for less than five years, and 64 hospitalists (53%) devoted greater than 50% of their time to patient care.

Approximately equal numbers of patient admission notes were pulled from each of the three hospitals. The average age of patients was 67.2 (SD 13.6) years, 145 (51%) were female, and 120 (42%) were of nonwhite race. The mean LOS for all patients was 4.0 (SD 3.4) days. A total of 44 (15%) patients were readmitted to the same health system within 30 days of discharge. None of the patients died during the incident hospitalization. The average charge for each of the hospitalizations was $10,646 (SD $9,964).

CRANAPL Data

Figure 1 shows the distribution of the scores given by each rater for each of the nine items. The mean of the total CRANAPL score given by both raters was 6.4 (SD 2.2). Scoring for some items were high (eg, summary statement: 1.5/2), whereas performance on others were low (eg, estimating LOS: 0.1/1 and describing the potential need for upgrade in care: 0.0/1).

Validity of the CRANAPL Tool’s Internal Structure

Cronbach’s alpha, which was used to measure internal consistency within the CRANAPL tool, was 0.43. The ICC, which was applied to measure the interrater reliability for both raters for the total CRANAPL score, was 0.83 (95% CI:  0.76-0.87). The ICC values for intrarater reliability for raters 1 and 2 were 0.73 (95% CI: 0.60-0.83) and 0.73 (95% CI: 0.45-0.86), respectively.

Relations to Other Variables Validity

Associations between CRANAPL total scores, global clinical reasoning, and global scores for note readability/clarity were statistically significant (P < .001), Figure 2.

Eight out of nine CRANAPL variables were statistically significantly different across the three hospitals (P <. 01) when data were analyzed by hospital site. Hospital C had the highest mean score of 7.4 (SD 2.0), followed by Hospital B with a score of 6.6 (SD 2.1), and Hospital A had the lowest total CRANAPL score of 5.2 (SD 1.9). This difference was statistically significant (P < .001). Five variables with respect to admission diagnoses (uncertainty acknowledged, differential diagnosis, plan for diagnosis, plan for treatment, and upgrade plan) were statistically significantly different across notes. Notes for syncope/dizziness generally yielded higher scores than those for abdominal pain and fever.

Factors Associated with High CRANAPL Scores

Table 2 shows the associations between CRANAPL scores and several covariates. Before adjustment, high CRANAPL scores were associated with high word counts of A&Ps (P < .001) and high hospital charges (P < .05). These associations were no longer significant after adjusting for hospital site and admitting diagnoses.

We reviewed the documentation of clinical reasoning in 285 admission notes at three different hospitals written by hospitalist physicians during routine clinical care. To our knowledge, this is the first study that assessed the documentation of hospitalists’ clinical reasoning with real patient notes. Wide variability exists in the documentation of clinical reasoning within the A&Ps of hospitalists’ admission notes. We have provided validity evidence to support the use of the user-friendly CRANAPL tool.

Prior studies have described rubrics for evaluating the clinical reasoning skills of medical students.^14,15 The ICCs for the IDEA rubric used to assess medical students’ documentation of clinical reasoning were fair to moderate (0.29-0.67), whereas the ICC for the CRANAPL tool was high at 0.83. This measure of reliability is similar to that for the P-HAPEE rubric used to assess medical students’ documentation of pediatric history and physical notes.¹⁵ These data are markedly different from the data in previous studies that have found low interrater reliability for psychometric evaluations related to judgment and decision-making.^36-39 CRANAPL was also found to have high intrarater reliability, which shows the reproducibility of an individual’s assessment over time. The strong association between the total CRANAPL score and global clinical reasoning assessment found in the present study is similar to that found in previous studies that have also embedded global rating scales as comparators when assessing clinical reasoning.^13,,15,40,41 Global rating scales represent an overarching structure for comparison given the absence of an accepted method or gold standard for assessing clinical reasoning documentation. High-quality provider notes are defined by clarity, thoroughness, and accuracy;³⁵ and effective documentation promotes communication and the coordination of care among the members of the care team.³

The total CRANAPL scores varied by hospital site with academic hospitals (B and C) scoring higher than the community hospital (A) in our study. Similarly, lengthy A&Ps were associated with high CRANAPL scores (P < .001) prior to adjustment for hospital site. Healthcare providers consider that the thoroughness of documentation denotes quality and attention to detail.^35,42 Comprehensive documentation takes time; the longer notes by academic hospitalists than those by community hospitalists may be attributed to the fewer number of patients generally carried by hospitalists at academic centers than that by hospitalists at community hospitals.⁴³

The documentation of the estimations of LOS, possibility of potential upgrade, and thoughts about disposition were consistently poorly described across all hospital sites and diagnoses. In contrast to CRANAPL, other clinical reasoning rubrics have excluded these items or discussed uncertainty.^14,15,44 These elements represent the forward thinking that may be essential for high-quality progressive care by hospitalists. Physicians’s difficulty in acknowledging uncertainty has been associated with resource overuse, including the excessive ordering of tests, iatrogenic injury, and heavy financial burden on the healthcare system.^45,46 The lack of thoughtful clinical and management reasoning at the time of admission is believed to be associated with medical errors.⁴⁷ If used as a guide, the CRANAPL tool may promote reflection on the part of the admitting physician. The estimations of LOS, potential for upgrade to a higher level of care, and disposition are markers of optimal inpatient care, especially for hospitalists who work in shifts with embedded handoffs. When shared with colleagues (through documentation), there is the potential for distributed cognition¹⁰ to extend throughout the social network of the hospitalist group. The fact that so few providers are currently including these items in their A&P’s show that the providers are either not performing or documenting the ‘reasoning’. Either way, this is an opportunity that has been highlighted by the CRANAPL tool.

Several limitations of this study should be considered. First, the CRANAPL tool may not have captured elements of optimal clinical reasoning documentation. The reliance on multiple methods and an iterative process in the refinement of the CRANAPL tool should have minimized this. Second, this study was conducted across a single healthcare system that uses the same EHR; this EHR or institutional culture may influence documentation practices and behaviors. Given that using the CRANAPL tool to score an A&P is quick and easy, the benefit of giving providers feedback on their notes remains to be seen—here and at other hospitals. Third, our sample size could limit the generalizability of the results and the significance of the associations. However, the sample assessed in our study was significantly larger than that assessed in other studies that have validated clinical reasoning rubrics.^14,15 Fourth, clinical reasoning is a broad and multidimensional construct. The CRANAPL tool focuses exclusively on hospitalists’ documentation of clinical reasoning and therefore does not assess aspects of clinical reasoning occurring in the physicians’ minds. Finally, given our goal to optimally validate the CRANAPL tool, we chose to test the tool on specific presentations that are known to be associated with diagnostic practice variation and errors. We may have observed different results had we chosen a different set of diagnoses from each hospital. Further validity evidence will be established when applying the CRANPL tool to different diagnoses and to notes from other clinical settings.

In conclusion, this study focuses on the development and validation of the CRANAPL tool that assesses how hospitalists document their clinical reasoning in the A&P section of admission notes. Our results show that wide variability exists in the documentation of clinical reasoning by hospitalists within and across hospitals. Given the CRANAPL tool’s ease-of-use and its versatility, hospitalist divisions in academic and nonacademic settings may use the CRANAPL tool to assess and provide feedback on the documentation of hospitalists’ clinical reasoning. Beyond studying whether physicians can be taught to improve their notes with feedback based on the CRANAPL tool, future studies may explore whether enhancing clinical reasoning documentation may be associated with improvements in patient care and clinical outcomes.

Acknowledgments

Dr. Wright is the Anne Gaines and G. Thomas Miller Professor of Medicine which is supported through Hopkins’ Center for Innovative Medicine.

The authors thank Christine Caufield-Noll, MLIS, AHIP (Johns Hopkins Bayview Medical Center, Baltimore, Maryland) for her assistance with this project.

Disclosures

The authors have nothing to disclose.

Approximately 60,000 hospitalists were working in the United States in 2018.¹ Hospitalist groups work collaboratively because of the shiftwork required for 24/7 patient coverage, and first-rate clinical documentation is essential for quality care.² Thoughtful clinical documentation not only transmits one provider’s clinical reasoning to other providers but is a professional responsibility.³ Hospitalists spend two-thirds of their time in indirect patient-care activities and approximately one quarter of their time on documentation in electronic health records (EHRs).⁴ Despite documentation occupying a substantial portion of the clinician’s time, published literature on the best practices for the documentation of clinical reasoning in hospital medicine or its assessment remains scant.^5-7

Clinical reasoning involves establishing a diagnosis and developing a therapeutic plan that fits the unique circumstances and needs of the patient.⁸ Inpatient providers who admit patients to the hospital end the admission note with their assessment and plan (A&P) after reflecting about a patient’s presenting illness. The A&P generally represents the interpretations, deductions, and clinical reasoning of the inpatient providers; this is the section of the note that fellow physicians concentrate on over others.⁹ The documentation of clinical reasoning in the A&P allows for many to consider how the recorded interpretations relate to their own elucidations resulting in distributed cognition.¹⁰

Disorganized documentation can contribute to cognitive overload and impede thoughtful consideration about the clinical presentation.³ The assessment of clinical documentation may translate into reduced medical errors and improved note quality.^11,12 Studies that have formally evaluated the documentation of clinical reasoning have focused exclusively on medical students.^13-15 The nonexistence of a detailed rubric for evaluating clinical reasoning in the A&Ps of hospitalists represents a missed opportunity for evaluating what hospitalists “do”; if this evolves into a mechanism for offering formative feedback, such professional development would impact the highest level of Miller’s assessment pyramid.¹⁶ We therefore undertook this study to establish a metric to assess the hospitalist providers’ documentation of clinical reasoning in the A&P of an admission note.

Study Design, Setting, and Subjects

This was a retrospective study that reviewed the admission notes of hospitalists for patients admitted over the period of January 2014 and October 2017 at three hospitals in Maryland. One is a community hospital (Hospital A) and two are academic medical centers (Hospital B and Hospital C). Even though these three hospitals are part of one health system, they have distinct cultures and leadership, serve different populations, and are staffed by different provider teams.

The notes of physicians working for the hospitalist groups at each of the three hospitals were the focus of the analysis in this study.

Development of the Documentation Assessment Rubric

A team was assembled to develop the Clinical Reasoning in Admission Note Assessment & PLan (CRANAPL) tool. The CRANAPL was designed to assess the comprehensiveness and thoughtfulness of the clinical reasoning documented in the A&P sections of the notes of patients who were admitted to the hospital with an acute illness. Validity evidence for CRANAPL was summarized on the basis of Messick’s unified validity framework by using four of the five sources of validity: content, response process, internal structure, and relations to other variables.¹⁷

Content Validity

The development team consisted of members who have an average of 10 years of clinical experience in hospital medicine; have studied clinical excellence and clinical reasoning; and have expertise in feedback, assessment, and professional development.^18-22 The development of the CRANAPL tool by the team was informed by a review of the clinical reasoning literature, with particular attention paid to the standards and competencies outlined by the Liaison Committee on Medical Education, the Association of American Medical Colleges, the Accreditation Council on Graduate Medical Education, the Internal Medicine Milestone Project, and the Society of Hospital Medicine.^23-26 For each of these parties, diagnostic reasoning and its impact on clinical decision-making are considered to be a core competency. Several works that heavily influenced the CRANAPL tool’s development were Baker’s Interpretive Summary, Differential Diagnosis, Explanation of Reasoning, And Alternatives (IDEA) assessment tool;¹⁴ King’s Pediatric History and Physical Exam Evaluation (P-HAPEE) rubric;¹⁵ and three other studies related to diagnostic reasoning.^16,27,28 These manuscripts and other works substantively informed the preliminary behavioral-based anchors that formed the initial foundation for the tool under development. The CRANAPL tool was shown to colleagues at other institutions who are leaders on clinical reasoning and was presented at academic conferences in the Division of General Internal Medicine and the Division of Hospital Medicine of our institution. Feedback resulted in iterative revisions. The aforementioned methods established content validity evidence for the CRANAPL tool.

Response Process Validity

Several of the authors pilot-tested earlier iterations on admission notes that were excluded from the sample when refining the CRANAPL tool. The weaknesses and sources of confusion with specific items were addressed by scoring 10 A&Ps individually and then comparing data captured on the tool. This cycle was repeated three times for the iterative enhancement and finalization of the CRANAPL tool. On several occasions when two authors were piloting the near-final CRANAPL tool, a third author interviewed each of the two authors about reactivity while assessing individual items and exploring with probes how their own clinical documentation practices were being considered when scoring the notes. The reasonable and thoughtful answers provided by the two authors as they explained and justified the scores they were selecting during the pilot testing served to confer response process validity evidence.

Finalizing the CRANAPL Tool

The nine-item CRANAPL tool includes elements for problem representation, leading diagnosis, uncertainty, differential diagnosis, plans for diagnosis and treatment, estimated length of stay (LOS), potential for upgrade in status to a higher level of care, and consideration of disposition. Although the final three items are not core clinical reasoning domains in the medical education literature, they represent clinical judgments that are especially relevant for the delivery of the high-quality and cost-effective care of hospitalized patients. Given that the probabilities and estimations of these three elements evolve over the course of any hospitalization on the basis of test results and response to therapy, the documentation of initial expectations on these fronts can facilitate distributed cognition with all individuals becoming wiser from shared insights.¹⁰ The tool uses two- and three-point rating scales, with each number score being clearly defined by specific written criteria (total score range: 0-14; Appendix).

Data Collection

Hospitalists’ admission notes from the three hospitals were used to validate the CRANAPL tool. Admission notes from patients hospitalized to the general medical floors with an admission diagnosis of either fever, syncope/dizziness, or abdominal pain were used. These diagnoses were purposefully examined because they (1) have a wide differential diagnosis, (2) are common presenting symptoms, and (3) are prone to diagnostic errors.^29-32

The centralized EHR system across the three hospitals identified admission notes with one of these primary diagnoses of patients admitted over the period of January 2014 to October 2017. We submitted a request for 650 admission notes to be randomly selected from the centralized institutional records system. The notes were stratified by hospital and diagnosis. The sample size of our study was comparable with that of prior psychometric validation studies.^33,34 Upon reviewing the A&Ps associated with these admissions, 365 notes were excluded for one of three reasons: (1) the note was written by a nurse practitioner, physician assistant, resident, or medical student; (2) the admission diagnosis had been definitively confirmed in the emergency department (eg, abdominal pain due to diverticulitis seen on CT); and (3) the note represented the fourth or more note by any single provider (to sample notes of many providers, no more than three notes written by any single provider were analyzed). A total of 285 admission notes were ultimately included in the sample.

Data were deidentified, and the A&P sections of the admission notes were each copied from the EHR into a unique Word document. Patient and hospital demographic data (including age, gender, race, number of comorbid conditions, LOS, hospital charges, and readmission to the same health system within 30 days) were collected separately from the EHR. Select physician characteristics were also collected from the hospitalist groups at each of the three hospitals, as was the length (word count) of each A&P.

The study was approved by our institutional review board.

Data Analysis

Two authors scored all deidentified A&Ps by using the finalized version of the CRANAPL tool. Prior to using the CRANAPL tool on each of the notes, these raters read each A&P and scored them by using two single-item rating scales: a global clinical reasoning and a global readability/clarity measure. Both of these global scales used three-item Likert scales (below average, average, and above average). These global rating scales collected the reviewers’ gestalt about the quality and clarity of the A&P. The use of gestalt ratings as comparators is supported by other research.³⁵

Descriptive statistics were computed for all variables. Each rater rescored a sample of 48 records (one month after the initial scoring) and intraclass correlations (ICCs) were computed for intrarater reliability. ICCs were calculated for each item and for the CRANAPL total to determine interrater reliability.

The averaged ratings from the two raters were used for all other analyses. For CRANAPL’s internal structure validity evidence, Cronbach’s alpha was calculated as a measure of internal consistency. For relations to other variables validity evidence, CRANAPL total scores were compared with the two global assessment variables with linear regressions.

Bivariate analyses were performed by applying parametric and nonparametric tests as appropriate. A series of multivariate linear regressions, controlling for diagnosis and clustered variance by hospital site, were performed using CRANAPL total as the dependent variable and patient variables as predictors.

All data were analyzed using Stata (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, Texas: StataCorp LP.)

The admission notes of 120 hospitalists were evaluated (Table 1). A total of 39 (33%) physicians were moonlighters with primary appointments outside of the hospitalist division, and 81 (68%) were full-time hospitalists. Among the 120 hospitalists, 48 (40%) were female, 60 (50%) were international medical graduates, and 90 (75%) were of nonwhite race. Most hospitalist physicians (n = 47, 58%) had worked in our health system for less than five years, and 64 hospitalists (53%) devoted greater than 50% of their time to patient care.

Approximately equal numbers of patient admission notes were pulled from each of the three hospitals. The average age of patients was 67.2 (SD 13.6) years, 145 (51%) were female, and 120 (42%) were of nonwhite race. The mean LOS for all patients was 4.0 (SD 3.4) days. A total of 44 (15%) patients were readmitted to the same health system within 30 days of discharge. None of the patients died during the incident hospitalization. The average charge for each of the hospitalizations was $10,646 (SD $9,964).

CRANAPL Data

Figure 1 shows the distribution of the scores given by each rater for each of the nine items. The mean of the total CRANAPL score given by both raters was 6.4 (SD 2.2). Scoring for some items were high (eg, summary statement: 1.5/2), whereas performance on others were low (eg, estimating LOS: 0.1/1 and describing the potential need for upgrade in care: 0.0/1).

Validity of the CRANAPL Tool’s Internal Structure

Cronbach’s alpha, which was used to measure internal consistency within the CRANAPL tool, was 0.43. The ICC, which was applied to measure the interrater reliability for both raters for the total CRANAPL score, was 0.83 (95% CI:  0.76-0.87). The ICC values for intrarater reliability for raters 1 and 2 were 0.73 (95% CI: 0.60-0.83) and 0.73 (95% CI: 0.45-0.86), respectively.

Relations to Other Variables Validity

Associations between CRANAPL total scores, global clinical reasoning, and global scores for note readability/clarity were statistically significant (P < .001), Figure 2.

Eight out of nine CRANAPL variables were statistically significantly different across the three hospitals (P <. 01) when data were analyzed by hospital site. Hospital C had the highest mean score of 7.4 (SD 2.0), followed by Hospital B with a score of 6.6 (SD 2.1), and Hospital A had the lowest total CRANAPL score of 5.2 (SD 1.9). This difference was statistically significant (P < .001). Five variables with respect to admission diagnoses (uncertainty acknowledged, differential diagnosis, plan for diagnosis, plan for treatment, and upgrade plan) were statistically significantly different across notes. Notes for syncope/dizziness generally yielded higher scores than those for abdominal pain and fever.

Factors Associated with High CRANAPL Scores

Table 2 shows the associations between CRANAPL scores and several covariates. Before adjustment, high CRANAPL scores were associated with high word counts of A&Ps (P < .001) and high hospital charges (P < .05). These associations were no longer significant after adjusting for hospital site and admitting diagnoses.

We reviewed the documentation of clinical reasoning in 285 admission notes at three different hospitals written by hospitalist physicians during routine clinical care. To our knowledge, this is the first study that assessed the documentation of hospitalists’ clinical reasoning with real patient notes. Wide variability exists in the documentation of clinical reasoning within the A&Ps of hospitalists’ admission notes. We have provided validity evidence to support the use of the user-friendly CRANAPL tool.

Prior studies have described rubrics for evaluating the clinical reasoning skills of medical students.^14,15 The ICCs for the IDEA rubric used to assess medical students’ documentation of clinical reasoning were fair to moderate (0.29-0.67), whereas the ICC for the CRANAPL tool was high at 0.83. This measure of reliability is similar to that for the P-HAPEE rubric used to assess medical students’ documentation of pediatric history and physical notes.¹⁵ These data are markedly different from the data in previous studies that have found low interrater reliability for psychometric evaluations related to judgment and decision-making.^36-39 CRANAPL was also found to have high intrarater reliability, which shows the reproducibility of an individual’s assessment over time. The strong association between the total CRANAPL score and global clinical reasoning assessment found in the present study is similar to that found in previous studies that have also embedded global rating scales as comparators when assessing clinical reasoning.^13,,15,40,41 Global rating scales represent an overarching structure for comparison given the absence of an accepted method or gold standard for assessing clinical reasoning documentation. High-quality provider notes are defined by clarity, thoroughness, and accuracy;³⁵ and effective documentation promotes communication and the coordination of care among the members of the care team.³

The total CRANAPL scores varied by hospital site with academic hospitals (B and C) scoring higher than the community hospital (A) in our study. Similarly, lengthy A&Ps were associated with high CRANAPL scores (P < .001) prior to adjustment for hospital site. Healthcare providers consider that the thoroughness of documentation denotes quality and attention to detail.^35,42 Comprehensive documentation takes time; the longer notes by academic hospitalists than those by community hospitalists may be attributed to the fewer number of patients generally carried by hospitalists at academic centers than that by hospitalists at community hospitals.⁴³

The documentation of the estimations of LOS, possibility of potential upgrade, and thoughts about disposition were consistently poorly described across all hospital sites and diagnoses. In contrast to CRANAPL, other clinical reasoning rubrics have excluded these items or discussed uncertainty.^14,15,44 These elements represent the forward thinking that may be essential for high-quality progressive care by hospitalists. Physicians’s difficulty in acknowledging uncertainty has been associated with resource overuse, including the excessive ordering of tests, iatrogenic injury, and heavy financial burden on the healthcare system.^45,46 The lack of thoughtful clinical and management reasoning at the time of admission is believed to be associated with medical errors.⁴⁷ If used as a guide, the CRANAPL tool may promote reflection on the part of the admitting physician. The estimations of LOS, potential for upgrade to a higher level of care, and disposition are markers of optimal inpatient care, especially for hospitalists who work in shifts with embedded handoffs. When shared with colleagues (through documentation), there is the potential for distributed cognition¹⁰ to extend throughout the social network of the hospitalist group. The fact that so few providers are currently including these items in their A&P’s show that the providers are either not performing or documenting the ‘reasoning’. Either way, this is an opportunity that has been highlighted by the CRANAPL tool.

Several limitations of this study should be considered. First, the CRANAPL tool may not have captured elements of optimal clinical reasoning documentation. The reliance on multiple methods and an iterative process in the refinement of the CRANAPL tool should have minimized this. Second, this study was conducted across a single healthcare system that uses the same EHR; this EHR or institutional culture may influence documentation practices and behaviors. Given that using the CRANAPL tool to score an A&P is quick and easy, the benefit of giving providers feedback on their notes remains to be seen—here and at other hospitals. Third, our sample size could limit the generalizability of the results and the significance of the associations. However, the sample assessed in our study was significantly larger than that assessed in other studies that have validated clinical reasoning rubrics.^14,15 Fourth, clinical reasoning is a broad and multidimensional construct. The CRANAPL tool focuses exclusively on hospitalists’ documentation of clinical reasoning and therefore does not assess aspects of clinical reasoning occurring in the physicians’ minds. Finally, given our goal to optimally validate the CRANAPL tool, we chose to test the tool on specific presentations that are known to be associated with diagnostic practice variation and errors. We may have observed different results had we chosen a different set of diagnoses from each hospital. Further validity evidence will be established when applying the CRANPL tool to different diagnoses and to notes from other clinical settings.

In conclusion, this study focuses on the development and validation of the CRANAPL tool that assesses how hospitalists document their clinical reasoning in the A&P section of admission notes. Our results show that wide variability exists in the documentation of clinical reasoning by hospitalists within and across hospitals. Given the CRANAPL tool’s ease-of-use and its versatility, hospitalist divisions in academic and nonacademic settings may use the CRANAPL tool to assess and provide feedback on the documentation of hospitalists’ clinical reasoning. Beyond studying whether physicians can be taught to improve their notes with feedback based on the CRANAPL tool, future studies may explore whether enhancing clinical reasoning documentation may be associated with improvements in patient care and clinical outcomes.

Acknowledgments

Dr. Wright is the Anne Gaines and G. Thomas Miller Professor of Medicine which is supported through Hopkins’ Center for Innovative Medicine.

The authors thank Christine Caufield-Noll, MLIS, AHIP (Johns Hopkins Bayview Medical Center, Baltimore, Maryland) for her assistance with this project.

Disclosures

The authors have nothing to disclose.

ABSTRACT

Purpose To avoid disruption of administrative and clinical workflow in an increasingly complex system of health information technology, health care systems and providers have started using medical scribes. The purpose of this study was to investigate the impact of medical scribes on patient satisfaction, physician satisfaction, and quality measure documentation in a family medicine office.

Methods We reviewed 1000 electronic health records for documentation of specified quality measures in the family medicine setting, before and after the use of medical scribes. We surveyed 150 patients on attitude, comfort, and acceptance of medical scribes during their visit. Five physicians shared their perceptions related to productivity, efficiency, and overall job satisfaction on working with medical scribes.

Results Documentation of 4 quality measures improved with the use of scribes, demonstrating statistical significance: fall risk assessment (odds ratio [OR] = 5.5; P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Patients reported comfort with scribes in the examination room (96%, n = 144), a more focused health care provider (76%, n = 113), increased efficiency (74%, n = 109), and a higher degree of satisfaction with the office visit (61%, n = 90). Physicians believed they were providing better care and developing better relationships with patients while spending less time documenting and experiencing less stress.

Conclusions Use of medical scribes in a primary care setting was associated with higher patient and physician satisfaction. Patients felt comfortable with a medical scribe in the room, attested to their professionalism, and understood their purpose during the visit. The use of medical scribes in this primary care setting improved documentation of 4 quality measures.

[polldaddy:10339849]

The widespread implementation and adoption of electronic health records (EHRs) continues to increase, primarily motivated by federal incentives through the Centers for Medicare and Medicaid Services to positively impact patient care. Physician use of the EHR in the exam room has the potential to affect the patient-physician relationship, patient satisfaction, physician satisfaction, physician productivity, and physician reimbursement. In the United States, the Health Information Technology for Economic and Clinical Health Act of 2009 established incentive programs to promote meaningful use of EHRs in primary care.¹ Integrating EHRs into physician practice, adoption of meaningful use, and the increasing challenge of pay-for-performance quality measures have generated additional hours of administrative work for health care providers. These intrusions on routine clinical care, while hypothesized to improve care, have diminished physician satisfaction, increased stress, and contributed to physician burnout.²

The expanded role of clinicians incentivized to capture metrics for value-based care introduces an unprecedented level of multitasking required at the point of care. In a clinical setting, multitasking undermines the core clinical activities of observation, communication, problem solving, and, ultimately, the development of trusting relationships.^3,4 EHR documentation creates a barrier to patient engagement and may contribute to patients feeling isolated when unable to view data being entered.^5,6

Potential benefits of scribes. One means of increasing physician satisfaction and productivity may be the integration of medical scribes into health care systems. Medical scribes do not operate independently but are able to document activities or receive dictation critical for patient management—eg, recording patient histories, documenting physical examination findings and procedures, and following up on lab reports.⁷

Continue to: In a 2015 systematic review...

In a 2015 systematic review, Shultz and Holmstrom found that medical scribes in specialty settings may improve clinician satisfaction, productivity, time-related efficiency, revenue, and patient-clinician interactions.⁸ The use of scribes in one study increased the number of patients seen and time saved by emergency physicians, thereby increasing physician productivity.⁹ Studies have also shown that physicians were more satisfied during scribe engagement, related to increased time spent with patients, decreased work-related stress, and increased overall workplace satisfaction.^10-12

Sixty-one percent of patients were more satisfied with their office visit with a scribe present.

Studies on the use of medical scribes have mainly focused on physician satisfaction and productivity; however, the data on patient satisfaction are limited. Data about the use of the medical scribe in the primary care setting are also limited. The aim of our research was threefold. We wanted to evaluate the effects of using a medical scribe on: (1) patient satisfaction, (2) documentation of primary care pay-for-performance quality measures, and (3) physicians’ perceptions of the use of scribes in the primary care setting.

METHODS

Data collection

This study was conducted at Family Practice Group in Arlington, Massachusetts, where 5 part-time physicians and 3 full-time physician assistants see approximately 400 patients each week. The representative patient population is approximately 80% privately insured, 10% Medicaid, and 10% Medicare. The EHR system is eClinicalWorks.

The scribes were undergraduate college students who were interested in careers as health care professionals. They had no scribe training or experience working in a medical office. These scribes underwent 4 hours of training in EHR functionality, pay-for-performance quality measures, and risk coding (using appropriate medical codes that capture the patient’s level of medical complexity). The Independent Physician Association affiliated with Family Practice Group provided this training at no cost to the practice. The 3 scribes worked full-time with the 5 part-time physicians in the study. Scribes were not required to have had a medical background prior to entering the program.

After the aforementioned training, scribes began working full-time with physicians during patient visits and continued learning on the job through feedback from supervising physicians. Scribes documented the patient encounters, recording medical and social histories and physical exam findings, and transcribing discussions of treatment plans and physicians’ instructions to patients.

Continue to: We reviewed patient EHRs...

We reviewed patient EHRs of 5 family physicians over 2 time periods: the 3 months prior to having a medical scribe and the 3 months after beginning to work with a medical scribe. Chart data extraction occurred from 4/11/13 to 8/28/14. We reviewed 1000 patient EHRs—100 EHRs each for the 5 participating physicians before and after scribe use. Selected EHRs ran chronologically from the start of each 3-month period. Reviewing EHRs at 3 months after the onset of the medical scribe program allowed time for the scribes to be fully integrated into the practice and confident in their job responsibilities. Chart review was performed by an office administrator who was blinded as to whether documentation had been done with or without a scribe present during the visit.

Eight quality measures were evaluated in chart review. These measures were drawn from the Healthcare Effectiveness Data and Information Set (HEDIS), a tool used to measure performance in medical care and service.

We surveyed 30 patients of each of the 5 providers, yielding a total of 150 survey responses. A medical assistant gave surveys to patients in the exam room following each office visit, to be completed anonymously and privately. Patients were told that surveys would take less than 2 minutes to complete. Office visits included episodic visits, physical exams, and chronic disease management.

Physicians believed they were saving, on average, 1.5 hours each day with the use of a scribe.

After the trial period, we surveyed participating physicians regarding medical scribe assistance with documentation. We also asked the physicians 3 open-ended questions regarding their experiences with their medical scribe.

This study was reviewed and approved (IRB Approval #11424) by the Tufts Health Science Campus Institutional Review Board.

Continue to: Data analysis

During chart review, we assessed the rate at which documentation was completed for 8 quality outcome measures commonly used in the primary care setting (TABLE 1), before and after the introduction of medical scribes. These quality measures and pertinent descriptors are listed in TABLE 2.¹³ Presence or absence of documentation on all quality measures was noted for all applicable patients.

One hundred fifty patients were surveyed immediately after their office visit on their perceptions of medical scribes, including their attitude toward, comfort with, and acceptance of medical scribes (TABLE 3). Five participating physicians were surveyed to assess their perceptions related to productivity and job satisfaction with the use of medical scribes (TABLE 4), and regarding time saved and additional patients seen. Those who collected and analyzed the data from the surveys were blinded to patient and physician identifiers.

Statistical analysis

Using chi-squared tests, we compared the number of positive documentations for the 8 outcome measures before and after the use of medical scribes. Two-sided P values < .05 were considered statistically significant. All statistical analyses were performed with the use of STATA version 9 (StataCorp LP. College Station, Tex).

Physician survey data were calculated on a Likert scale, with a score of 1 corresponding to “strongly disagree,” 2 “disagree,” 3 “neither agree nor disagree,” 4 “agree,” and 5 “strongly agree.” Using the 5 answers generated from the 5 physicians, we calculated the mean for each question.

RESULTS

The use of scribes demonstrated a statistically significant improvement in the documentation of 4 (out of 8) pay-for-performance measures (TABLE 1): fall risk assessment (odds ratio [OR] = 5.5, P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index (BMI) plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Sample sizes of each quality measure vary as there were differing numbers of applicable patients for each quality measure within the overall 1000 charts.

Continue to: We established at the beginning...

We established at the beginning of the study a target of obtaining surveys from 30 patients of each of the 5 physicians (total of 150). Response rates for surveys were 100% for both the 150 patients and the 5 physicians. No patients declined to complete the survey, although some did not answer every question.

Patients generally had positive experiences with medical scribes (TABLE 3). The majority of patients (96%, n = 144) felt comfortable with the scribe in the room during the visit with their provider. Patients felt that the provider focused on them “a little to a lot more” (75.8%, n = 113) and thought their visit was more efficient (73.6%, n = 109) as a result of the scribe being present vs not being present. Most patients were more satisfied with their office visit with the scribe being present (60.8%, n = 90).

Physicians felt that working with a medical scribe helped them connect with their patients, made patients feel that their physician was more attentive to them, contributed to better patient care, decreased the time they spent documenting in EHR, and contributed to faster work flow (TABLE 4). The physicians also believed they had saved a mean of 1.5 hours each day with the use of a medical scribe, and that they did not have to change their schedule in any way to accommodate additional patients as a result of having a scribe.

DISCUSSION

Documentation of fall risk assessment, follow-up tobacco screening, follow-up BMI plan, and follow-up blood pressure plan all demonstrated statistically significant increases with the use of medical scribes compared with practice before scribes. Follow-up depression screen and transition of care management had relatively high ORs (3.2 and 8, respectively), but did not yield statistically significant values, in part due to small sample sizes as the number of patients who were hospitalized and the number of patients who screened positive for depression were relatively small out of the total group of 1000 patients. The use of scribes had little effect on depression screen and tobacco screen. This is likely due to the fact that there were already effective office systems in place at the practice that alerted medical assistants to complete these screens for each appropriate patient.

We found that the use of medical scribes in a primary care setting was associated with both higher patient and physician satisfaction. Although the 5 physicians in this study chose not to see additional patients when using a medical scribe, they believed they were saving, on average, 1.5 hours of time each day with the use of a scribe. All 5 physicians reported that medical scribes enabled them to provide better patient care and to help patients feel as though they had more of the physician’s attention. Patient respondents attested to their provider focusing on them more during the visit. According to patient surveys, 40.4% of respondents felt that physicians addressed their concerns more thoroughly during the visit, while the remainder of patients did not.

Continue to: Some concerns...

Some concerns of introducing medical scribes into a health care system include possible patient discomfort with a third party being present during the visit and the cost of employing medical scribes. In this study, the vast majority of patients (96%) felt comfortable with a scribe in the room. Future research could compare patient discomfort due to the presence of a medical scribe with patient discomfort due to a physician using a computer during the visit.

Limitations of this study include the small sample size of both physicians and patients; a lack of validated measures for calculating productivity, time/efficiency, and overall satisfaction; and short time periods leading up to and following the introduction of medical scribes. In addition, EHRs of patients were chosen sequentially and not randomly, which could be a confounder. Participating physicians were aware of being studied; therefore, documentation could have been affected by the Hawthorne effect. The study also was limited to one family medicine site. Although improved documentation of primary care pay-for-performance quality measures was reported, wide confidence intervals and small patient numbers hindered generalizability of findings.

Documentation of 4 out of 8 pay-forperformance measures showed statistically significant improvement with the use of scribes.

Additional studies are needed with a robust analytic plan sufficient to demonstrate baseline provider familiarity with EHRs, accuracy of medical scribe documentation, and improved documentation of pay-for-performance quality measures. Additional investigation regarding the variable competency of different medical scribes could be useful in measuring the effects of the scribe on a variety of outcomes related to both the physician and patient.

It is possible that the improved documentation yielded by the use of medical scribes could generate billing codes that reimburse physicians at a higher level (eg, a higher ratio of 99214 to 99213), leading to increased pay. Future research could aim to quantify this source of increased revenue. Furthermore, investigations could aim to quantify the revenue that medical scribes generate via improved quality measure pay-for-performance documentation.

CORRESPONDENCE
Jessica Platt, MD, 195 Canal Street, Malden, MA 02148; [email protected].

ABSTRACT

Purpose To avoid disruption of administrative and clinical workflow in an increasingly complex system of health information technology, health care systems and providers have started using medical scribes. The purpose of this study was to investigate the impact of medical scribes on patient satisfaction, physician satisfaction, and quality measure documentation in a family medicine office.

Methods We reviewed 1000 electronic health records for documentation of specified quality measures in the family medicine setting, before and after the use of medical scribes. We surveyed 150 patients on attitude, comfort, and acceptance of medical scribes during their visit. Five physicians shared their perceptions related to productivity, efficiency, and overall job satisfaction on working with medical scribes.

Results Documentation of 4 quality measures improved with the use of scribes, demonstrating statistical significance: fall risk assessment (odds ratio [OR] = 5.5; P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Patients reported comfort with scribes in the examination room (96%, n = 144), a more focused health care provider (76%, n = 113), increased efficiency (74%, n = 109), and a higher degree of satisfaction with the office visit (61%, n = 90). Physicians believed they were providing better care and developing better relationships with patients while spending less time documenting and experiencing less stress.

Conclusions Use of medical scribes in a primary care setting was associated with higher patient and physician satisfaction. Patients felt comfortable with a medical scribe in the room, attested to their professionalism, and understood their purpose during the visit. The use of medical scribes in this primary care setting improved documentation of 4 quality measures.

[polldaddy:10339849]

The widespread implementation and adoption of electronic health records (EHRs) continues to increase, primarily motivated by federal incentives through the Centers for Medicare and Medicaid Services to positively impact patient care. Physician use of the EHR in the exam room has the potential to affect the patient-physician relationship, patient satisfaction, physician satisfaction, physician productivity, and physician reimbursement. In the United States, the Health Information Technology for Economic and Clinical Health Act of 2009 established incentive programs to promote meaningful use of EHRs in primary care.¹ Integrating EHRs into physician practice, adoption of meaningful use, and the increasing challenge of pay-for-performance quality measures have generated additional hours of administrative work for health care providers. These intrusions on routine clinical care, while hypothesized to improve care, have diminished physician satisfaction, increased stress, and contributed to physician burnout.²

The expanded role of clinicians incentivized to capture metrics for value-based care introduces an unprecedented level of multitasking required at the point of care. In a clinical setting, multitasking undermines the core clinical activities of observation, communication, problem solving, and, ultimately, the development of trusting relationships.^3,4 EHR documentation creates a barrier to patient engagement and may contribute to patients feeling isolated when unable to view data being entered.^5,6

Potential benefits of scribes. One means of increasing physician satisfaction and productivity may be the integration of medical scribes into health care systems. Medical scribes do not operate independently but are able to document activities or receive dictation critical for patient management—eg, recording patient histories, documenting physical examination findings and procedures, and following up on lab reports.⁷

Continue to: In a 2015 systematic review...

In a 2015 systematic review, Shultz and Holmstrom found that medical scribes in specialty settings may improve clinician satisfaction, productivity, time-related efficiency, revenue, and patient-clinician interactions.⁸ The use of scribes in one study increased the number of patients seen and time saved by emergency physicians, thereby increasing physician productivity.⁹ Studies have also shown that physicians were more satisfied during scribe engagement, related to increased time spent with patients, decreased work-related stress, and increased overall workplace satisfaction.^10-12

Sixty-one percent of patients were more satisfied with their office visit with a scribe present.

Studies on the use of medical scribes have mainly focused on physician satisfaction and productivity; however, the data on patient satisfaction are limited. Data about the use of the medical scribe in the primary care setting are also limited. The aim of our research was threefold. We wanted to evaluate the effects of using a medical scribe on: (1) patient satisfaction, (2) documentation of primary care pay-for-performance quality measures, and (3) physicians’ perceptions of the use of scribes in the primary care setting.

METHODS

Data collection

This study was conducted at Family Practice Group in Arlington, Massachusetts, where 5 part-time physicians and 3 full-time physician assistants see approximately 400 patients each week. The representative patient population is approximately 80% privately insured, 10% Medicaid, and 10% Medicare. The EHR system is eClinicalWorks.

The scribes were undergraduate college students who were interested in careers as health care professionals. They had no scribe training or experience working in a medical office. These scribes underwent 4 hours of training in EHR functionality, pay-for-performance quality measures, and risk coding (using appropriate medical codes that capture the patient’s level of medical complexity). The Independent Physician Association affiliated with Family Practice Group provided this training at no cost to the practice. The 3 scribes worked full-time with the 5 part-time physicians in the study. Scribes were not required to have had a medical background prior to entering the program.

After the aforementioned training, scribes began working full-time with physicians during patient visits and continued learning on the job through feedback from supervising physicians. Scribes documented the patient encounters, recording medical and social histories and physical exam findings, and transcribing discussions of treatment plans and physicians’ instructions to patients.

Continue to: We reviewed patient EHRs...

We reviewed patient EHRs of 5 family physicians over 2 time periods: the 3 months prior to having a medical scribe and the 3 months after beginning to work with a medical scribe. Chart data extraction occurred from 4/11/13 to 8/28/14. We reviewed 1000 patient EHRs—100 EHRs each for the 5 participating physicians before and after scribe use. Selected EHRs ran chronologically from the start of each 3-month period. Reviewing EHRs at 3 months after the onset of the medical scribe program allowed time for the scribes to be fully integrated into the practice and confident in their job responsibilities. Chart review was performed by an office administrator who was blinded as to whether documentation had been done with or without a scribe present during the visit.

Eight quality measures were evaluated in chart review. These measures were drawn from the Healthcare Effectiveness Data and Information Set (HEDIS), a tool used to measure performance in medical care and service.

We surveyed 30 patients of each of the 5 providers, yielding a total of 150 survey responses. A medical assistant gave surveys to patients in the exam room following each office visit, to be completed anonymously and privately. Patients were told that surveys would take less than 2 minutes to complete. Office visits included episodic visits, physical exams, and chronic disease management.

Physicians believed they were saving, on average, 1.5 hours each day with the use of a scribe.

After the trial period, we surveyed participating physicians regarding medical scribe assistance with documentation. We also asked the physicians 3 open-ended questions regarding their experiences with their medical scribe.

This study was reviewed and approved (IRB Approval #11424) by the Tufts Health Science Campus Institutional Review Board.

Continue to: Data analysis

During chart review, we assessed the rate at which documentation was completed for 8 quality outcome measures commonly used in the primary care setting (TABLE 1), before and after the introduction of medical scribes. These quality measures and pertinent descriptors are listed in TABLE 2.¹³ Presence or absence of documentation on all quality measures was noted for all applicable patients.

One hundred fifty patients were surveyed immediately after their office visit on their perceptions of medical scribes, including their attitude toward, comfort with, and acceptance of medical scribes (TABLE 3). Five participating physicians were surveyed to assess their perceptions related to productivity and job satisfaction with the use of medical scribes (TABLE 4), and regarding time saved and additional patients seen. Those who collected and analyzed the data from the surveys were blinded to patient and physician identifiers.

Statistical analysis

Using chi-squared tests, we compared the number of positive documentations for the 8 outcome measures before and after the use of medical scribes. Two-sided P values < .05 were considered statistically significant. All statistical analyses were performed with the use of STATA version 9 (StataCorp LP. College Station, Tex).

Physician survey data were calculated on a Likert scale, with a score of 1 corresponding to “strongly disagree,” 2 “disagree,” 3 “neither agree nor disagree,” 4 “agree,” and 5 “strongly agree.” Using the 5 answers generated from the 5 physicians, we calculated the mean for each question.

RESULTS

The use of scribes demonstrated a statistically significant improvement in the documentation of 4 (out of 8) pay-for-performance measures (TABLE 1): fall risk assessment (odds ratio [OR] = 5.5, P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index (BMI) plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Sample sizes of each quality measure vary as there were differing numbers of applicable patients for each quality measure within the overall 1000 charts.

Continue to: We established at the beginning...

We established at the beginning of the study a target of obtaining surveys from 30 patients of each of the 5 physicians (total of 150). Response rates for surveys were 100% for both the 150 patients and the 5 physicians. No patients declined to complete the survey, although some did not answer every question.

Patients generally had positive experiences with medical scribes (TABLE 3). The majority of patients (96%, n = 144) felt comfortable with the scribe in the room during the visit with their provider. Patients felt that the provider focused on them “a little to a lot more” (75.8%, n = 113) and thought their visit was more efficient (73.6%, n = 109) as a result of the scribe being present vs not being present. Most patients were more satisfied with their office visit with the scribe being present (60.8%, n = 90).

Physicians felt that working with a medical scribe helped them connect with their patients, made patients feel that their physician was more attentive to them, contributed to better patient care, decreased the time they spent documenting in EHR, and contributed to faster work flow (TABLE 4). The physicians also believed they had saved a mean of 1.5 hours each day with the use of a medical scribe, and that they did not have to change their schedule in any way to accommodate additional patients as a result of having a scribe.

DISCUSSION

Documentation of fall risk assessment, follow-up tobacco screening, follow-up BMI plan, and follow-up blood pressure plan all demonstrated statistically significant increases with the use of medical scribes compared with practice before scribes. Follow-up depression screen and transition of care management had relatively high ORs (3.2 and 8, respectively), but did not yield statistically significant values, in part due to small sample sizes as the number of patients who were hospitalized and the number of patients who screened positive for depression were relatively small out of the total group of 1000 patients. The use of scribes had little effect on depression screen and tobacco screen. This is likely due to the fact that there were already effective office systems in place at the practice that alerted medical assistants to complete these screens for each appropriate patient.

We found that the use of medical scribes in a primary care setting was associated with both higher patient and physician satisfaction. Although the 5 physicians in this study chose not to see additional patients when using a medical scribe, they believed they were saving, on average, 1.5 hours of time each day with the use of a scribe. All 5 physicians reported that medical scribes enabled them to provide better patient care and to help patients feel as though they had more of the physician’s attention. Patient respondents attested to their provider focusing on them more during the visit. According to patient surveys, 40.4% of respondents felt that physicians addressed their concerns more thoroughly during the visit, while the remainder of patients did not.

Continue to: Some concerns...

Some concerns of introducing medical scribes into a health care system include possible patient discomfort with a third party being present during the visit and the cost of employing medical scribes. In this study, the vast majority of patients (96%) felt comfortable with a scribe in the room. Future research could compare patient discomfort due to the presence of a medical scribe with patient discomfort due to a physician using a computer during the visit.

Limitations of this study include the small sample size of both physicians and patients; a lack of validated measures for calculating productivity, time/efficiency, and overall satisfaction; and short time periods leading up to and following the introduction of medical scribes. In addition, EHRs of patients were chosen sequentially and not randomly, which could be a confounder. Participating physicians were aware of being studied; therefore, documentation could have been affected by the Hawthorne effect. The study also was limited to one family medicine site. Although improved documentation of primary care pay-for-performance quality measures was reported, wide confidence intervals and small patient numbers hindered generalizability of findings.

Documentation of 4 out of 8 pay-forperformance measures showed statistically significant improvement with the use of scribes.

Additional studies are needed with a robust analytic plan sufficient to demonstrate baseline provider familiarity with EHRs, accuracy of medical scribe documentation, and improved documentation of pay-for-performance quality measures. Additional investigation regarding the variable competency of different medical scribes could be useful in measuring the effects of the scribe on a variety of outcomes related to both the physician and patient.

It is possible that the improved documentation yielded by the use of medical scribes could generate billing codes that reimburse physicians at a higher level (eg, a higher ratio of 99214 to 99213), leading to increased pay. Future research could aim to quantify this source of increased revenue. Furthermore, investigations could aim to quantify the revenue that medical scribes generate via improved quality measure pay-for-performance documentation.

CORRESPONDENCE
Jessica Platt, MD, 195 Canal Street, Malden, MA 02148; [email protected].

ABSTRACT

Purpose To avoid disruption of administrative and clinical workflow in an increasingly complex system of health information technology, health care systems and providers have started using medical scribes. The purpose of this study was to investigate the impact of medical scribes on patient satisfaction, physician satisfaction, and quality measure documentation in a family medicine office.

Methods We reviewed 1000 electronic health records for documentation of specified quality measures in the family medicine setting, before and after the use of medical scribes. We surveyed 150 patients on attitude, comfort, and acceptance of medical scribes during their visit. Five physicians shared their perceptions related to productivity, efficiency, and overall job satisfaction on working with medical scribes.

Results Documentation of 4 quality measures improved with the use of scribes, demonstrating statistical significance: fall risk assessment (odds ratio [OR] = 5.5; P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Patients reported comfort with scribes in the examination room (96%, n = 144), a more focused health care provider (76%, n = 113), increased efficiency (74%, n = 109), and a higher degree of satisfaction with the office visit (61%, n = 90). Physicians believed they were providing better care and developing better relationships with patients while spending less time documenting and experiencing less stress.

Conclusions Use of medical scribes in a primary care setting was associated with higher patient and physician satisfaction. Patients felt comfortable with a medical scribe in the room, attested to their professionalism, and understood their purpose during the visit. The use of medical scribes in this primary care setting improved documentation of 4 quality measures.

[polldaddy:10339849]

The widespread implementation and adoption of electronic health records (EHRs) continues to increase, primarily motivated by federal incentives through the Centers for Medicare and Medicaid Services to positively impact patient care. Physician use of the EHR in the exam room has the potential to affect the patient-physician relationship, patient satisfaction, physician satisfaction, physician productivity, and physician reimbursement. In the United States, the Health Information Technology for Economic and Clinical Health Act of 2009 established incentive programs to promote meaningful use of EHRs in primary care.¹ Integrating EHRs into physician practice, adoption of meaningful use, and the increasing challenge of pay-for-performance quality measures have generated additional hours of administrative work for health care providers. These intrusions on routine clinical care, while hypothesized to improve care, have diminished physician satisfaction, increased stress, and contributed to physician burnout.²

The expanded role of clinicians incentivized to capture metrics for value-based care introduces an unprecedented level of multitasking required at the point of care. In a clinical setting, multitasking undermines the core clinical activities of observation, communication, problem solving, and, ultimately, the development of trusting relationships.^3,4 EHR documentation creates a barrier to patient engagement and may contribute to patients feeling isolated when unable to view data being entered.^5,6

Potential benefits of scribes. One means of increasing physician satisfaction and productivity may be the integration of medical scribes into health care systems. Medical scribes do not operate independently but are able to document activities or receive dictation critical for patient management—eg, recording patient histories, documenting physical examination findings and procedures, and following up on lab reports.⁷

Continue to: In a 2015 systematic review...

In a 2015 systematic review, Shultz and Holmstrom found that medical scribes in specialty settings may improve clinician satisfaction, productivity, time-related efficiency, revenue, and patient-clinician interactions.⁸ The use of scribes in one study increased the number of patients seen and time saved by emergency physicians, thereby increasing physician productivity.⁹ Studies have also shown that physicians were more satisfied during scribe engagement, related to increased time spent with patients, decreased work-related stress, and increased overall workplace satisfaction.^10-12

Sixty-one percent of patients were more satisfied with their office visit with a scribe present.

Studies on the use of medical scribes have mainly focused on physician satisfaction and productivity; however, the data on patient satisfaction are limited. Data about the use of the medical scribe in the primary care setting are also limited. The aim of our research was threefold. We wanted to evaluate the effects of using a medical scribe on: (1) patient satisfaction, (2) documentation of primary care pay-for-performance quality measures, and (3) physicians’ perceptions of the use of scribes in the primary care setting.

METHODS

Data collection

This study was conducted at Family Practice Group in Arlington, Massachusetts, where 5 part-time physicians and 3 full-time physician assistants see approximately 400 patients each week. The representative patient population is approximately 80% privately insured, 10% Medicaid, and 10% Medicare. The EHR system is eClinicalWorks.

The scribes were undergraduate college students who were interested in careers as health care professionals. They had no scribe training or experience working in a medical office. These scribes underwent 4 hours of training in EHR functionality, pay-for-performance quality measures, and risk coding (using appropriate medical codes that capture the patient’s level of medical complexity). The Independent Physician Association affiliated with Family Practice Group provided this training at no cost to the practice. The 3 scribes worked full-time with the 5 part-time physicians in the study. Scribes were not required to have had a medical background prior to entering the program.

After the aforementioned training, scribes began working full-time with physicians during patient visits and continued learning on the job through feedback from supervising physicians. Scribes documented the patient encounters, recording medical and social histories and physical exam findings, and transcribing discussions of treatment plans and physicians’ instructions to patients.

Continue to: We reviewed patient EHRs...

We reviewed patient EHRs of 5 family physicians over 2 time periods: the 3 months prior to having a medical scribe and the 3 months after beginning to work with a medical scribe. Chart data extraction occurred from 4/11/13 to 8/28/14. We reviewed 1000 patient EHRs—100 EHRs each for the 5 participating physicians before and after scribe use. Selected EHRs ran chronologically from the start of each 3-month period. Reviewing EHRs at 3 months after the onset of the medical scribe program allowed time for the scribes to be fully integrated into the practice and confident in their job responsibilities. Chart review was performed by an office administrator who was blinded as to whether documentation had been done with or without a scribe present during the visit.

Eight quality measures were evaluated in chart review. These measures were drawn from the Healthcare Effectiveness Data and Information Set (HEDIS), a tool used to measure performance in medical care and service.

We surveyed 30 patients of each of the 5 providers, yielding a total of 150 survey responses. A medical assistant gave surveys to patients in the exam room following each office visit, to be completed anonymously and privately. Patients were told that surveys would take less than 2 minutes to complete. Office visits included episodic visits, physical exams, and chronic disease management.

Physicians believed they were saving, on average, 1.5 hours each day with the use of a scribe.

After the trial period, we surveyed participating physicians regarding medical scribe assistance with documentation. We also asked the physicians 3 open-ended questions regarding their experiences with their medical scribe.

This study was reviewed and approved (IRB Approval #11424) by the Tufts Health Science Campus Institutional Review Board.

Continue to: Data analysis

During chart review, we assessed the rate at which documentation was completed for 8 quality outcome measures commonly used in the primary care setting (TABLE 1), before and after the introduction of medical scribes. These quality measures and pertinent descriptors are listed in TABLE 2.¹³ Presence or absence of documentation on all quality measures was noted for all applicable patients.

One hundred fifty patients were surveyed immediately after their office visit on their perceptions of medical scribes, including their attitude toward, comfort with, and acceptance of medical scribes (TABLE 3). Five participating physicians were surveyed to assess their perceptions related to productivity and job satisfaction with the use of medical scribes (TABLE 4), and regarding time saved and additional patients seen. Those who collected and analyzed the data from the surveys were blinded to patient and physician identifiers.

Statistical analysis

Using chi-squared tests, we compared the number of positive documentations for the 8 outcome measures before and after the use of medical scribes. Two-sided P values < .05 were considered statistically significant. All statistical analyses were performed with the use of STATA version 9 (StataCorp LP. College Station, Tex).

Physician survey data were calculated on a Likert scale, with a score of 1 corresponding to “strongly disagree,” 2 “disagree,” 3 “neither agree nor disagree,” 4 “agree,” and 5 “strongly agree.” Using the 5 answers generated from the 5 physicians, we calculated the mean for each question.

RESULTS

The use of scribes demonstrated a statistically significant improvement in the documentation of 4 (out of 8) pay-for-performance measures (TABLE 1): fall risk assessment (odds ratio [OR] = 5.5, P = .02), follow-up tobacco screen (OR = 6.4; P = .01), follow-up body mass index (BMI) plan (OR = 6.2; P < .01), and follow-up blood pressure plan (OR = 39.6; P < .01). Sample sizes of each quality measure vary as there were differing numbers of applicable patients for each quality measure within the overall 1000 charts.

Continue to: We established at the beginning...

We established at the beginning of the study a target of obtaining surveys from 30 patients of each of the 5 physicians (total of 150). Response rates for surveys were 100% for both the 150 patients and the 5 physicians. No patients declined to complete the survey, although some did not answer every question.

Patients generally had positive experiences with medical scribes (TABLE 3). The majority of patients (96%, n = 144) felt comfortable with the scribe in the room during the visit with their provider. Patients felt that the provider focused on them “a little to a lot more” (75.8%, n = 113) and thought their visit was more efficient (73.6%, n = 109) as a result of the scribe being present vs not being present. Most patients were more satisfied with their office visit with the scribe being present (60.8%, n = 90).

Physicians felt that working with a medical scribe helped them connect with their patients, made patients feel that their physician was more attentive to them, contributed to better patient care, decreased the time they spent documenting in EHR, and contributed to faster work flow (TABLE 4). The physicians also believed they had saved a mean of 1.5 hours each day with the use of a medical scribe, and that they did not have to change their schedule in any way to accommodate additional patients as a result of having a scribe.

DISCUSSION

Documentation of fall risk assessment, follow-up tobacco screening, follow-up BMI plan, and follow-up blood pressure plan all demonstrated statistically significant increases with the use of medical scribes compared with practice before scribes. Follow-up depression screen and transition of care management had relatively high ORs (3.2 and 8, respectively), but did not yield statistically significant values, in part due to small sample sizes as the number of patients who were hospitalized and the number of patients who screened positive for depression were relatively small out of the total group of 1000 patients. The use of scribes had little effect on depression screen and tobacco screen. This is likely due to the fact that there were already effective office systems in place at the practice that alerted medical assistants to complete these screens for each appropriate patient.

We found that the use of medical scribes in a primary care setting was associated with both higher patient and physician satisfaction. Although the 5 physicians in this study chose not to see additional patients when using a medical scribe, they believed they were saving, on average, 1.5 hours of time each day with the use of a scribe. All 5 physicians reported that medical scribes enabled them to provide better patient care and to help patients feel as though they had more of the physician’s attention. Patient respondents attested to their provider focusing on them more during the visit. According to patient surveys, 40.4% of respondents felt that physicians addressed their concerns more thoroughly during the visit, while the remainder of patients did not.

Continue to: Some concerns...

Some concerns of introducing medical scribes into a health care system include possible patient discomfort with a third party being present during the visit and the cost of employing medical scribes. In this study, the vast majority of patients (96%) felt comfortable with a scribe in the room. Future research could compare patient discomfort due to the presence of a medical scribe with patient discomfort due to a physician using a computer during the visit.

Limitations of this study include the small sample size of both physicians and patients; a lack of validated measures for calculating productivity, time/efficiency, and overall satisfaction; and short time periods leading up to and following the introduction of medical scribes. In addition, EHRs of patients were chosen sequentially and not randomly, which could be a confounder. Participating physicians were aware of being studied; therefore, documentation could have been affected by the Hawthorne effect. The study also was limited to one family medicine site. Although improved documentation of primary care pay-for-performance quality measures was reported, wide confidence intervals and small patient numbers hindered generalizability of findings.

Documentation of 4 out of 8 pay-forperformance measures showed statistically significant improvement with the use of scribes.

Additional studies are needed with a robust analytic plan sufficient to demonstrate baseline provider familiarity with EHRs, accuracy of medical scribe documentation, and improved documentation of pay-for-performance quality measures. Additional investigation regarding the variable competency of different medical scribes could be useful in measuring the effects of the scribe on a variety of outcomes related to both the physician and patient.

It is possible that the improved documentation yielded by the use of medical scribes could generate billing codes that reimburse physicians at a higher level (eg, a higher ratio of 99214 to 99213), leading to increased pay. Future research could aim to quantify this source of increased revenue. Furthermore, investigations could aim to quantify the revenue that medical scribes generate via improved quality measure pay-for-performance documentation.

CORRESPONDENCE
Jessica Platt, MD, 195 Canal Street, Malden, MA 02148; [email protected].

This article is part of a series that illustrates strategies intended to redesign primary care education at the Veterans Health Administration (VHA), using interprofessional workplace learning. All have been implemented in the VA Centers of Excellence in Primary Care Education (CoEPCE). These models embody visionary transformation of clinical and educational environments that have potential for replication and dissemination throughout VA and other primary care clinical educational environments. For an introduction to the series see Klink K. Transforming primary care clinical learning environments to optimize education, outcomes, and satisfaction. Fed Pract. 2018;35(9):8-10.

Background

In 2011, 5 US Department of Veterans Affairs (VA) medical centers were selected by the VA Office of Academic Affiliations (OAA) to establish Centers of Excellence in Primary Care Education (CoEPCE). Part of the New Models of Care initiative, the 5 CoEPCEs use VA primary care settings to develop and test innovative approaches to prepare physician residents, medical students, advanced practice registered nurses, undergraduate nursing students, and other health professions’ trainees, such as social workers, pharmacists, psychologists, and physician assistants, for improved primary care practice. The CoEPCEs are interprofessional Academic PACTs (iAPACTs) with ≥ 2 professions of trainees engaged in learning on the PACT team.

The VA Puget Sound Seattle CoEPCE curriculum is embedded in a well-established academic VA primary care training site.¹ Trainees include doctor of nursing practice (DNP) students in adult, family, and psychiatric mental health nurse practitioner (NP) programs; NP residents; internal medicine physician residents; postgraduate pharmacy residents; and other health professions’ trainees. A Seattle CoEPCE priority is to provide DNP students, DNP residents, and physician residents with a longitudinal experience in team-based care as well as interprofessional education and collaborative practice (IPECP). Learners spend the majority of CoEPCE time in supervised, direct patient care, including primary care, women’s health, deployment health, homeless care, and home care. Formal IPECP activities comprise about 20% of time, supported by 3 educational strategies: (1) Panel management (PM)/quality improvement (QI); (2) Team building/ communications; and (3) Clinical content seminars to expand trainee clinical knowledge and skills and curriculum developed with the CoEPCE enterprise core domains in mind (Table).

Panel Management

Clinicians are increasingly being required to proactively optimize the health of an assigned population of patients in addition to assessing and managing the health of individual patients presenting for care. To address the objectives of increased accountability for population health outcomes and improved face-to-face care, Seattle CoEPCE developed curriculum for trainees to learn PM, a set of tools and processes that can be applied in the primary care setting.

PM clinical providers use data to proactively provide care to their patients between traditional clinic visits. The process is proactive in that gaps are identified whether or not an in-person visit occurs and involves an outreach mechanism to increase continuity of care, such as follow-up communications with the patients.² PM also has been associated with improvements in chronic disease care.^3-5

The Seattle CoEPCE developed an interprofessional team approach to PM that teaches trainees about the tools and resources used to close the gaps in care, including the use of clinical team members as health care systems subject matter experts. CoEPCE trainees are taught to analyze the care they provide to their panel of veterans (eg, identifying patients who have not refilled chronic medications or those who use the emergency department [ED] for nonacute conditions) and take action to improve care. PM yields rich discussions on systems resources and processes and is easily applied to a range of health conditions as well as delivery system issues. PM gives learners the tools they can use to close these gaps, such as the expertise of their peers, clinical team, and specialists.⁶

Planning and Implementation

In addition to completing a literature review to determine the state of PM practice and models, CoEPCE faculty polled recent graduates inquiring about strategies they did not learn prior to graduation. Based on their responses, CoEPCE faculty identified 2 skill deficits: management of chronic diseases and proficiency with data and statistics about performance improvement in panel patient care over time. Addressing these unmet needs became the impetus for developing curriculum for conducting PM. Planning and launching the CoEPCE approach to PM took about 3 months and involved CoEPCE faculty, a data manager, and administrative support. The learning objectives of Seattle’s PM initiative are to:

• Promote preventive health and chronic disease care by use performance data;
• Develop individual- and populationfocused action plans;
• Work collaboratively, strategically, and effectively with an interprofessional care team; and
• Learn how to effectively use system resources.

Curriculum

The PM curriculum is a longitudinal, experiential approach to learning how to manage chronic diseases between visits by using patient data. It is designed for trainees in a continuity clinic to review the care of their patients on a regular basis. Seattle CoEPCE medicine residents are assigned patient panels, which increase from 70 patients in the first year to about 140 patients by the end of the third year. DNP postgraduate trainees are assigned an initial panel of 50 patients that increases incrementally over the year-long residency.

CoEPCE faculty determined the focus of PM sessions to be diabetes mellitus (DM), hypertension, obesity, chronic opioid therapy, and low-acuity ED use. Because PM sessions are designed to allow participants to identify systems issues that may affect multiple patients, some of these topics have expanded into QI projects. PM sessions run 2 to 3 hours per session and are held 4 to 6 times a year. Each session is repeated twice to accommodate diverse trainee schedules. PM participants must have their patient visit time blocked for each session (Appendix).

Faculty Roles and Development

PM faculty involved in any individual session may include a combination of a CoEPCE clinical pharmacy specialist, a registered nurse (RN) care manager, a social worker, a NP, a physician, a clinical psychologist, and a medicine outpatient chief resident (PGY4, termed clinician-teacher fellow at Seattle VA medical center). The chief resident is a medicine residency graduate and takes on teaching responsibilities depending on the topic of the session. The CoEPCE clinical pharmacist role varies depending on the session topic: They may facilitate the session or provide recommendations for medication management for individual cases. The RN care manager often knows the patients and brings a unique perspective that complements that of the primary care providers and ideally participates in every session. The patients of multiple RN care managers may be presented at each session, and it was not feasible to include all RN care managers in every session. After case discussions, trainees often communicated with the RN care managers about the case, using instant messaging, and CoEPCE provides other avenues for patient care discussion through huddles involving the provider, RN care manager, clinical pharmacist, and other clinical professions.

Resources

The primary resource required to support PM is an information technology (IT) system that provides relevant health outcome and health care utilization data on patients assigned to trainees. PM sessions include teaching trainees how to access patient data. Since discussion about the care of panel patients during the learning sessions often results in real-time adjustments in the care plan, modest administrative support required post-PM sessions, such as clerical scheduling of the requested clinic or telephone follow-up with the physician, nurse, or pharmacist.

Monitoring and Assessment

Panel performance is evaluated at each educational session. To assess the CoEPCE PM curriculum, participants provide feedback in 8 questions over 3 domains: trainee perception of curriculum content, confidence in performing PM involving completion of a PM workshop, and likelihood of using PM techniques in the future. CoEPCE faculty use the feedback to improve their instruction of panel management skill and develop new sessions that target additional population groups. Evaluation of the curriculum also includes monitoring of panel patients’ chronic disease measures.

Several partnerships have contributed to the success and integrations of PM into facility activities. First, having the primary care clinic director as a member of the Co- EPCE faculty has encouraged faculty and staff to operationalize and implement PM broadly by distributing data monthly to all clinic staff. Second, high facility staff interest outside the CoEPCE and primary care clinic has facilitated establishing communications outside the CoEPCE regarding clinic data.

Challenges and Solutions

Trainees at earlier academic levels often desire more instruction in clinical knowledge, such as treatment options for DM or goals of therapy in hypertension. In contrast, advanced trainees are able to review patient data, brainstorm, and optimize solutions. Seattle CoEPCE balances these different learning needs via a flexible approach to the 3-hour sessions. For example, advanced trainees progress from structured short lectures to informal sessions, which train them to perform PM on their own. In addition, the flexible design integrates trainees with diverse schedules, particularly among DNP students and residents, pharmacy residents, and physician residents. Some of this work falls on the RN care management team and administrative support staff.

Competing Priorities

The demand for direct patient care points to the importance of indirect patient care activities like PM to demonstrate improved results. Managing chronic conditions and matching appropriate services and resources should improve clinical outcomes and efficiency longterm. In the interim, it is important to note that PM demonstrates the continuous aspect of clinical care, particularly for trainees who have strict guidelines defining clinical care for the experiences to count toward eligibility for licensure. Additionally, PM results in trainees who are making decisions with VA patients and are more efficiently providing and supporting patient care. Therefore, it is critical to secure important resources, such as provider time for conducting PM.

Data Access

No single data system in VA covers the broad range of topics covered in the PM sessions, and not all trainees have their own assigned panels. For example, health professions students are not assigned a panel of patients. While they do not have access to panel data such as those generated by Primary Care Almanac in VSSC (a data source in the VA Support Service Center database),the Seattle CoEPCE data manager pulls a set of patient data from the students’ paired faculty preceptors’ panels for review. Thus they learn PM principles and strategies for improving patient care via PM as part of the unique VA longitudinal clinic experience and the opportunity to learn from a multidisciplinary team that is not available at other clinical sites. Postgraduate NP residents in CoEPCE training have their own panels of patients and thus the ability to directly access their panel performance data.

Success Factors

A key success factor includes CoEPCE faculty’s ability to develop and operationalize a panel management model that simultaneously aligns with the educational goals of an interprofessional education training program and supports VA adoption of the medical home or patient aligned care teams (PACT). The CoEPCE contributes staff expertise in accessing and reporting patient data, accessing appropriate teaching space, managing panels of patients with chronic diseases, and facilitating a team-based approach to care. Additionally, the CoEPCE brand is helpful for getting buy-in from the clinical and academic stakeholders necessary for moving PM forward.

Colocating CoEPCE trainees and faculty in the primary care clinic promotes team identity around the RN care managers and facilitated communications with non-CoEPCE clinical teams that have trainees from other professions. RN care managers serve as the locus of highquality PM since they share patient panels with the trainees and already track admissions, ED visits, and numerous chronic health care metrics. RN care managers offer a level of insight into chronic disease that other providers may not possess, such as the specific details on medication adherence and the impact of adverse effects (AEs) for that particular patient. RN care managers are able to teach about their team role and responsibilities, strengthening the model.

PM is an opportunity to expand CoEPCE interprofessional education capacity by creating colocation of different trainee and faculty professions during the PM sessions; the sharing of data with trainees; and sharing and reflecting on data, strengthening communications between professions and within the PACT. The Seattle CoEPCE now has systems in place that allow the RN care manager to send notes to a physician and DNP resident, and the resident is expected to respond. In addition, the PM approach provides experience with analyzing data to improve care in an interprofessional team setting, which is a requirement of the Accreditation Council for Graduate Medical Education.

Interprofessional Collaboration

PM sessions are intentionally designed to improve communication among team members and foster a team approach to care. PM sessions provide an opportunity for trainees and clinician faculty to be together and learn about each profession’s perspectives. For example, early in the process physician and DNP trainees learn about the importance of clinical pharmacists to the team who prescribe and make medication adjustments within their scope of practice as well as the importance of making appropriate pharmacy referrals. Additionally, the RN care manager and clinical pharmacy specialists who serve as faculty in the CoEPCE provide pertinent information on individual patients, increasing integration with the PACT. Finally, there is anecdotal evidence that faculty also are learning more about interprofessional education and expanding their own skills.

Clinical Performance

CoEPCE trainees, non-CoEPCE physician residents, and CoEPCE faculty participants regularly receive patient data with which they can proactively develop or amend a treatment plan between visits. PM has resulted in improved data sharing with providers. Instead of once a year, providers and clinic staff now receive patient data monthly on chronic conditions from the clinic director. Trainees on ambulatory rotations are expected to review their panel data at least a half day per week. CoEPCE staff evaluate trainee likelihood to use PM and ability to identify patients who benefit from team-based care.

At the population level of chronic disease management, preliminary evidence demonstrates that primary care clinic patient panels are increasingly within target for DM and blood pressure measures, as assessed by periodic clinical reports to providers. Some of the PM topics have resulted in systems-level improvements, such as reducing unnecessary ED use for nonacute conditions and better opioid prescription monitoring. Moreover, PM supports everyone working at the top of his/her professional capability. For example, the RN care manager has the impetus to initiate DM education with a particular patient.

Since CoEPCE began teaching PM, the Seattle primary care clinic has committed to the regular access and review of data. This has encouraged the alignment of standards of care for chronic disease management so that all care providers are working toward the same benchmark goals.

Patient Outcomes

At the individual level, PM provide a mechanism to systemically review trainee panel patients with out-of-target clinical measures, and develop new care approaches involving interprofessional strategies and problem solving. PM also helps identify patients who have missed follow-up, reducing the risk that patients with chronic care needs will be lost to clinical engagement if they are not reminded or do not pursue appointments. The PM-trained PACT reaches out to patients who might not otherwise get care before the next clinic visit and provides new care plans. Second, patients have the benefit of a team that manages their health needs. For example, including the clinical pharmacists in the PM sessions ensures timely identification of medication interactions and the potential AEs. Additionally, PM contributes to the care coordination model by involving individuals on the primary care team who know the patient. These members review the patient’s data between visits and initiate team-based changes to the care plan to improve care. More team members connect with a patient, resulting in more intense care and quicker follow-up to determine the effectiveness of a treatment plan.

PM topics have spun off QI projects resulting in new clinic processes and programs, including processes for managing wounds in primary care and to assure timely post-ED visit follow-ups. Areas for expansion include a follow-up QI project to reduce nonacute ED visits by patients on the homeless PACT panel and interventions for better management of care for women veterans with mental health needs. PM also has extended to non-Co- EPCE teams and to other clinic activities, such as strengthening huddles of team members specifically related to panel data and addressing selected patient cases between visits. Pharmacy residents and faculty are more involved in reviewing the panel before patients are seen to review medication lists and identify duplications.

The Future

Under stage 2 of the program, the Seattle CoEPCE intends to lead in the creation of a PM toolkit as well as a data access guide that will allow VA facilities with limited data management expertise to access chronic disease metrics. Second, the CoEPCE will continue its dissemination efforts locally to other residents in the internal medicine residency program in all of its continuity clinics. Additionally, there is high interest by DNP training programs to expand and export longitudinal training experience PM curriculum to non-VA based students.

This article is part of a series that illustrates strategies intended to redesign primary care education at the Veterans Health Administration (VHA), using interprofessional workplace learning. All have been implemented in the VA Centers of Excellence in Primary Care Education (CoEPCE). These models embody visionary transformation of clinical and educational environments that have potential for replication and dissemination throughout VA and other primary care clinical educational environments. For an introduction to the series see Klink K. Transforming primary care clinical learning environments to optimize education, outcomes, and satisfaction. Fed Pract. 2018;35(9):8-10.

Background

In 2011, 5 US Department of Veterans Affairs (VA) medical centers were selected by the VA Office of Academic Affiliations (OAA) to establish Centers of Excellence in Primary Care Education (CoEPCE). Part of the New Models of Care initiative, the 5 CoEPCEs use VA primary care settings to develop and test innovative approaches to prepare physician residents, medical students, advanced practice registered nurses, undergraduate nursing students, and other health professions’ trainees, such as social workers, pharmacists, psychologists, and physician assistants, for improved primary care practice. The CoEPCEs are interprofessional Academic PACTs (iAPACTs) with ≥ 2 professions of trainees engaged in learning on the PACT team.

The VA Puget Sound Seattle CoEPCE curriculum is embedded in a well-established academic VA primary care training site.¹ Trainees include doctor of nursing practice (DNP) students in adult, family, and psychiatric mental health nurse practitioner (NP) programs; NP residents; internal medicine physician residents; postgraduate pharmacy residents; and other health professions’ trainees. A Seattle CoEPCE priority is to provide DNP students, DNP residents, and physician residents with a longitudinal experience in team-based care as well as interprofessional education and collaborative practice (IPECP). Learners spend the majority of CoEPCE time in supervised, direct patient care, including primary care, women’s health, deployment health, homeless care, and home care. Formal IPECP activities comprise about 20% of time, supported by 3 educational strategies: (1) Panel management (PM)/quality improvement (QI); (2) Team building/ communications; and (3) Clinical content seminars to expand trainee clinical knowledge and skills and curriculum developed with the CoEPCE enterprise core domains in mind (Table).

Panel Management

Clinicians are increasingly being required to proactively optimize the health of an assigned population of patients in addition to assessing and managing the health of individual patients presenting for care. To address the objectives of increased accountability for population health outcomes and improved face-to-face care, Seattle CoEPCE developed curriculum for trainees to learn PM, a set of tools and processes that can be applied in the primary care setting.

PM clinical providers use data to proactively provide care to their patients between traditional clinic visits. The process is proactive in that gaps are identified whether or not an in-person visit occurs and involves an outreach mechanism to increase continuity of care, such as follow-up communications with the patients.² PM also has been associated with improvements in chronic disease care.^3-5

The Seattle CoEPCE developed an interprofessional team approach to PM that teaches trainees about the tools and resources used to close the gaps in care, including the use of clinical team members as health care systems subject matter experts. CoEPCE trainees are taught to analyze the care they provide to their panel of veterans (eg, identifying patients who have not refilled chronic medications or those who use the emergency department [ED] for nonacute conditions) and take action to improve care. PM yields rich discussions on systems resources and processes and is easily applied to a range of health conditions as well as delivery system issues. PM gives learners the tools they can use to close these gaps, such as the expertise of their peers, clinical team, and specialists.⁶

Planning and Implementation

In addition to completing a literature review to determine the state of PM practice and models, CoEPCE faculty polled recent graduates inquiring about strategies they did not learn prior to graduation. Based on their responses, CoEPCE faculty identified 2 skill deficits: management of chronic diseases and proficiency with data and statistics about performance improvement in panel patient care over time. Addressing these unmet needs became the impetus for developing curriculum for conducting PM. Planning and launching the CoEPCE approach to PM took about 3 months and involved CoEPCE faculty, a data manager, and administrative support. The learning objectives of Seattle’s PM initiative are to:

• Promote preventive health and chronic disease care by use performance data;
• Develop individual- and populationfocused action plans;
• Work collaboratively, strategically, and effectively with an interprofessional care team; and
• Learn how to effectively use system resources.

Curriculum

The PM curriculum is a longitudinal, experiential approach to learning how to manage chronic diseases between visits by using patient data. It is designed for trainees in a continuity clinic to review the care of their patients on a regular basis. Seattle CoEPCE medicine residents are assigned patient panels, which increase from 70 patients in the first year to about 140 patients by the end of the third year. DNP postgraduate trainees are assigned an initial panel of 50 patients that increases incrementally over the year-long residency.

CoEPCE faculty determined the focus of PM sessions to be diabetes mellitus (DM), hypertension, obesity, chronic opioid therapy, and low-acuity ED use. Because PM sessions are designed to allow participants to identify systems issues that may affect multiple patients, some of these topics have expanded into QI projects. PM sessions run 2 to 3 hours per session and are held 4 to 6 times a year. Each session is repeated twice to accommodate diverse trainee schedules. PM participants must have their patient visit time blocked for each session (Appendix).

Faculty Roles and Development

PM faculty involved in any individual session may include a combination of a CoEPCE clinical pharmacy specialist, a registered nurse (RN) care manager, a social worker, a NP, a physician, a clinical psychologist, and a medicine outpatient chief resident (PGY4, termed clinician-teacher fellow at Seattle VA medical center). The chief resident is a medicine residency graduate and takes on teaching responsibilities depending on the topic of the session. The CoEPCE clinical pharmacist role varies depending on the session topic: They may facilitate the session or provide recommendations for medication management for individual cases. The RN care manager often knows the patients and brings a unique perspective that complements that of the primary care providers and ideally participates in every session. The patients of multiple RN care managers may be presented at each session, and it was not feasible to include all RN care managers in every session. After case discussions, trainees often communicated with the RN care managers about the case, using instant messaging, and CoEPCE provides other avenues for patient care discussion through huddles involving the provider, RN care manager, clinical pharmacist, and other clinical professions.

Resources

The primary resource required to support PM is an information technology (IT) system that provides relevant health outcome and health care utilization data on patients assigned to trainees. PM sessions include teaching trainees how to access patient data. Since discussion about the care of panel patients during the learning sessions often results in real-time adjustments in the care plan, modest administrative support required post-PM sessions, such as clerical scheduling of the requested clinic or telephone follow-up with the physician, nurse, or pharmacist.

Monitoring and Assessment

Panel performance is evaluated at each educational session. To assess the CoEPCE PM curriculum, participants provide feedback in 8 questions over 3 domains: trainee perception of curriculum content, confidence in performing PM involving completion of a PM workshop, and likelihood of using PM techniques in the future. CoEPCE faculty use the feedback to improve their instruction of panel management skill and develop new sessions that target additional population groups. Evaluation of the curriculum also includes monitoring of panel patients’ chronic disease measures.

Several partnerships have contributed to the success and integrations of PM into facility activities. First, having the primary care clinic director as a member of the Co- EPCE faculty has encouraged faculty and staff to operationalize and implement PM broadly by distributing data monthly to all clinic staff. Second, high facility staff interest outside the CoEPCE and primary care clinic has facilitated establishing communications outside the CoEPCE regarding clinic data.

Challenges and Solutions

Trainees at earlier academic levels often desire more instruction in clinical knowledge, such as treatment options for DM or goals of therapy in hypertension. In contrast, advanced trainees are able to review patient data, brainstorm, and optimize solutions. Seattle CoEPCE balances these different learning needs via a flexible approach to the 3-hour sessions. For example, advanced trainees progress from structured short lectures to informal sessions, which train them to perform PM on their own. In addition, the flexible design integrates trainees with diverse schedules, particularly among DNP students and residents, pharmacy residents, and physician residents. Some of this work falls on the RN care management team and administrative support staff.

Competing Priorities

The demand for direct patient care points to the importance of indirect patient care activities like PM to demonstrate improved results. Managing chronic conditions and matching appropriate services and resources should improve clinical outcomes and efficiency longterm. In the interim, it is important to note that PM demonstrates the continuous aspect of clinical care, particularly for trainees who have strict guidelines defining clinical care for the experiences to count toward eligibility for licensure. Additionally, PM results in trainees who are making decisions with VA patients and are more efficiently providing and supporting patient care. Therefore, it is critical to secure important resources, such as provider time for conducting PM.

Data Access

No single data system in VA covers the broad range of topics covered in the PM sessions, and not all trainees have their own assigned panels. For example, health professions students are not assigned a panel of patients. While they do not have access to panel data such as those generated by Primary Care Almanac in VSSC (a data source in the VA Support Service Center database),the Seattle CoEPCE data manager pulls a set of patient data from the students’ paired faculty preceptors’ panels for review. Thus they learn PM principles and strategies for improving patient care via PM as part of the unique VA longitudinal clinic experience and the opportunity to learn from a multidisciplinary team that is not available at other clinical sites. Postgraduate NP residents in CoEPCE training have their own panels of patients and thus the ability to directly access their panel performance data.

Success Factors

A key success factor includes CoEPCE faculty’s ability to develop and operationalize a panel management model that simultaneously aligns with the educational goals of an interprofessional education training program and supports VA adoption of the medical home or patient aligned care teams (PACT). The CoEPCE contributes staff expertise in accessing and reporting patient data, accessing appropriate teaching space, managing panels of patients with chronic diseases, and facilitating a team-based approach to care. Additionally, the CoEPCE brand is helpful for getting buy-in from the clinical and academic stakeholders necessary for moving PM forward.

Colocating CoEPCE trainees and faculty in the primary care clinic promotes team identity around the RN care managers and facilitated communications with non-CoEPCE clinical teams that have trainees from other professions. RN care managers serve as the locus of highquality PM since they share patient panels with the trainees and already track admissions, ED visits, and numerous chronic health care metrics. RN care managers offer a level of insight into chronic disease that other providers may not possess, such as the specific details on medication adherence and the impact of adverse effects (AEs) for that particular patient. RN care managers are able to teach about their team role and responsibilities, strengthening the model.

PM is an opportunity to expand CoEPCE interprofessional education capacity by creating colocation of different trainee and faculty professions during the PM sessions; the sharing of data with trainees; and sharing and reflecting on data, strengthening communications between professions and within the PACT. The Seattle CoEPCE now has systems in place that allow the RN care manager to send notes to a physician and DNP resident, and the resident is expected to respond. In addition, the PM approach provides experience with analyzing data to improve care in an interprofessional team setting, which is a requirement of the Accreditation Council for Graduate Medical Education.

Interprofessional Collaboration

PM sessions are intentionally designed to improve communication among team members and foster a team approach to care. PM sessions provide an opportunity for trainees and clinician faculty to be together and learn about each profession’s perspectives. For example, early in the process physician and DNP trainees learn about the importance of clinical pharmacists to the team who prescribe and make medication adjustments within their scope of practice as well as the importance of making appropriate pharmacy referrals. Additionally, the RN care manager and clinical pharmacy specialists who serve as faculty in the CoEPCE provide pertinent information on individual patients, increasing integration with the PACT. Finally, there is anecdotal evidence that faculty also are learning more about interprofessional education and expanding their own skills.

Clinical Performance

CoEPCE trainees, non-CoEPCE physician residents, and CoEPCE faculty participants regularly receive patient data with which they can proactively develop or amend a treatment plan between visits. PM has resulted in improved data sharing with providers. Instead of once a year, providers and clinic staff now receive patient data monthly on chronic conditions from the clinic director. Trainees on ambulatory rotations are expected to review their panel data at least a half day per week. CoEPCE staff evaluate trainee likelihood to use PM and ability to identify patients who benefit from team-based care.

At the population level of chronic disease management, preliminary evidence demonstrates that primary care clinic patient panels are increasingly within target for DM and blood pressure measures, as assessed by periodic clinical reports to providers. Some of the PM topics have resulted in systems-level improvements, such as reducing unnecessary ED use for nonacute conditions and better opioid prescription monitoring. Moreover, PM supports everyone working at the top of his/her professional capability. For example, the RN care manager has the impetus to initiate DM education with a particular patient.

Since CoEPCE began teaching PM, the Seattle primary care clinic has committed to the regular access and review of data. This has encouraged the alignment of standards of care for chronic disease management so that all care providers are working toward the same benchmark goals.

Patient Outcomes

At the individual level, PM provide a mechanism to systemically review trainee panel patients with out-of-target clinical measures, and develop new care approaches involving interprofessional strategies and problem solving. PM also helps identify patients who have missed follow-up, reducing the risk that patients with chronic care needs will be lost to clinical engagement if they are not reminded or do not pursue appointments. The PM-trained PACT reaches out to patients who might not otherwise get care before the next clinic visit and provides new care plans. Second, patients have the benefit of a team that manages their health needs. For example, including the clinical pharmacists in the PM sessions ensures timely identification of medication interactions and the potential AEs. Additionally, PM contributes to the care coordination model by involving individuals on the primary care team who know the patient. These members review the patient’s data between visits and initiate team-based changes to the care plan to improve care. More team members connect with a patient, resulting in more intense care and quicker follow-up to determine the effectiveness of a treatment plan.

PM topics have spun off QI projects resulting in new clinic processes and programs, including processes for managing wounds in primary care and to assure timely post-ED visit follow-ups. Areas for expansion include a follow-up QI project to reduce nonacute ED visits by patients on the homeless PACT panel and interventions for better management of care for women veterans with mental health needs. PM also has extended to non-Co- EPCE teams and to other clinic activities, such as strengthening huddles of team members specifically related to panel data and addressing selected patient cases between visits. Pharmacy residents and faculty are more involved in reviewing the panel before patients are seen to review medication lists and identify duplications.

The Future

Under stage 2 of the program, the Seattle CoEPCE intends to lead in the creation of a PM toolkit as well as a data access guide that will allow VA facilities with limited data management expertise to access chronic disease metrics. Second, the CoEPCE will continue its dissemination efforts locally to other residents in the internal medicine residency program in all of its continuity clinics. Additionally, there is high interest by DNP training programs to expand and export longitudinal training experience PM curriculum to non-VA based students.

This article is part of a series that illustrates strategies intended to redesign primary care education at the Veterans Health Administration (VHA), using interprofessional workplace learning. All have been implemented in the VA Centers of Excellence in Primary Care Education (CoEPCE). These models embody visionary transformation of clinical and educational environments that have potential for replication and dissemination throughout VA and other primary care clinical educational environments. For an introduction to the series see Klink K. Transforming primary care clinical learning environments to optimize education, outcomes, and satisfaction. Fed Pract. 2018;35(9):8-10.

Background

In 2011, 5 US Department of Veterans Affairs (VA) medical centers were selected by the VA Office of Academic Affiliations (OAA) to establish Centers of Excellence in Primary Care Education (CoEPCE). Part of the New Models of Care initiative, the 5 CoEPCEs use VA primary care settings to develop and test innovative approaches to prepare physician residents, medical students, advanced practice registered nurses, undergraduate nursing students, and other health professions’ trainees, such as social workers, pharmacists, psychologists, and physician assistants, for improved primary care practice. The CoEPCEs are interprofessional Academic PACTs (iAPACTs) with ≥ 2 professions of trainees engaged in learning on the PACT team.

The VA Puget Sound Seattle CoEPCE curriculum is embedded in a well-established academic VA primary care training site.¹ Trainees include doctor of nursing practice (DNP) students in adult, family, and psychiatric mental health nurse practitioner (NP) programs; NP residents; internal medicine physician residents; postgraduate pharmacy residents; and other health professions’ trainees. A Seattle CoEPCE priority is to provide DNP students, DNP residents, and physician residents with a longitudinal experience in team-based care as well as interprofessional education and collaborative practice (IPECP). Learners spend the majority of CoEPCE time in supervised, direct patient care, including primary care, women’s health, deployment health, homeless care, and home care. Formal IPECP activities comprise about 20% of time, supported by 3 educational strategies: (1) Panel management (PM)/quality improvement (QI); (2) Team building/ communications; and (3) Clinical content seminars to expand trainee clinical knowledge and skills and curriculum developed with the CoEPCE enterprise core domains in mind (Table).

Panel Management

Clinicians are increasingly being required to proactively optimize the health of an assigned population of patients in addition to assessing and managing the health of individual patients presenting for care. To address the objectives of increased accountability for population health outcomes and improved face-to-face care, Seattle CoEPCE developed curriculum for trainees to learn PM, a set of tools and processes that can be applied in the primary care setting.

PM clinical providers use data to proactively provide care to their patients between traditional clinic visits. The process is proactive in that gaps are identified whether or not an in-person visit occurs and involves an outreach mechanism to increase continuity of care, such as follow-up communications with the patients.² PM also has been associated with improvements in chronic disease care.^3-5

The Seattle CoEPCE developed an interprofessional team approach to PM that teaches trainees about the tools and resources used to close the gaps in care, including the use of clinical team members as health care systems subject matter experts. CoEPCE trainees are taught to analyze the care they provide to their panel of veterans (eg, identifying patients who have not refilled chronic medications or those who use the emergency department [ED] for nonacute conditions) and take action to improve care. PM yields rich discussions on systems resources and processes and is easily applied to a range of health conditions as well as delivery system issues. PM gives learners the tools they can use to close these gaps, such as the expertise of their peers, clinical team, and specialists.⁶

Planning and Implementation

In addition to completing a literature review to determine the state of PM practice and models, CoEPCE faculty polled recent graduates inquiring about strategies they did not learn prior to graduation. Based on their responses, CoEPCE faculty identified 2 skill deficits: management of chronic diseases and proficiency with data and statistics about performance improvement in panel patient care over time. Addressing these unmet needs became the impetus for developing curriculum for conducting PM. Planning and launching the CoEPCE approach to PM took about 3 months and involved CoEPCE faculty, a data manager, and administrative support. The learning objectives of Seattle’s PM initiative are to:

• Promote preventive health and chronic disease care by use performance data;
• Develop individual- and populationfocused action plans;
• Work collaboratively, strategically, and effectively with an interprofessional care team; and
• Learn how to effectively use system resources.

Curriculum

The PM curriculum is a longitudinal, experiential approach to learning how to manage chronic diseases between visits by using patient data. It is designed for trainees in a continuity clinic to review the care of their patients on a regular basis. Seattle CoEPCE medicine residents are assigned patient panels, which increase from 70 patients in the first year to about 140 patients by the end of the third year. DNP postgraduate trainees are assigned an initial panel of 50 patients that increases incrementally over the year-long residency.

CoEPCE faculty determined the focus of PM sessions to be diabetes mellitus (DM), hypertension, obesity, chronic opioid therapy, and low-acuity ED use. Because PM sessions are designed to allow participants to identify systems issues that may affect multiple patients, some of these topics have expanded into QI projects. PM sessions run 2 to 3 hours per session and are held 4 to 6 times a year. Each session is repeated twice to accommodate diverse trainee schedules. PM participants must have their patient visit time blocked for each session (Appendix).

Faculty Roles and Development

PM faculty involved in any individual session may include a combination of a CoEPCE clinical pharmacy specialist, a registered nurse (RN) care manager, a social worker, a NP, a physician, a clinical psychologist, and a medicine outpatient chief resident (PGY4, termed clinician-teacher fellow at Seattle VA medical center). The chief resident is a medicine residency graduate and takes on teaching responsibilities depending on the topic of the session. The CoEPCE clinical pharmacist role varies depending on the session topic: They may facilitate the session or provide recommendations for medication management for individual cases. The RN care manager often knows the patients and brings a unique perspective that complements that of the primary care providers and ideally participates in every session. The patients of multiple RN care managers may be presented at each session, and it was not feasible to include all RN care managers in every session. After case discussions, trainees often communicated with the RN care managers about the case, using instant messaging, and CoEPCE provides other avenues for patient care discussion through huddles involving the provider, RN care manager, clinical pharmacist, and other clinical professions.

Resources

The primary resource required to support PM is an information technology (IT) system that provides relevant health outcome and health care utilization data on patients assigned to trainees. PM sessions include teaching trainees how to access patient data. Since discussion about the care of panel patients during the learning sessions often results in real-time adjustments in the care plan, modest administrative support required post-PM sessions, such as clerical scheduling of the requested clinic or telephone follow-up with the physician, nurse, or pharmacist.

Monitoring and Assessment

Panel performance is evaluated at each educational session. To assess the CoEPCE PM curriculum, participants provide feedback in 8 questions over 3 domains: trainee perception of curriculum content, confidence in performing PM involving completion of a PM workshop, and likelihood of using PM techniques in the future. CoEPCE faculty use the feedback to improve their instruction of panel management skill and develop new sessions that target additional population groups. Evaluation of the curriculum also includes monitoring of panel patients’ chronic disease measures.

Several partnerships have contributed to the success and integrations of PM into facility activities. First, having the primary care clinic director as a member of the Co- EPCE faculty has encouraged faculty and staff to operationalize and implement PM broadly by distributing data monthly to all clinic staff. Second, high facility staff interest outside the CoEPCE and primary care clinic has facilitated establishing communications outside the CoEPCE regarding clinic data.

Challenges and Solutions

Trainees at earlier academic levels often desire more instruction in clinical knowledge, such as treatment options for DM or goals of therapy in hypertension. In contrast, advanced trainees are able to review patient data, brainstorm, and optimize solutions. Seattle CoEPCE balances these different learning needs via a flexible approach to the 3-hour sessions. For example, advanced trainees progress from structured short lectures to informal sessions, which train them to perform PM on their own. In addition, the flexible design integrates trainees with diverse schedules, particularly among DNP students and residents, pharmacy residents, and physician residents. Some of this work falls on the RN care management team and administrative support staff.

Competing Priorities

The demand for direct patient care points to the importance of indirect patient care activities like PM to demonstrate improved results. Managing chronic conditions and matching appropriate services and resources should improve clinical outcomes and efficiency longterm. In the interim, it is important to note that PM demonstrates the continuous aspect of clinical care, particularly for trainees who have strict guidelines defining clinical care for the experiences to count toward eligibility for licensure. Additionally, PM results in trainees who are making decisions with VA patients and are more efficiently providing and supporting patient care. Therefore, it is critical to secure important resources, such as provider time for conducting PM.

Data Access

No single data system in VA covers the broad range of topics covered in the PM sessions, and not all trainees have their own assigned panels. For example, health professions students are not assigned a panel of patients. While they do not have access to panel data such as those generated by Primary Care Almanac in VSSC (a data source in the VA Support Service Center database),the Seattle CoEPCE data manager pulls a set of patient data from the students’ paired faculty preceptors’ panels for review. Thus they learn PM principles and strategies for improving patient care via PM as part of the unique VA longitudinal clinic experience and the opportunity to learn from a multidisciplinary team that is not available at other clinical sites. Postgraduate NP residents in CoEPCE training have their own panels of patients and thus the ability to directly access their panel performance data.

Success Factors

A key success factor includes CoEPCE faculty’s ability to develop and operationalize a panel management model that simultaneously aligns with the educational goals of an interprofessional education training program and supports VA adoption of the medical home or patient aligned care teams (PACT). The CoEPCE contributes staff expertise in accessing and reporting patient data, accessing appropriate teaching space, managing panels of patients with chronic diseases, and facilitating a team-based approach to care. Additionally, the CoEPCE brand is helpful for getting buy-in from the clinical and academic stakeholders necessary for moving PM forward.

Colocating CoEPCE trainees and faculty in the primary care clinic promotes team identity around the RN care managers and facilitated communications with non-CoEPCE clinical teams that have trainees from other professions. RN care managers serve as the locus of highquality PM since they share patient panels with the trainees and already track admissions, ED visits, and numerous chronic health care metrics. RN care managers offer a level of insight into chronic disease that other providers may not possess, such as the specific details on medication adherence and the impact of adverse effects (AEs) for that particular patient. RN care managers are able to teach about their team role and responsibilities, strengthening the model.

PM is an opportunity to expand CoEPCE interprofessional education capacity by creating colocation of different trainee and faculty professions during the PM sessions; the sharing of data with trainees; and sharing and reflecting on data, strengthening communications between professions and within the PACT. The Seattle CoEPCE now has systems in place that allow the RN care manager to send notes to a physician and DNP resident, and the resident is expected to respond. In addition, the PM approach provides experience with analyzing data to improve care in an interprofessional team setting, which is a requirement of the Accreditation Council for Graduate Medical Education.

Interprofessional Collaboration

PM sessions are intentionally designed to improve communication among team members and foster a team approach to care. PM sessions provide an opportunity for trainees and clinician faculty to be together and learn about each profession’s perspectives. For example, early in the process physician and DNP trainees learn about the importance of clinical pharmacists to the team who prescribe and make medication adjustments within their scope of practice as well as the importance of making appropriate pharmacy referrals. Additionally, the RN care manager and clinical pharmacy specialists who serve as faculty in the CoEPCE provide pertinent information on individual patients, increasing integration with the PACT. Finally, there is anecdotal evidence that faculty also are learning more about interprofessional education and expanding their own skills.

Clinical Performance

CoEPCE trainees, non-CoEPCE physician residents, and CoEPCE faculty participants regularly receive patient data with which they can proactively develop or amend a treatment plan between visits. PM has resulted in improved data sharing with providers. Instead of once a year, providers and clinic staff now receive patient data monthly on chronic conditions from the clinic director. Trainees on ambulatory rotations are expected to review their panel data at least a half day per week. CoEPCE staff evaluate trainee likelihood to use PM and ability to identify patients who benefit from team-based care.

At the population level of chronic disease management, preliminary evidence demonstrates that primary care clinic patient panels are increasingly within target for DM and blood pressure measures, as assessed by periodic clinical reports to providers. Some of the PM topics have resulted in systems-level improvements, such as reducing unnecessary ED use for nonacute conditions and better opioid prescription monitoring. Moreover, PM supports everyone working at the top of his/her professional capability. For example, the RN care manager has the impetus to initiate DM education with a particular patient.

Since CoEPCE began teaching PM, the Seattle primary care clinic has committed to the regular access and review of data. This has encouraged the alignment of standards of care for chronic disease management so that all care providers are working toward the same benchmark goals.

Patient Outcomes

At the individual level, PM provide a mechanism to systemically review trainee panel patients with out-of-target clinical measures, and develop new care approaches involving interprofessional strategies and problem solving. PM also helps identify patients who have missed follow-up, reducing the risk that patients with chronic care needs will be lost to clinical engagement if they are not reminded or do not pursue appointments. The PM-trained PACT reaches out to patients who might not otherwise get care before the next clinic visit and provides new care plans. Second, patients have the benefit of a team that manages their health needs. For example, including the clinical pharmacists in the PM sessions ensures timely identification of medication interactions and the potential AEs. Additionally, PM contributes to the care coordination model by involving individuals on the primary care team who know the patient. These members review the patient’s data between visits and initiate team-based changes to the care plan to improve care. More team members connect with a patient, resulting in more intense care and quicker follow-up to determine the effectiveness of a treatment plan.

PM topics have spun off QI projects resulting in new clinic processes and programs, including processes for managing wounds in primary care and to assure timely post-ED visit follow-ups. Areas for expansion include a follow-up QI project to reduce nonacute ED visits by patients on the homeless PACT panel and interventions for better management of care for women veterans with mental health needs. PM also has extended to non-Co- EPCE teams and to other clinic activities, such as strengthening huddles of team members specifically related to panel data and addressing selected patient cases between visits. Pharmacy residents and faculty are more involved in reviewing the panel before patients are seen to review medication lists and identify duplications.

The Future

Under stage 2 of the program, the Seattle CoEPCE intends to lead in the creation of a PM toolkit as well as a data access guide that will allow VA facilities with limited data management expertise to access chronic disease metrics. Second, the CoEPCE will continue its dissemination efforts locally to other residents in the internal medicine residency program in all of its continuity clinics. Additionally, there is high interest by DNP training programs to expand and export longitudinal training experience PM curriculum to non-VA based students.

From Tufts Medical Center, Boston, MA.

Abstract

• Objective: Audits at our academic medical center revealed near 100% compliance with protocols for perioperative venous thromboembolism (VTE) prophylaxis, but recent National Surgical Quality Improvement Program data demonstrated a higher than expected incidence of VTE (observed/expected = 1.32). The objective of this study was to identify potential causes of this discrepancy.
• Design: Retrospective case-control study.
• Setting: Urban academic medical center with high case-mix indices (Medicare approximately 2.4, non-Medicare approximately 2.0).
• Participants: 102 surgical inpatients with VTE (September 2012 to October 2015) matched with controls for age, gender, and type of procedure.
• Measurements: Prevalence of common VTE risk factors, length of stay, number of procedures, index operation times, and postoperative bed rest > 12 hours were assessed. Utilization of and compliance with our VTE risk assessment tool was also investigated.
• Results: Cases underwent more procedures and had longer lengths of stay and index procedures than controls. In addition, cases were more likely to have had > 12 hours of postoperative bed rest and central venous access than controls. Cases had more infections and were more likely to have severe lung disease, thrombophilia, and a history of prior VTE than controls. No differences in body mass index, tobacco use, current or previous malignancy, or VTE risk assessment form use were observed. Overall, care complexity and risk factors were equally important in determining VTE incidence. Our analyses also revealed lack of strict adherence to our VTE risk stratification protocol and frequent use of suboptimal prophylactic regimens.
• Conclusion: Well-accepted risk factors and overall care complexity determine VTE risk. Preventing VTE in high-risk patients requires assiduous attention to detail in VTE risk assessment and in delivery of optimal prophylaxis. Patients at especially high risk may require customized prophylactic regimens.

Keywords: hospital-acquired venous thromboembolic disease; VTE prophylaxis, surgical patients.

Deep vein thrombosis (DVT) and pulmonary embolism (PE) are well-recognized causes of morbidity and mortality in surgical patients. Between 350,000 and 600,000 cases of venous thromboembolism (VTE) occur each year in the United States, and it is responsible for approximately 10% of preventable in-hospital fatalities.^1-3 Given VTE’s impact on patients and the healthcare system and the fact that it is preventable, intense effort has been focused on developing more effective prophylactic measures to decrease its incidence.^2-4 In 2008, the surgeon general issued a “call to action” for increased efforts to prevent VTE.⁵

The American College of Chest Physicians (ACCP) guidelines subcategorize patients based on type of surgery. In addition, the ACCP guidelines support the use of a Caprini-based scoring system to aid in risk stratification and improve clinical decision-making (Table 1).^4,6-9 In general, scores ≥ 5 qualify individuals as high risk. Based on their risk category, patients receive mechanical prophylaxis, chemical prophylaxis, or a combination of the 2. Lower-risk patients who are ambulatory typically receive only mechanical prophylaxis while in bed, whereas higher-risk patients receive a combination of mechanical prophylaxis and chemoprophylaxis measures.⁷ In general, low-molecular-weight heparin (40 mg daily) and low-dose unfractionated heparin (5000 units 3 times daily) have been the standard evidence-based options for chemoprophylaxis in surgical patients. Absolute contraindications for prophylaxis include active bleeding and known increased risk of bleeding based on patient- or procedure-specific factors.

Our hospital, a 350-bed academic medical center in downtown Boston, MA, serving a diverse population with a very high case-mix index (2.4 Medicare and 2.0 non-Medicare), has strict protocols for VTE prophylaxis consistent with the ACCP guidelines and based on the Surgical Care Improvement Project (SCIP) measures published in 2006.¹⁰ The SCIP mandates allow for considerable surgeon discretion in the use of chemoprophylaxis for neurosurgical cases and general and orthopedic surgery cases deemed to be at high risk for bleeding. In addition, SCIP requires only that prophylaxis be initiated within 24 hours of surgical end time. Although recent audits revealed nearly 100% compliance with SCIP-mandated protocols, National Surgical Quality Improvement Program (NSQIP) data showed that the incidence of VTE events at our institution was higher than expected (observed/expected [O/E] = 1.32).

In order to determine the reasons for this mismatch between process and outcome performance, we investigated whether there were characteristics of our patient population that contributed to the higher than expected rates of VTE, and we scrutinized our VTE prophylaxis protocol to determine if there were aspects of our process that were also contributory.

Methods

Study Sample

This is a retrospective case-control study of surgical inpatients at our hospital during the period September 2012 to October 2015. Cases were identified as patients diagnosed with a VTE (DVT or PE). Controls were identified from a pool of surgical patients whose courses were not complicated by VTE during the same time frame as the cases and who were matched as closely as possible by procedure code, age, and gender.

Patient and hospital course variables that were analyzed included demographics, comorbidities, length of stay, number of procedures, index operation times, duration of postoperative bed rest, use of mechanical prophylaxis, and type of chemoprophylaxis and time frame within which it was initiated. Data were collected via chart review using International Classification of Diseases-9 and -10 codes to identify surgical cases within the allotted time period who were diagnosed with VTE. Demographic variables included age, sex, and ethnicity. Comorbidities included hypertension, diabetes, coronary artery disease, serious lung disease, previous or current malignancy, documented hypercoagulable state, and previous history of VTE. Body mass index (BMI) was also recorded. The aforementioned disease-specific variables were not matched between the case and control groups, as this data was obtained retrospectively during data collection.

Analysis

Associations between case and matched control were analyzed using the paired t-test for continuous variables and McNemar’s test for categorical variables. P values < 0.05 were considered statistically significant. SAS Enterprise Guide 7.15 (Cary, NC) was used for all statistical analyses.

The requirement for informed consent was waived by our Institutional Review Board, as the study was initially deemed to be a quality improvement project, and all data used for this report were de-identified.

Results

Our retrospective case-control analysis included a sample of 102 surgical patients whose courses were complicated by VTE between September 2012 and October 2015. The cases were distributed among 6 different surgical categories (Figure 1): trauma (20%), cancer (10%), cardiovascular (21%), noncancer neurosurgery (28%), elective orthopedics (11%), and miscellaneous general surgery (10%).

Comparisons between cases and controls in terms of patient demographics and risk factors are shown in Table 2. No statistically significant difference was observed in ethnicity or race between the 2 groups. Overall, cases had more hip/pelvis/leg fractures at presentation (P = 0.0008). The case group also had higher proportions of patients with postoperative bed rest greater than 12 hours (P = 0.009), central venous access (P < 0.0001), infection (P < 0.0001), and lower extremity edema documented during the hospitalization prior to development of DVT (P < 0.0001). Additionally, cases had significantly greater rates of previous VTE (P = 0.0004), inherited or acquired thrombophilia (P = 0.03), history of stroke (P = 0.0003), and severe lung disease, including pneumonia (P = 0.0008). No significant differences were noted between cases and matched controls in BMI (P = 0.43), current tobacco use (P = 0.71), current malignancy (P = 0.80), previous malignancy (P = 0.83), head trauma (P = 0.17), or acute cardiac disease (myocardial infarction or congestive heart failure; P = 0.12).

Variables felt to indicate overall complexity of hospital course for cases as compared to controls are outlined in Table 3. Cases were found to have significantly longer lengths of stay (median, 15.5 days versus 3 days, P < 0.0001). To account for the possibility that the development of VTE contributed to the increased length of stay in the cases, we also looked at the duration between admission date and the date of VTE diagnosis and determined that cases still had a longer length of stay when this was accounted for (median, 7 days versus 3 days, P < 0.0001). A much higher proportion of cases underwent more than 1 procedure compared to controls (P < 0.0001), and cases had significantly longer index operations as compared to controls (P = 0.002).

Seventeen cases received heparin on induction during their index procedure, compared to 23 controls (P = 0.24). Additionally, 63 cases began a prophylaxis regimen within 24 hours of surgery end time, compared to 68 controls (P = 0.24). The chemoprophylactic regimens utilized in cases and in controls are summarized in Figure 2. Of note, only 26 cases and 32 controls received standard prophylactic regimens with no missed doses (heparin 5000 units 3 times daily or enoxaparin 40 mg daily). Additionally, in over half of cases and a third of controls, nonstandard regimens were ordered. Examples of nonstandard regimens included nonstandard heparin or enoxaparin doses, low-dose warfarin, or aspirin alone. In most cases, nonstandard regimens were justified on the basis of high risk for bleeding.

Mechanical prophylaxis with pneumatic sequential compression devices (SCDs) was ordered in 93 (91%) cases and 87 (85%) controls; however, we were unable to accurately document uniform compliance in the use of these devices.

With regard to evaluation of our process measures, we found only 17% of cases and controls combined actually had a VTE risk assessment in their chart, and when it was present, it was often incomplete or was completed inaccurately.

Discussion

The goal of this study was to identify factors (patient characteristics and/or processes of care) that may be contributing to the higher than expected incidence of VTE events at our medical center, despite internal audits suggesting near perfect compliance with SCIP-mandated protocols. We found that in addition to usual risk factors for VTE, an overarching theme of our case cohort was their high complexity of illness. At baseline, these patients had significantly greater rates of stroke, thrombophilia, severe lung disease, infection, and history of VTE than controls. Moreover, the hospital courses of cases were significantly more complex than those of controls, as these patients had more procedures, longer lengths of stay and longer index operations, higher rates of postoperative bed rest exceeding 12 hours, and more prevalent central venous access than controls (Table 2). Several of these risk factors have been found to contribute to VTE development despite compliance with prophylaxis protocols.

Cassidy et al reviewed a cohort of nontrauma general surgery patients who developed VTE despite receiving appropriate prophylaxis and found that both multiple operations and emergency procedures contributed to the failure of VTE prophylaxis.¹¹ Similarly, Wang et al identified several independent risk factors for VTE despite thromboprophylaxis, including central venous access and infection, as well as intensive care unit admission, hospitalization for cranial surgery, and admission from a long-term care facility.¹² While our study did not capture some of these additional factors considered by Wang et al, the presence of risk factors not captured in traditional assessment tools suggests that additional consideration for complex patients is warranted.

In addition to these nonmodifiable patient characteristics, aspects of our VTE prophylaxis processes likely contributed to the higher than expected rate of VTE. While the electronic medical record at our institution does contain a VTE risk assessment tool based on the Caprini score, we found it often is not used at all or is used incorrectly/incompletely, which likely reflects the fact that physicians are neither prompted nor required to complete the assessment prior to prescribing VTE prophylaxis.

There is a significant body of evidence demonstrating that mandatory computerized VTE risk assessments can effectively reduce VTE rates and that improved outcomes occur shortly after implementation. Cassidy et al demonstrated the benefits of instituting a hospital-wide, mandatory, Caprini-based computerized VTE risk assessment that provides prophylaxis/early ambulation recommendations. Two years after implementing this system, they observed an 84% reduction in DVTs (P < 0.001) and a 55% reduction in PEs (P < 0.001).¹³ Nimeri et al had similarly impressive success, achieving a reduction in their NSQIP O/E for PE/DVT in general surgery from 6.00 in 2010 to 0.82 (for DVTs) and 0.78 (for PEs) 5 years after implementation of mandatory VTE risk assessment (though they noted that the most dramatic reduction occurred 1 year after implementation).¹⁴ Additionally, a recent systematic review and meta-analysis by Borab et al found computerized VTE risk assessments to be associated with a significant decrease in VTE events.¹⁵

The risk assessment tool used at our institution is qualitative in nature, and current literature suggests that employing a more quantitative tool may yield improved outcomes. Numerous studies have highlighted the importance of identifying patients at very high risk for VTE, as higher risk may necessitate more careful consideration of their prophylactic regimens. Obi et al found patients with Caprini scores higher than 8 to be at significantly greater risk of developing VTE compared to patients with scores of 7 or 8. Also, patients with scores of 7 or 8 were significantly more likely to have a VTE compared to those with scores of 5 or 6.¹⁶ In another study, Lobastov et al identified Caprini scores of 11 or higher as representing an extremely high-risk category for which standard prophylaxis regimens may not be effective.¹⁷ Thus, while having mandatory risk assessment has been shown to dramatically decrease VTE incidence, it is important to consider the magnitude of the numerical risk score. This is of particular importance at medical centers with high case-mix indices where patients at the highest risk might need to be managed with different prophylactic guidelines.

Another notable aspect of the process at our hospital was the great variation in the types of prophylactic regimens ordered, and the adherence to what was ordered. Only 25.5% of patients were maintained on a standard prophylactic regimen with no missed doses (heparin 5000 every 8 hours or enoxaparin 40 mg daily). Thus, the vast majority of the patients who went on to develop VTE either were prescribed a nontraditional prophylaxis regimen or missed doses of standard agents. The need for secondary surgical procedures or other invasive interventions may explain many, but not all, of the missed doses.

The timing of prophylaxis initiation for our patients was also found to deviate from accepted standards. Only 16.8% of cases received prophylaxis upon induction of anesthesia, and furthermore, 38% of cases did not receive any anticoagulation within 24 hours of their index operation. While this variability in prophylaxis implementation was acceptable within the SCIP guidelines based on “high risk for bleeding” or other considerations, it likely contributed to our suboptimal outcomes. The variations and interruptions in prophylactic regimens speak to barriers that have previously been reported as contributing factors to noncompliance with VTE prophylaxis.¹⁸

Given these known barriers and the observed underutilization and improper use of our risk assessment tool, we have recently changed our surgical admission order sets such that a mandatory quantitative risk assessment must be done for every surgical patient at the time of admission/operation before other orders can be completed. Following completion of the assessment, the physician will be presented with an appropriate standard regimen based on the individual patient’s risk assessment. Early results of our VTE quality improvement project have been satisfying: in the most recent NSQIP semi-annual report, our O/E for VTE was 0.74, placing us in the first decile. Some of these early reports may simply be the product of the Hawthorne effect; however, we are encouraged by the early improvements seen in other research. While we are hopeful that these changes will result in sustainable improvements in outcomes, patients at extremely high risk may require novel weight-based or otherwise customized aggressive prophylactic regimens. Such regimens have already been proposed for arthroplasty and other high-risk patients.

Future research may identify other risk factors not captured by traditional risk assessments. In addition, research should continue to explore the use and efficacy of standard prophylactic regimens in these populations to help determine if they are sufficient. Currently, weight-based low-molecular-weight heparin dosing and alternative regimens employing fondaparinux are under investigation for very-high-risk patients.¹⁹

There were several limitations to the present study. First, due to the retrospective design of our study, we could collect only data that had been uniformly recorded in the charts throughout the study period. Second, we were unable to accurately assess compliance with mechanical prophylaxis. While our chart review showed that the vast majority of cases and controls were ordered to have mechanical prophylaxis, it is impossible to document how often these devices were used appropriately in a retrospective analysis. Anecdotal observation suggests that once patients are out of post-anesthesia or critical care units, SCD use is not standardized. The inability to measure compliance precisely may be leading to an overestimation of our compliance with prophylaxis. Finally, because our study included only patients who underwent surgery at our hospital, our observations may not be generalizable outside our institution.

Conclusion

Our study findings reinforce the importance of attention to detail in VTE risk assessment and in ordering and administering VTE prophylactic regimens, especially in high-risk surgical patients. While we adhered to the SCIP-mandated prophylaxis requirements, the complexity of our patients and our lack of a truly standardized approach to risk assessment and prophylactic regimens resulted in suboptimal outcomes. Stricter and more quantitative mandatory VTE risk assessment, along with highly standardized VTE prophylaxis regimens, are required to achieve optimal outcomes.

Corresponding author: Jason C. DeGiovanni, MS, BA, [email protected].

Financial disclosures: None.

From Tufts Medical Center, Boston, MA.

Abstract

• Objective: Audits at our academic medical center revealed near 100% compliance with protocols for perioperative venous thromboembolism (VTE) prophylaxis, but recent National Surgical Quality Improvement Program data demonstrated a higher than expected incidence of VTE (observed/expected = 1.32). The objective of this study was to identify potential causes of this discrepancy.
• Design: Retrospective case-control study.
• Setting: Urban academic medical center with high case-mix indices (Medicare approximately 2.4, non-Medicare approximately 2.0).
• Participants: 102 surgical inpatients with VTE (September 2012 to October 2015) matched with controls for age, gender, and type of procedure.
• Measurements: Prevalence of common VTE risk factors, length of stay, number of procedures, index operation times, and postoperative bed rest > 12 hours were assessed. Utilization of and compliance with our VTE risk assessment tool was also investigated.
• Results: Cases underwent more procedures and had longer lengths of stay and index procedures than controls. In addition, cases were more likely to have had > 12 hours of postoperative bed rest and central venous access than controls. Cases had more infections and were more likely to have severe lung disease, thrombophilia, and a history of prior VTE than controls. No differences in body mass index, tobacco use, current or previous malignancy, or VTE risk assessment form use were observed. Overall, care complexity and risk factors were equally important in determining VTE incidence. Our analyses also revealed lack of strict adherence to our VTE risk stratification protocol and frequent use of suboptimal prophylactic regimens.
• Conclusion: Well-accepted risk factors and overall care complexity determine VTE risk. Preventing VTE in high-risk patients requires assiduous attention to detail in VTE risk assessment and in delivery of optimal prophylaxis. Patients at especially high risk may require customized prophylactic regimens.

Keywords: hospital-acquired venous thromboembolic disease; VTE prophylaxis, surgical patients.

Deep vein thrombosis (DVT) and pulmonary embolism (PE) are well-recognized causes of morbidity and mortality in surgical patients. Between 350,000 and 600,000 cases of venous thromboembolism (VTE) occur each year in the United States, and it is responsible for approximately 10% of preventable in-hospital fatalities.^1-3 Given VTE’s impact on patients and the healthcare system and the fact that it is preventable, intense effort has been focused on developing more effective prophylactic measures to decrease its incidence.^2-4 In 2008, the surgeon general issued a “call to action” for increased efforts to prevent VTE.⁵

The American College of Chest Physicians (ACCP) guidelines subcategorize patients based on type of surgery. In addition, the ACCP guidelines support the use of a Caprini-based scoring system to aid in risk stratification and improve clinical decision-making (Table 1).^4,6-9 In general, scores ≥ 5 qualify individuals as high risk. Based on their risk category, patients receive mechanical prophylaxis, chemical prophylaxis, or a combination of the 2. Lower-risk patients who are ambulatory typically receive only mechanical prophylaxis while in bed, whereas higher-risk patients receive a combination of mechanical prophylaxis and chemoprophylaxis measures.⁷ In general, low-molecular-weight heparin (40 mg daily) and low-dose unfractionated heparin (5000 units 3 times daily) have been the standard evidence-based options for chemoprophylaxis in surgical patients. Absolute contraindications for prophylaxis include active bleeding and known increased risk of bleeding based on patient- or procedure-specific factors.

Our hospital, a 350-bed academic medical center in downtown Boston, MA, serving a diverse population with a very high case-mix index (2.4 Medicare and 2.0 non-Medicare), has strict protocols for VTE prophylaxis consistent with the ACCP guidelines and based on the Surgical Care Improvement Project (SCIP) measures published in 2006.¹⁰ The SCIP mandates allow for considerable surgeon discretion in the use of chemoprophylaxis for neurosurgical cases and general and orthopedic surgery cases deemed to be at high risk for bleeding. In addition, SCIP requires only that prophylaxis be initiated within 24 hours of surgical end time. Although recent audits revealed nearly 100% compliance with SCIP-mandated protocols, National Surgical Quality Improvement Program (NSQIP) data showed that the incidence of VTE events at our institution was higher than expected (observed/expected [O/E] = 1.32).

In order to determine the reasons for this mismatch between process and outcome performance, we investigated whether there were characteristics of our patient population that contributed to the higher than expected rates of VTE, and we scrutinized our VTE prophylaxis protocol to determine if there were aspects of our process that were also contributory.

Methods

Study Sample

This is a retrospective case-control study of surgical inpatients at our hospital during the period September 2012 to October 2015. Cases were identified as patients diagnosed with a VTE (DVT or PE). Controls were identified from a pool of surgical patients whose courses were not complicated by VTE during the same time frame as the cases and who were matched as closely as possible by procedure code, age, and gender.

Patient and hospital course variables that were analyzed included demographics, comorbidities, length of stay, number of procedures, index operation times, duration of postoperative bed rest, use of mechanical prophylaxis, and type of chemoprophylaxis and time frame within which it was initiated. Data were collected via chart review using International Classification of Diseases-9 and -10 codes to identify surgical cases within the allotted time period who were diagnosed with VTE. Demographic variables included age, sex, and ethnicity. Comorbidities included hypertension, diabetes, coronary artery disease, serious lung disease, previous or current malignancy, documented hypercoagulable state, and previous history of VTE. Body mass index (BMI) was also recorded. The aforementioned disease-specific variables were not matched between the case and control groups, as this data was obtained retrospectively during data collection.

Analysis

Associations between case and matched control were analyzed using the paired t-test for continuous variables and McNemar’s test for categorical variables. P values < 0.05 were considered statistically significant. SAS Enterprise Guide 7.15 (Cary, NC) was used for all statistical analyses.

The requirement for informed consent was waived by our Institutional Review Board, as the study was initially deemed to be a quality improvement project, and all data used for this report were de-identified.

Results

Our retrospective case-control analysis included a sample of 102 surgical patients whose courses were complicated by VTE between September 2012 and October 2015. The cases were distributed among 6 different surgical categories (Figure 1): trauma (20%), cancer (10%), cardiovascular (21%), noncancer neurosurgery (28%), elective orthopedics (11%), and miscellaneous general surgery (10%).

Comparisons between cases and controls in terms of patient demographics and risk factors are shown in Table 2. No statistically significant difference was observed in ethnicity or race between the 2 groups. Overall, cases had more hip/pelvis/leg fractures at presentation (P = 0.0008). The case group also had higher proportions of patients with postoperative bed rest greater than 12 hours (P = 0.009), central venous access (P < 0.0001), infection (P < 0.0001), and lower extremity edema documented during the hospitalization prior to development of DVT (P < 0.0001). Additionally, cases had significantly greater rates of previous VTE (P = 0.0004), inherited or acquired thrombophilia (P = 0.03), history of stroke (P = 0.0003), and severe lung disease, including pneumonia (P = 0.0008). No significant differences were noted between cases and matched controls in BMI (P = 0.43), current tobacco use (P = 0.71), current malignancy (P = 0.80), previous malignancy (P = 0.83), head trauma (P = 0.17), or acute cardiac disease (myocardial infarction or congestive heart failure; P = 0.12).

Variables felt to indicate overall complexity of hospital course for cases as compared to controls are outlined in Table 3. Cases were found to have significantly longer lengths of stay (median, 15.5 days versus 3 days, P < 0.0001). To account for the possibility that the development of VTE contributed to the increased length of stay in the cases, we also looked at the duration between admission date and the date of VTE diagnosis and determined that cases still had a longer length of stay when this was accounted for (median, 7 days versus 3 days, P < 0.0001). A much higher proportion of cases underwent more than 1 procedure compared to controls (P < 0.0001), and cases had significantly longer index operations as compared to controls (P = 0.002).

Seventeen cases received heparin on induction during their index procedure, compared to 23 controls (P = 0.24). Additionally, 63 cases began a prophylaxis regimen within 24 hours of surgery end time, compared to 68 controls (P = 0.24). The chemoprophylactic regimens utilized in cases and in controls are summarized in Figure 2. Of note, only 26 cases and 32 controls received standard prophylactic regimens with no missed doses (heparin 5000 units 3 times daily or enoxaparin 40 mg daily). Additionally, in over half of cases and a third of controls, nonstandard regimens were ordered. Examples of nonstandard regimens included nonstandard heparin or enoxaparin doses, low-dose warfarin, or aspirin alone. In most cases, nonstandard regimens were justified on the basis of high risk for bleeding.

Mechanical prophylaxis with pneumatic sequential compression devices (SCDs) was ordered in 93 (91%) cases and 87 (85%) controls; however, we were unable to accurately document uniform compliance in the use of these devices.

With regard to evaluation of our process measures, we found only 17% of cases and controls combined actually had a VTE risk assessment in their chart, and when it was present, it was often incomplete or was completed inaccurately.

Discussion

The goal of this study was to identify factors (patient characteristics and/or processes of care) that may be contributing to the higher than expected incidence of VTE events at our medical center, despite internal audits suggesting near perfect compliance with SCIP-mandated protocols. We found that in addition to usual risk factors for VTE, an overarching theme of our case cohort was their high complexity of illness. At baseline, these patients had significantly greater rates of stroke, thrombophilia, severe lung disease, infection, and history of VTE than controls. Moreover, the hospital courses of cases were significantly more complex than those of controls, as these patients had more procedures, longer lengths of stay and longer index operations, higher rates of postoperative bed rest exceeding 12 hours, and more prevalent central venous access than controls (Table 2). Several of these risk factors have been found to contribute to VTE development despite compliance with prophylaxis protocols.

Cassidy et al reviewed a cohort of nontrauma general surgery patients who developed VTE despite receiving appropriate prophylaxis and found that both multiple operations and emergency procedures contributed to the failure of VTE prophylaxis.¹¹ Similarly, Wang et al identified several independent risk factors for VTE despite thromboprophylaxis, including central venous access and infection, as well as intensive care unit admission, hospitalization for cranial surgery, and admission from a long-term care facility.¹² While our study did not capture some of these additional factors considered by Wang et al, the presence of risk factors not captured in traditional assessment tools suggests that additional consideration for complex patients is warranted.

In addition to these nonmodifiable patient characteristics, aspects of our VTE prophylaxis processes likely contributed to the higher than expected rate of VTE. While the electronic medical record at our institution does contain a VTE risk assessment tool based on the Caprini score, we found it often is not used at all or is used incorrectly/incompletely, which likely reflects the fact that physicians are neither prompted nor required to complete the assessment prior to prescribing VTE prophylaxis.

There is a significant body of evidence demonstrating that mandatory computerized VTE risk assessments can effectively reduce VTE rates and that improved outcomes occur shortly after implementation. Cassidy et al demonstrated the benefits of instituting a hospital-wide, mandatory, Caprini-based computerized VTE risk assessment that provides prophylaxis/early ambulation recommendations. Two years after implementing this system, they observed an 84% reduction in DVTs (P < 0.001) and a 55% reduction in PEs (P < 0.001).¹³ Nimeri et al had similarly impressive success, achieving a reduction in their NSQIP O/E for PE/DVT in general surgery from 6.00 in 2010 to 0.82 (for DVTs) and 0.78 (for PEs) 5 years after implementation of mandatory VTE risk assessment (though they noted that the most dramatic reduction occurred 1 year after implementation).¹⁴ Additionally, a recent systematic review and meta-analysis by Borab et al found computerized VTE risk assessments to be associated with a significant decrease in VTE events.¹⁵

The risk assessment tool used at our institution is qualitative in nature, and current literature suggests that employing a more quantitative tool may yield improved outcomes. Numerous studies have highlighted the importance of identifying patients at very high risk for VTE, as higher risk may necessitate more careful consideration of their prophylactic regimens. Obi et al found patients with Caprini scores higher than 8 to be at significantly greater risk of developing VTE compared to patients with scores of 7 or 8. Also, patients with scores of 7 or 8 were significantly more likely to have a VTE compared to those with scores of 5 or 6.¹⁶ In another study, Lobastov et al identified Caprini scores of 11 or higher as representing an extremely high-risk category for which standard prophylaxis regimens may not be effective.¹⁷ Thus, while having mandatory risk assessment has been shown to dramatically decrease VTE incidence, it is important to consider the magnitude of the numerical risk score. This is of particular importance at medical centers with high case-mix indices where patients at the highest risk might need to be managed with different prophylactic guidelines.

Another notable aspect of the process at our hospital was the great variation in the types of prophylactic regimens ordered, and the adherence to what was ordered. Only 25.5% of patients were maintained on a standard prophylactic regimen with no missed doses (heparin 5000 every 8 hours or enoxaparin 40 mg daily). Thus, the vast majority of the patients who went on to develop VTE either were prescribed a nontraditional prophylaxis regimen or missed doses of standard agents. The need for secondary surgical procedures or other invasive interventions may explain many, but not all, of the missed doses.

The timing of prophylaxis initiation for our patients was also found to deviate from accepted standards. Only 16.8% of cases received prophylaxis upon induction of anesthesia, and furthermore, 38% of cases did not receive any anticoagulation within 24 hours of their index operation. While this variability in prophylaxis implementation was acceptable within the SCIP guidelines based on “high risk for bleeding” or other considerations, it likely contributed to our suboptimal outcomes. The variations and interruptions in prophylactic regimens speak to barriers that have previously been reported as contributing factors to noncompliance with VTE prophylaxis.¹⁸

Given these known barriers and the observed underutilization and improper use of our risk assessment tool, we have recently changed our surgical admission order sets such that a mandatory quantitative risk assessment must be done for every surgical patient at the time of admission/operation before other orders can be completed. Following completion of the assessment, the physician will be presented with an appropriate standard regimen based on the individual patient’s risk assessment. Early results of our VTE quality improvement project have been satisfying: in the most recent NSQIP semi-annual report, our O/E for VTE was 0.74, placing us in the first decile. Some of these early reports may simply be the product of the Hawthorne effect; however, we are encouraged by the early improvements seen in other research. While we are hopeful that these changes will result in sustainable improvements in outcomes, patients at extremely high risk may require novel weight-based or otherwise customized aggressive prophylactic regimens. Such regimens have already been proposed for arthroplasty and other high-risk patients.

Future research may identify other risk factors not captured by traditional risk assessments. In addition, research should continue to explore the use and efficacy of standard prophylactic regimens in these populations to help determine if they are sufficient. Currently, weight-based low-molecular-weight heparin dosing and alternative regimens employing fondaparinux are under investigation for very-high-risk patients.¹⁹

There were several limitations to the present study. First, due to the retrospective design of our study, we could collect only data that had been uniformly recorded in the charts throughout the study period. Second, we were unable to accurately assess compliance with mechanical prophylaxis. While our chart review showed that the vast majority of cases and controls were ordered to have mechanical prophylaxis, it is impossible to document how often these devices were used appropriately in a retrospective analysis. Anecdotal observation suggests that once patients are out of post-anesthesia or critical care units, SCD use is not standardized. The inability to measure compliance precisely may be leading to an overestimation of our compliance with prophylaxis. Finally, because our study included only patients who underwent surgery at our hospital, our observations may not be generalizable outside our institution.

Conclusion

Our study findings reinforce the importance of attention to detail in VTE risk assessment and in ordering and administering VTE prophylactic regimens, especially in high-risk surgical patients. While we adhered to the SCIP-mandated prophylaxis requirements, the complexity of our patients and our lack of a truly standardized approach to risk assessment and prophylactic regimens resulted in suboptimal outcomes. Stricter and more quantitative mandatory VTE risk assessment, along with highly standardized VTE prophylaxis regimens, are required to achieve optimal outcomes.

Corresponding author: Jason C. DeGiovanni, MS, BA, [email protected].

Financial disclosures: None.

From Tufts Medical Center, Boston, MA.

Abstract

• Objective: Audits at our academic medical center revealed near 100% compliance with protocols for perioperative venous thromboembolism (VTE) prophylaxis, but recent National Surgical Quality Improvement Program data demonstrated a higher than expected incidence of VTE (observed/expected = 1.32). The objective of this study was to identify potential causes of this discrepancy.
• Design: Retrospective case-control study.
• Setting: Urban academic medical center with high case-mix indices (Medicare approximately 2.4, non-Medicare approximately 2.0).
• Participants: 102 surgical inpatients with VTE (September 2012 to October 2015) matched with controls for age, gender, and type of procedure.
• Measurements: Prevalence of common VTE risk factors, length of stay, number of procedures, index operation times, and postoperative bed rest > 12 hours were assessed. Utilization of and compliance with our VTE risk assessment tool was also investigated.
• Results: Cases underwent more procedures and had longer lengths of stay and index procedures than controls. In addition, cases were more likely to have had > 12 hours of postoperative bed rest and central venous access than controls. Cases had more infections and were more likely to have severe lung disease, thrombophilia, and a history of prior VTE than controls. No differences in body mass index, tobacco use, current or previous malignancy, or VTE risk assessment form use were observed. Overall, care complexity and risk factors were equally important in determining VTE incidence. Our analyses also revealed lack of strict adherence to our VTE risk stratification protocol and frequent use of suboptimal prophylactic regimens.
• Conclusion: Well-accepted risk factors and overall care complexity determine VTE risk. Preventing VTE in high-risk patients requires assiduous attention to detail in VTE risk assessment and in delivery of optimal prophylaxis. Patients at especially high risk may require customized prophylactic regimens.

Keywords: hospital-acquired venous thromboembolic disease; VTE prophylaxis, surgical patients.

Deep vein thrombosis (DVT) and pulmonary embolism (PE) are well-recognized causes of morbidity and mortality in surgical patients. Between 350,000 and 600,000 cases of venous thromboembolism (VTE) occur each year in the United States, and it is responsible for approximately 10% of preventable in-hospital fatalities.^1-3 Given VTE’s impact on patients and the healthcare system and the fact that it is preventable, intense effort has been focused on developing more effective prophylactic measures to decrease its incidence.^2-4 In 2008, the surgeon general issued a “call to action” for increased efforts to prevent VTE.⁵

The American College of Chest Physicians (ACCP) guidelines subcategorize patients based on type of surgery. In addition, the ACCP guidelines support the use of a Caprini-based scoring system to aid in risk stratification and improve clinical decision-making (Table 1).^4,6-9 In general, scores ≥ 5 qualify individuals as high risk. Based on their risk category, patients receive mechanical prophylaxis, chemical prophylaxis, or a combination of the 2. Lower-risk patients who are ambulatory typically receive only mechanical prophylaxis while in bed, whereas higher-risk patients receive a combination of mechanical prophylaxis and chemoprophylaxis measures.⁷ In general, low-molecular-weight heparin (40 mg daily) and low-dose unfractionated heparin (5000 units 3 times daily) have been the standard evidence-based options for chemoprophylaxis in surgical patients. Absolute contraindications for prophylaxis include active bleeding and known increased risk of bleeding based on patient- or procedure-specific factors.

Our hospital, a 350-bed academic medical center in downtown Boston, MA, serving a diverse population with a very high case-mix index (2.4 Medicare and 2.0 non-Medicare), has strict protocols for VTE prophylaxis consistent with the ACCP guidelines and based on the Surgical Care Improvement Project (SCIP) measures published in 2006.¹⁰ The SCIP mandates allow for considerable surgeon discretion in the use of chemoprophylaxis for neurosurgical cases and general and orthopedic surgery cases deemed to be at high risk for bleeding. In addition, SCIP requires only that prophylaxis be initiated within 24 hours of surgical end time. Although recent audits revealed nearly 100% compliance with SCIP-mandated protocols, National Surgical Quality Improvement Program (NSQIP) data showed that the incidence of VTE events at our institution was higher than expected (observed/expected [O/E] = 1.32).

In order to determine the reasons for this mismatch between process and outcome performance, we investigated whether there were characteristics of our patient population that contributed to the higher than expected rates of VTE, and we scrutinized our VTE prophylaxis protocol to determine if there were aspects of our process that were also contributory.

Methods

Study Sample

This is a retrospective case-control study of surgical inpatients at our hospital during the period September 2012 to October 2015. Cases were identified as patients diagnosed with a VTE (DVT or PE). Controls were identified from a pool of surgical patients whose courses were not complicated by VTE during the same time frame as the cases and who were matched as closely as possible by procedure code, age, and gender.

Patient and hospital course variables that were analyzed included demographics, comorbidities, length of stay, number of procedures, index operation times, duration of postoperative bed rest, use of mechanical prophylaxis, and type of chemoprophylaxis and time frame within which it was initiated. Data were collected via chart review using International Classification of Diseases-9 and -10 codes to identify surgical cases within the allotted time period who were diagnosed with VTE. Demographic variables included age, sex, and ethnicity. Comorbidities included hypertension, diabetes, coronary artery disease, serious lung disease, previous or current malignancy, documented hypercoagulable state, and previous history of VTE. Body mass index (BMI) was also recorded. The aforementioned disease-specific variables were not matched between the case and control groups, as this data was obtained retrospectively during data collection.

Analysis

Associations between case and matched control were analyzed using the paired t-test for continuous variables and McNemar’s test for categorical variables. P values < 0.05 were considered statistically significant. SAS Enterprise Guide 7.15 (Cary, NC) was used for all statistical analyses.

The requirement for informed consent was waived by our Institutional Review Board, as the study was initially deemed to be a quality improvement project, and all data used for this report were de-identified.

Results

Our retrospective case-control analysis included a sample of 102 surgical patients whose courses were complicated by VTE between September 2012 and October 2015. The cases were distributed among 6 different surgical categories (Figure 1): trauma (20%), cancer (10%), cardiovascular (21%), noncancer neurosurgery (28%), elective orthopedics (11%), and miscellaneous general surgery (10%).

Comparisons between cases and controls in terms of patient demographics and risk factors are shown in Table 2. No statistically significant difference was observed in ethnicity or race between the 2 groups. Overall, cases had more hip/pelvis/leg fractures at presentation (P = 0.0008). The case group also had higher proportions of patients with postoperative bed rest greater than 12 hours (P = 0.009), central venous access (P < 0.0001), infection (P < 0.0001), and lower extremity edema documented during the hospitalization prior to development of DVT (P < 0.0001). Additionally, cases had significantly greater rates of previous VTE (P = 0.0004), inherited or acquired thrombophilia (P = 0.03), history of stroke (P = 0.0003), and severe lung disease, including pneumonia (P = 0.0008). No significant differences were noted between cases and matched controls in BMI (P = 0.43), current tobacco use (P = 0.71), current malignancy (P = 0.80), previous malignancy (P = 0.83), head trauma (P = 0.17), or acute cardiac disease (myocardial infarction or congestive heart failure; P = 0.12).

Variables felt to indicate overall complexity of hospital course for cases as compared to controls are outlined in Table 3. Cases were found to have significantly longer lengths of stay (median, 15.5 days versus 3 days, P < 0.0001). To account for the possibility that the development of VTE contributed to the increased length of stay in the cases, we also looked at the duration between admission date and the date of VTE diagnosis and determined that cases still had a longer length of stay when this was accounted for (median, 7 days versus 3 days, P < 0.0001). A much higher proportion of cases underwent more than 1 procedure compared to controls (P < 0.0001), and cases had significantly longer index operations as compared to controls (P = 0.002).

Seventeen cases received heparin on induction during their index procedure, compared to 23 controls (P = 0.24). Additionally, 63 cases began a prophylaxis regimen within 24 hours of surgery end time, compared to 68 controls (P = 0.24). The chemoprophylactic regimens utilized in cases and in controls are summarized in Figure 2. Of note, only 26 cases and 32 controls received standard prophylactic regimens with no missed doses (heparin 5000 units 3 times daily or enoxaparin 40 mg daily). Additionally, in over half of cases and a third of controls, nonstandard regimens were ordered. Examples of nonstandard regimens included nonstandard heparin or enoxaparin doses, low-dose warfarin, or aspirin alone. In most cases, nonstandard regimens were justified on the basis of high risk for bleeding.

Mechanical prophylaxis with pneumatic sequential compression devices (SCDs) was ordered in 93 (91%) cases and 87 (85%) controls; however, we were unable to accurately document uniform compliance in the use of these devices.

With regard to evaluation of our process measures, we found only 17% of cases and controls combined actually had a VTE risk assessment in their chart, and when it was present, it was often incomplete or was completed inaccurately.

Discussion

The goal of this study was to identify factors (patient characteristics and/or processes of care) that may be contributing to the higher than expected incidence of VTE events at our medical center, despite internal audits suggesting near perfect compliance with SCIP-mandated protocols. We found that in addition to usual risk factors for VTE, an overarching theme of our case cohort was their high complexity of illness. At baseline, these patients had significantly greater rates of stroke, thrombophilia, severe lung disease, infection, and history of VTE than controls. Moreover, the hospital courses of cases were significantly more complex than those of controls, as these patients had more procedures, longer lengths of stay and longer index operations, higher rates of postoperative bed rest exceeding 12 hours, and more prevalent central venous access than controls (Table 2). Several of these risk factors have been found to contribute to VTE development despite compliance with prophylaxis protocols.

Cassidy et al reviewed a cohort of nontrauma general surgery patients who developed VTE despite receiving appropriate prophylaxis and found that both multiple operations and emergency procedures contributed to the failure of VTE prophylaxis.¹¹ Similarly, Wang et al identified several independent risk factors for VTE despite thromboprophylaxis, including central venous access and infection, as well as intensive care unit admission, hospitalization for cranial surgery, and admission from a long-term care facility.¹² While our study did not capture some of these additional factors considered by Wang et al, the presence of risk factors not captured in traditional assessment tools suggests that additional consideration for complex patients is warranted.

In addition to these nonmodifiable patient characteristics, aspects of our VTE prophylaxis processes likely contributed to the higher than expected rate of VTE. While the electronic medical record at our institution does contain a VTE risk assessment tool based on the Caprini score, we found it often is not used at all or is used incorrectly/incompletely, which likely reflects the fact that physicians are neither prompted nor required to complete the assessment prior to prescribing VTE prophylaxis.

There is a significant body of evidence demonstrating that mandatory computerized VTE risk assessments can effectively reduce VTE rates and that improved outcomes occur shortly after implementation. Cassidy et al demonstrated the benefits of instituting a hospital-wide, mandatory, Caprini-based computerized VTE risk assessment that provides prophylaxis/early ambulation recommendations. Two years after implementing this system, they observed an 84% reduction in DVTs (P < 0.001) and a 55% reduction in PEs (P < 0.001).¹³ Nimeri et al had similarly impressive success, achieving a reduction in their NSQIP O/E for PE/DVT in general surgery from 6.00 in 2010 to 0.82 (for DVTs) and 0.78 (for PEs) 5 years after implementation of mandatory VTE risk assessment (though they noted that the most dramatic reduction occurred 1 year after implementation).¹⁴ Additionally, a recent systematic review and meta-analysis by Borab et al found computerized VTE risk assessments to be associated with a significant decrease in VTE events.¹⁵

The risk assessment tool used at our institution is qualitative in nature, and current literature suggests that employing a more quantitative tool may yield improved outcomes. Numerous studies have highlighted the importance of identifying patients at very high risk for VTE, as higher risk may necessitate more careful consideration of their prophylactic regimens. Obi et al found patients with Caprini scores higher than 8 to be at significantly greater risk of developing VTE compared to patients with scores of 7 or 8. Also, patients with scores of 7 or 8 were significantly more likely to have a VTE compared to those with scores of 5 or 6.¹⁶ In another study, Lobastov et al identified Caprini scores of 11 or higher as representing an extremely high-risk category for which standard prophylaxis regimens may not be effective.¹⁷ Thus, while having mandatory risk assessment has been shown to dramatically decrease VTE incidence, it is important to consider the magnitude of the numerical risk score. This is of particular importance at medical centers with high case-mix indices where patients at the highest risk might need to be managed with different prophylactic guidelines.

Another notable aspect of the process at our hospital was the great variation in the types of prophylactic regimens ordered, and the adherence to what was ordered. Only 25.5% of patients were maintained on a standard prophylactic regimen with no missed doses (heparin 5000 every 8 hours or enoxaparin 40 mg daily). Thus, the vast majority of the patients who went on to develop VTE either were prescribed a nontraditional prophylaxis regimen or missed doses of standard agents. The need for secondary surgical procedures or other invasive interventions may explain many, but not all, of the missed doses.

The timing of prophylaxis initiation for our patients was also found to deviate from accepted standards. Only 16.8% of cases received prophylaxis upon induction of anesthesia, and furthermore, 38% of cases did not receive any anticoagulation within 24 hours of their index operation. While this variability in prophylaxis implementation was acceptable within the SCIP guidelines based on “high risk for bleeding” or other considerations, it likely contributed to our suboptimal outcomes. The variations and interruptions in prophylactic regimens speak to barriers that have previously been reported as contributing factors to noncompliance with VTE prophylaxis.¹⁸

Given these known barriers and the observed underutilization and improper use of our risk assessment tool, we have recently changed our surgical admission order sets such that a mandatory quantitative risk assessment must be done for every surgical patient at the time of admission/operation before other orders can be completed. Following completion of the assessment, the physician will be presented with an appropriate standard regimen based on the individual patient’s risk assessment. Early results of our VTE quality improvement project have been satisfying: in the most recent NSQIP semi-annual report, our O/E for VTE was 0.74, placing us in the first decile. Some of these early reports may simply be the product of the Hawthorne effect; however, we are encouraged by the early improvements seen in other research. While we are hopeful that these changes will result in sustainable improvements in outcomes, patients at extremely high risk may require novel weight-based or otherwise customized aggressive prophylactic regimens. Such regimens have already been proposed for arthroplasty and other high-risk patients.

Future research may identify other risk factors not captured by traditional risk assessments. In addition, research should continue to explore the use and efficacy of standard prophylactic regimens in these populations to help determine if they are sufficient. Currently, weight-based low-molecular-weight heparin dosing and alternative regimens employing fondaparinux are under investigation for very-high-risk patients.¹⁹

There were several limitations to the present study. First, due to the retrospective design of our study, we could collect only data that had been uniformly recorded in the charts throughout the study period. Second, we were unable to accurately assess compliance with mechanical prophylaxis. While our chart review showed that the vast majority of cases and controls were ordered to have mechanical prophylaxis, it is impossible to document how often these devices were used appropriately in a retrospective analysis. Anecdotal observation suggests that once patients are out of post-anesthesia or critical care units, SCD use is not standardized. The inability to measure compliance precisely may be leading to an overestimation of our compliance with prophylaxis. Finally, because our study included only patients who underwent surgery at our hospital, our observations may not be generalizable outside our institution.

Conclusion

Our study findings reinforce the importance of attention to detail in VTE risk assessment and in ordering and administering VTE prophylactic regimens, especially in high-risk surgical patients. While we adhered to the SCIP-mandated prophylaxis requirements, the complexity of our patients and our lack of a truly standardized approach to risk assessment and prophylactic regimens resulted in suboptimal outcomes. Stricter and more quantitative mandatory VTE risk assessment, along with highly standardized VTE prophylaxis regimens, are required to achieve optimal outcomes.

Corresponding author: Jason C. DeGiovanni, MS, BA, [email protected].

Financial disclosures: None.

Studies have shown that a majority of patients remain confused about their health care plans after being discharged from the hospital.^1,2 Furthermore, most patients do not recognize their lack of comprehension.² A substantial proportion of medical information is forgotten immediately after discharge. Kessels found that when larger amounts of information were presented, less was recalled, and almost half of the recalled information was incorrect.³ Researchers also have found that health information that was focused on individual needs not only increased patients’ understanding of their health needs and improved their health literacy, but supported self-management and promoted health outcomes for adults with chronic illness.^4,5

Health literacy is the “capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”⁶ To read instructions on a prescription bottle, patients need an intermediate level of health literacy. Even for patients with such a level of health literacy, comprehending and managing a health care plan for a chronic disease can be challenging. About 35% of Americans had lower than an intermediate level of health literacy.⁷ Insufficient health literacy is associated with increased health system use and costs, health disparities, and poor health outcomes.⁸ As a result, it is crucial to gear oral instructions to patients’ health literacy levels to ensure that patients understand health information and instructions and perform self-care at home. The teach-back method, a technique for verifying patients’ understanding of their health information, has been recommended by the Agency for Healthcare Research and Quality (AHRQ) and the Institute for Healthcare Improvement (IHI) as a strategy for taking universal precautions for health literacy. Patients are asked to repeat the instructions they receive from their health care professionals (HCPs). HCPs should use caring and plain language in a shame-free environment during patient education. By using the teach-back method, HCPs can assess patients’ understanding, and reteach or modify teaching if comprehension is not demonstrated. Patients have an important role in their health and their ability to understand health information has a significant impact on their health behavior and outcomes.

In our systematic research, we examined the effectiveness of using the teach-back method to understand health education as well as the impact of this method on patients’ disease self-management and health outcomes.

Methods

In the teach-back method, patients explain health information in their own words.⁹ To gauge the use and effectiveness of this method, investigators have studied patient perceptions and acknowledgments of the method as well as the effects of the method on health interventions. According to Dorothea Orem’s self-care deficit nursing theory, disease self-management is an “executive ability” to “control, handle, direct or govern” self-care activities.¹⁰ We define disease self-management as disease knowledge and disease management changes that promote self-care activities. In addition, we define health outcomes as health changes that result from the teach-back method, such as changes in postdischarge readmission rates, patient satisfaction, and health behavior.

Inclusion Criteria

We systematically reviewed evidence regarding the teach-back method as an educational intervention for patients aged ≥ 18 years. We included articles if they reported the process and outcomes of using the method alone or in combination with other educational strategies. The literature search focused on English-language articles published in peer-reviewed journals. Included in the review were qualitative, randomized controlled trials (RCTs); quasi-experimental studies; cohort studies; and pretest–posttest studies on the effects of the teach-back method. As the method can be applied in any health care setting, we used studies conducted in a variety of settings, including primary care, inpatient, outpatient, emergency department (ED), and community, in any time frame. Study participants had heart failure, diabetes mellitus (DM), hypertension, asthma, or other chronic diseases.

Exclusion Criteria

Studies that used the teach-back method as an outcome measurement but not an intervention were excluded. For example, those that used the method to measure patients’ postintervention understanding were excluded. Also excluded were those that used the method to examine HCP training or to measure HCP outcomes (ie, studies that did not use the method for patient education or outcomes).

Literature Search

In September 2017, we searched 4 databases: Ovid Medline, PubMed, EBSCO (Elton B. Stephens Co), CINAHL (Cumulative Index to Nursing and Allied Health Literature), and ProQuest. Also included were relevant studies from cited reference searching (Figure).

This systematic review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline for searches and formatting results. The literature search was performed with the term teach-back and terms from the structured PICO (population, intervention, comparison, outcomes) statement. The study population consisted of patients who received the teach-back intervention as part of the patient education process in a medical care setting, and the comparator population consisted of patients who did not receive the intervention in their patient education. Target outcomes were disease self-management, self-care, patient satisfaction, patient perception and acknowledgment of the teach-back method, and other health outcomes.

Data Collection

Data collected included authors, publication date, and journal; purpose; study design; setting, sample, and population; intervention; and outcomes.

The methodologic quality of papers retrieved for review was determined with Critical Appraisals Skills Programme (CASP) guidelines (casp-uk.net/casp-tools-checklists). CASP randomised controlled trial, cohort study, case control study, and qualitative checklists were used. The authors assessed the full texts for eligibility. Disagreements were resolved through discussion.

The initial literature search found 112, 135, and 161 articles from EBSCO CINAHL, Ovid Medline, and PubMed, respectively. Five articles from ProQuest were identified through the EBSCO CINAHL search. After inclusion and exclusion criteria were applied, duplicate articles removed, a cited reference added, and CASP criteria assessed, 26 articles remained in the review. The 26 studies consisted of 15 cohort studies, 5 case–control studies, 5 RCTs, and 1 qualitative interview. Twenty-two of the articles were published in the US, the other 4 in Australia and Iran (2 each).^11-14 All 26 studies used the teach-back method with other educational interventions to reinforce learning (eg, the method was used after heart failure or DM education). Of the 26 studies, 10 used a pretest–posttest intervention design,^15-24 and 10 used a quasi-experimental or experimental design.^{11,13,14,25-31}

Results

The common outcome measures used in the 26 studies fall into 5 categories: patient satisfaction; postdischarge readmission; patient perception of teach-back method effectiveness; disease knowledge and disease management improvements; and intervention effects on health-related quality of life (HR-QOL). A summary of included articles, study setting, design, outcomes, and details is available from the author.

Patient Satisfaction

Ten studies examined the impact of the teach-back method on patient satisfaction.^{15,17,19,21,23,26,27,29,31,32} Of these 10 studies, 6 explored the influence of the method on Hospital Consumer Assessment of Healthcare Providers and Systems survey scores.^{15,17,19,21,22,26} All included studies indicated improved satisfaction with medication education, discharge information, and health management—except for the Silva study, who found an upward trend but not a statistically significant improvement in patient understanding of the purpose of a medication.²³

Grice and colleagues also found that community-dwelling seniors expressed satisfaction with using the teach-back method while being evaluated and assessed for health services at home.³² Improvement or a positive trend in teach-back groups was reported in a majority of the studies except for those by Hyrkas and Wiggins, and Griffey and colleagues.^27,29 Hyrkas and Wiggins found the method slightly improved patients’ medication confidence after hospital discharge, though patient satisfaction scores were associated with patient–nurse relationships, not with use of the teach-back method and a motivational interview.²⁷ Similarly, Griffey and colleagues found that patients who had limited health literacy and received a standard discharge with teach-back scored higher on medication comprehension, compared with patients who received only a standard discharge, but there was no difference in patient satisfaction after an ED visit.²⁹

Postdischarge Readmission

Results emphasized the importance of teach-back in reinforcing discharge instructions and improving postdischarge readmission rates. Of the 6 studies on the effect that teach-back with discharge summary had on readmission rates, 2 found statistically significant improvement for patients with heart failure at 12 months (teach-back, 59%; non-teach-back, 44%; P = .005) and patients with coronary artery bypass grafting (CABG) at 30 days (preintervention, 25%; postintervention, 12%; P = .02).^11,16 In addition, 3 of the 6 studies reported improvement but did not provide P values.^18,20,22 One study indicated improvement in other measured outcomes but found no significant difference for patients who received teach-back with their discharge summaries.²⁷ In all studies, teach-back was added to an intervention and used to confirm and promote knowledge and self-care management.

Patient Perception of Teach-Back Effectiveness

In 2 qualitative studies, patients indicated teach-back was an effective educational method.^16,33 For patients with CABG, Bates and colleagues added a scheduled cardiology follow-up appointment and teach-back patient education to their State Action on Avoidable Rehospitalizations interventions; 96% of participants rated teach-back effective or highly effective.¹⁶ In the other study, Samuels-Kalow and colleagues interviewed 51 patients and parents who received teach-back as part of the discharge process in 2 EDs; participants indicated teach-back helped them remember what they learned from their HCPs, and gave them the opportunity to connect with their HCPs, though some with lower health literacy expressed concerns about perceived judgment by HCPs.³³

Disease Knowledge and Management

Thirteen studies examined knowledge improvement after interventions that included teach-back. Study participants answered most questions correctly after receiving teach-back.^20,32,34,35 Slater and colleagues found ED patients who received discharge instructions with teach-back had significantly higher scores measuring knowledge of diagnosis (P < .001), signs and symptoms indicating a need to return to the ED (P < .001), and follow-up instructions (P = .03); scores measuring knowledge of medication were higher as well, but were not statistically different (P = .14).²⁴ In multiple studies, improvement was not always statistically significant in terms of knowledge retention.^{12,25,29-31,36} Studies that compared medication adherence found teach-back was more effective than motivational interviews (P = .56).²⁷

Teach-back has been widely used in primary care, inpatient, and ED settings. Two studies on the effect of teach-back in primary care sampled patients with DM.^28,36 Kandula and colleagues found that participants who answered questions incorrectly after watching a multimedia DM education program could significantly improve their DM knowledge by engaging in teach-back immediately after the intervention; however, knowledge retention was not improved at 2-week follow-up (phone call).²⁸ In contrast, Swavely and colleagues compared patients who completed a 13-hour DM education program with or without teach-back and found that teach-back patients demonstrated significantly improved DM knowledge and self-care activities at 3 months.³⁶

Effects of Interventions on HR-QOL

The teach-back method had been used with QOL improvement programs and other interventions. Ahmadidarrehsima and colleagues incorporated teach-back into their medical self-management program (8 to 11 sessions, each lasting 1.5 to 2 hours) for women with breast cancer and found that the mean happiness score increased to 62.9 from 37.2 (P < .001) in the intervention group, whereas the score for the usual-care group decreased from 41.4 to 29.8.¹³ Ghiasvand and colleagues compared QOL of postpartum mothers who received routine care with QOL of those who received routine care plus 2 sessions of postpartum self-care with teach-back; mean QOL scores were significantly (P < .001) higher for the teach-back group (124.73) than for the no teach-back group (115.03).¹⁴

Discussion

This review examined the use and effectiveness of the teach-back method in health education and its influence in patients’ disease self-management and health outcomes. Results showed positive effects of teach-back on patient satisfaction, patient perceptions and acknowledgments, postdischarge readmissions, disease self-management and knowledge, and HR-QOL.

The teach-back method has been widely used in inpatient, outpatient, ED, and community settings as part of health education programs and interventions. It has been paired with educational interventions ranging from short instructions to 20-hour programs. These differences reflect the broad application of the method in patient education. Many studies have found that teach-back improves disease knowledge and self-management, though their results are not always statistically significant. In an RCT of patients with low health literacy, Griffey and colleagues studied the effect of ED discharge education with and without teach-back and found teach-back did not increase post-ED comprehension of diagnoses, medical examinations, and treatments or perceived comprehension of treatment and care; however, compared with the no teach-back group, the teach-back group had significantly higher scores on comprehension of post-ED self-care (P < .02), follow-up (P < .0001), and medication (P = .054).²⁹ This finding indicates teach-back is an effective method for helping patients understand self-care and disease self-management at home.

Comprehending medical diagnoses, examinations, and treatments involves acquiring, analyzing, and comparing multiple pieces of health information. Because comprehension requires a level of abstract thinking usually present in patients with intermediate and proficient health literacy,improvements might be more difficult to see in patients with low health literacy.⁸ Press and colleagues found that asthma patients who repeated respiratory inhaler instructions with teach-back during discharge education had less misuse of (P = .01) metered-dose and Diskus (P = .05) inhalers and lower 30-day readmission rates (P = .02) compared with the misuse of patients who received only 1 set of oral and written instructions.³¹ Even though the Diskus result was not statistically significant, it demonstrated teach-back can be used to improve patient self-care and education.³¹

Most participants in the reviewed studies improved their disease knowledge with teach-back, though the evidence regarding improved health care knowledge retention was limited. For example, the 2 studies on use of teach-back in primary care clinics had contradictory knowledge retention results.^28,36 As both studies incorporated teach-back into existing interventions, these results could be associated with those interventions and not with the teach-back method.

Health literacy is achieved through a complicated process of obtaining, analyzing, choosing, and communicating health information. Even though its knowledge retention results are inconsistent, the teach-back method is recommended by the American Academy of Family Physicians at strength of recommendation taxonomy level C.⁸ Such a designation indicates that the recommendation is based on expert opinion, bench research, consensus guideline, usual practice, clinical experience, or a case series and is appropriate for assessment of patient comprehension.³⁷ Teach-back is also suggested by AHRQ and IHI for university precautions regarding health literacy and as such should remain a standard of practice. More study is needed to understand the inconsistent results of knowledge retention and the long-term effects of the teach-back method.

Limitations

Although this review did not limit the publication years of its articles, no pre-2011 articles were found. The teach-back method has been used to measure patients’ postintervention understanding and to educate HCPs on ways to improve patient communication. As this review did not include studies of teach-back as an outcome measurement or studies of training and adaptation of teach-back in HCP or nurse education, other study results may have a bearing on the current findings. Teach-back has been used to close communication gaps between patients and HCPs.

All articles included in this review used the teach-back method with other educational or organizational interventions. The outcomes found in this review may be associated with those interventions and not with teach-back itself. Data reported here have not demonstrated a definite association between teach-back and the measured outcomes; therefore, caution should be exercised when drawing conclusions based on these data. In addition, most of the studies considered in this review were cohort or case–control studies; only 5 RCTs were included. Other confounding factors, including patient health literacy levels, HCP types, HCP competencies in use of teach-back, and type and duration of interventions used before teach-back, may have contributed to this review’s findings.

Conclusion

Findings of this systematic review support use of the teach-back method as effective in reinforcing or confirming patient education. As none of the included studies reported harmful outcomes, the teach-back method poses little risk with respect to increasing patients’ understanding of their education. The findings emphasize the importance of conducting more studies to try to understand the inconsistent results of knowledge retention and determine ways to preserve the long-term effects of teach-back.

Studies have shown that a majority of patients remain confused about their health care plans after being discharged from the hospital.^1,2 Furthermore, most patients do not recognize their lack of comprehension.² A substantial proportion of medical information is forgotten immediately after discharge. Kessels found that when larger amounts of information were presented, less was recalled, and almost half of the recalled information was incorrect.³ Researchers also have found that health information that was focused on individual needs not only increased patients’ understanding of their health needs and improved their health literacy, but supported self-management and promoted health outcomes for adults with chronic illness.^4,5

Health literacy is the “capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”⁶ To read instructions on a prescription bottle, patients need an intermediate level of health literacy. Even for patients with such a level of health literacy, comprehending and managing a health care plan for a chronic disease can be challenging. About 35% of Americans had lower than an intermediate level of health literacy.⁷ Insufficient health literacy is associated with increased health system use and costs, health disparities, and poor health outcomes.⁸ As a result, it is crucial to gear oral instructions to patients’ health literacy levels to ensure that patients understand health information and instructions and perform self-care at home. The teach-back method, a technique for verifying patients’ understanding of their health information, has been recommended by the Agency for Healthcare Research and Quality (AHRQ) and the Institute for Healthcare Improvement (IHI) as a strategy for taking universal precautions for health literacy. Patients are asked to repeat the instructions they receive from their health care professionals (HCPs). HCPs should use caring and plain language in a shame-free environment during patient education. By using the teach-back method, HCPs can assess patients’ understanding, and reteach or modify teaching if comprehension is not demonstrated. Patients have an important role in their health and their ability to understand health information has a significant impact on their health behavior and outcomes.

In our systematic research, we examined the effectiveness of using the teach-back method to understand health education as well as the impact of this method on patients’ disease self-management and health outcomes.

Methods

In the teach-back method, patients explain health information in their own words.⁹ To gauge the use and effectiveness of this method, investigators have studied patient perceptions and acknowledgments of the method as well as the effects of the method on health interventions. According to Dorothea Orem’s self-care deficit nursing theory, disease self-management is an “executive ability” to “control, handle, direct or govern” self-care activities.¹⁰ We define disease self-management as disease knowledge and disease management changes that promote self-care activities. In addition, we define health outcomes as health changes that result from the teach-back method, such as changes in postdischarge readmission rates, patient satisfaction, and health behavior.

Inclusion Criteria

We systematically reviewed evidence regarding the teach-back method as an educational intervention for patients aged ≥ 18 years. We included articles if they reported the process and outcomes of using the method alone or in combination with other educational strategies. The literature search focused on English-language articles published in peer-reviewed journals. Included in the review were qualitative, randomized controlled trials (RCTs); quasi-experimental studies; cohort studies; and pretest–posttest studies on the effects of the teach-back method. As the method can be applied in any health care setting, we used studies conducted in a variety of settings, including primary care, inpatient, outpatient, emergency department (ED), and community, in any time frame. Study participants had heart failure, diabetes mellitus (DM), hypertension, asthma, or other chronic diseases.

Exclusion Criteria

Studies that used the teach-back method as an outcome measurement but not an intervention were excluded. For example, those that used the method to measure patients’ postintervention understanding were excluded. Also excluded were those that used the method to examine HCP training or to measure HCP outcomes (ie, studies that did not use the method for patient education or outcomes).

Literature Search

In September 2017, we searched 4 databases: Ovid Medline, PubMed, EBSCO (Elton B. Stephens Co), CINAHL (Cumulative Index to Nursing and Allied Health Literature), and ProQuest. Also included were relevant studies from cited reference searching (Figure).

This systematic review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline for searches and formatting results. The literature search was performed with the term teach-back and terms from the structured PICO (population, intervention, comparison, outcomes) statement. The study population consisted of patients who received the teach-back intervention as part of the patient education process in a medical care setting, and the comparator population consisted of patients who did not receive the intervention in their patient education. Target outcomes were disease self-management, self-care, patient satisfaction, patient perception and acknowledgment of the teach-back method, and other health outcomes.

Data Collection

Data collected included authors, publication date, and journal; purpose; study design; setting, sample, and population; intervention; and outcomes.

The methodologic quality of papers retrieved for review was determined with Critical Appraisals Skills Programme (CASP) guidelines (casp-uk.net/casp-tools-checklists). CASP randomised controlled trial, cohort study, case control study, and qualitative checklists were used. The authors assessed the full texts for eligibility. Disagreements were resolved through discussion.

The initial literature search found 112, 135, and 161 articles from EBSCO CINAHL, Ovid Medline, and PubMed, respectively. Five articles from ProQuest were identified through the EBSCO CINAHL search. After inclusion and exclusion criteria were applied, duplicate articles removed, a cited reference added, and CASP criteria assessed, 26 articles remained in the review. The 26 studies consisted of 15 cohort studies, 5 case–control studies, 5 RCTs, and 1 qualitative interview. Twenty-two of the articles were published in the US, the other 4 in Australia and Iran (2 each).^11-14 All 26 studies used the teach-back method with other educational interventions to reinforce learning (eg, the method was used after heart failure or DM education). Of the 26 studies, 10 used a pretest–posttest intervention design,^15-24 and 10 used a quasi-experimental or experimental design.^{11,13,14,25-31}

Results

The common outcome measures used in the 26 studies fall into 5 categories: patient satisfaction; postdischarge readmission; patient perception of teach-back method effectiveness; disease knowledge and disease management improvements; and intervention effects on health-related quality of life (HR-QOL). A summary of included articles, study setting, design, outcomes, and details is available from the author.

Patient Satisfaction

Ten studies examined the impact of the teach-back method on patient satisfaction.^{15,17,19,21,23,26,27,29,31,32} Of these 10 studies, 6 explored the influence of the method on Hospital Consumer Assessment of Healthcare Providers and Systems survey scores.^{15,17,19,21,22,26} All included studies indicated improved satisfaction with medication education, discharge information, and health management—except for the Silva study, who found an upward trend but not a statistically significant improvement in patient understanding of the purpose of a medication.²³

Grice and colleagues also found that community-dwelling seniors expressed satisfaction with using the teach-back method while being evaluated and assessed for health services at home.³² Improvement or a positive trend in teach-back groups was reported in a majority of the studies except for those by Hyrkas and Wiggins, and Griffey and colleagues.^27,29 Hyrkas and Wiggins found the method slightly improved patients’ medication confidence after hospital discharge, though patient satisfaction scores were associated with patient–nurse relationships, not with use of the teach-back method and a motivational interview.²⁷ Similarly, Griffey and colleagues found that patients who had limited health literacy and received a standard discharge with teach-back scored higher on medication comprehension, compared with patients who received only a standard discharge, but there was no difference in patient satisfaction after an ED visit.²⁹

Postdischarge Readmission

Results emphasized the importance of teach-back in reinforcing discharge instructions and improving postdischarge readmission rates. Of the 6 studies on the effect that teach-back with discharge summary had on readmission rates, 2 found statistically significant improvement for patients with heart failure at 12 months (teach-back, 59%; non-teach-back, 44%; P = .005) and patients with coronary artery bypass grafting (CABG) at 30 days (preintervention, 25%; postintervention, 12%; P = .02).^11,16 In addition, 3 of the 6 studies reported improvement but did not provide P values.^18,20,22 One study indicated improvement in other measured outcomes but found no significant difference for patients who received teach-back with their discharge summaries.²⁷ In all studies, teach-back was added to an intervention and used to confirm and promote knowledge and self-care management.

Patient Perception of Teach-Back Effectiveness

In 2 qualitative studies, patients indicated teach-back was an effective educational method.^16,33 For patients with CABG, Bates and colleagues added a scheduled cardiology follow-up appointment and teach-back patient education to their State Action on Avoidable Rehospitalizations interventions; 96% of participants rated teach-back effective or highly effective.¹⁶ In the other study, Samuels-Kalow and colleagues interviewed 51 patients and parents who received teach-back as part of the discharge process in 2 EDs; participants indicated teach-back helped them remember what they learned from their HCPs, and gave them the opportunity to connect with their HCPs, though some with lower health literacy expressed concerns about perceived judgment by HCPs.³³

Disease Knowledge and Management

Thirteen studies examined knowledge improvement after interventions that included teach-back. Study participants answered most questions correctly after receiving teach-back.^20,32,34,35 Slater and colleagues found ED patients who received discharge instructions with teach-back had significantly higher scores measuring knowledge of diagnosis (P < .001), signs and symptoms indicating a need to return to the ED (P < .001), and follow-up instructions (P = .03); scores measuring knowledge of medication were higher as well, but were not statistically different (P = .14).²⁴ In multiple studies, improvement was not always statistically significant in terms of knowledge retention.^{12,25,29-31,36} Studies that compared medication adherence found teach-back was more effective than motivational interviews (P = .56).²⁷

Teach-back has been widely used in primary care, inpatient, and ED settings. Two studies on the effect of teach-back in primary care sampled patients with DM.^28,36 Kandula and colleagues found that participants who answered questions incorrectly after watching a multimedia DM education program could significantly improve their DM knowledge by engaging in teach-back immediately after the intervention; however, knowledge retention was not improved at 2-week follow-up (phone call).²⁸ In contrast, Swavely and colleagues compared patients who completed a 13-hour DM education program with or without teach-back and found that teach-back patients demonstrated significantly improved DM knowledge and self-care activities at 3 months.³⁶

Effects of Interventions on HR-QOL

The teach-back method had been used with QOL improvement programs and other interventions. Ahmadidarrehsima and colleagues incorporated teach-back into their medical self-management program (8 to 11 sessions, each lasting 1.5 to 2 hours) for women with breast cancer and found that the mean happiness score increased to 62.9 from 37.2 (P < .001) in the intervention group, whereas the score for the usual-care group decreased from 41.4 to 29.8.¹³ Ghiasvand and colleagues compared QOL of postpartum mothers who received routine care with QOL of those who received routine care plus 2 sessions of postpartum self-care with teach-back; mean QOL scores were significantly (P < .001) higher for the teach-back group (124.73) than for the no teach-back group (115.03).¹⁴

Discussion

This review examined the use and effectiveness of the teach-back method in health education and its influence in patients’ disease self-management and health outcomes. Results showed positive effects of teach-back on patient satisfaction, patient perceptions and acknowledgments, postdischarge readmissions, disease self-management and knowledge, and HR-QOL.

The teach-back method has been widely used in inpatient, outpatient, ED, and community settings as part of health education programs and interventions. It has been paired with educational interventions ranging from short instructions to 20-hour programs. These differences reflect the broad application of the method in patient education. Many studies have found that teach-back improves disease knowledge and self-management, though their results are not always statistically significant. In an RCT of patients with low health literacy, Griffey and colleagues studied the effect of ED discharge education with and without teach-back and found teach-back did not increase post-ED comprehension of diagnoses, medical examinations, and treatments or perceived comprehension of treatment and care; however, compared with the no teach-back group, the teach-back group had significantly higher scores on comprehension of post-ED self-care (P < .02), follow-up (P < .0001), and medication (P = .054).²⁹ This finding indicates teach-back is an effective method for helping patients understand self-care and disease self-management at home.

Comprehending medical diagnoses, examinations, and treatments involves acquiring, analyzing, and comparing multiple pieces of health information. Because comprehension requires a level of abstract thinking usually present in patients with intermediate and proficient health literacy,improvements might be more difficult to see in patients with low health literacy.⁸ Press and colleagues found that asthma patients who repeated respiratory inhaler instructions with teach-back during discharge education had less misuse of (P = .01) metered-dose and Diskus (P = .05) inhalers and lower 30-day readmission rates (P = .02) compared with the misuse of patients who received only 1 set of oral and written instructions.³¹ Even though the Diskus result was not statistically significant, it demonstrated teach-back can be used to improve patient self-care and education.³¹

Most participants in the reviewed studies improved their disease knowledge with teach-back, though the evidence regarding improved health care knowledge retention was limited. For example, the 2 studies on use of teach-back in primary care clinics had contradictory knowledge retention results.^28,36 As both studies incorporated teach-back into existing interventions, these results could be associated with those interventions and not with the teach-back method.

Health literacy is achieved through a complicated process of obtaining, analyzing, choosing, and communicating health information. Even though its knowledge retention results are inconsistent, the teach-back method is recommended by the American Academy of Family Physicians at strength of recommendation taxonomy level C.⁸ Such a designation indicates that the recommendation is based on expert opinion, bench research, consensus guideline, usual practice, clinical experience, or a case series and is appropriate for assessment of patient comprehension.³⁷ Teach-back is also suggested by AHRQ and IHI for university precautions regarding health literacy and as such should remain a standard of practice. More study is needed to understand the inconsistent results of knowledge retention and the long-term effects of the teach-back method.

Limitations

Although this review did not limit the publication years of its articles, no pre-2011 articles were found. The teach-back method has been used to measure patients’ postintervention understanding and to educate HCPs on ways to improve patient communication. As this review did not include studies of teach-back as an outcome measurement or studies of training and adaptation of teach-back in HCP or nurse education, other study results may have a bearing on the current findings. Teach-back has been used to close communication gaps between patients and HCPs.

All articles included in this review used the teach-back method with other educational or organizational interventions. The outcomes found in this review may be associated with those interventions and not with teach-back itself. Data reported here have not demonstrated a definite association between teach-back and the measured outcomes; therefore, caution should be exercised when drawing conclusions based on these data. In addition, most of the studies considered in this review were cohort or case–control studies; only 5 RCTs were included. Other confounding factors, including patient health literacy levels, HCP types, HCP competencies in use of teach-back, and type and duration of interventions used before teach-back, may have contributed to this review’s findings.

Conclusion

Findings of this systematic review support use of the teach-back method as effective in reinforcing or confirming patient education. As none of the included studies reported harmful outcomes, the teach-back method poses little risk with respect to increasing patients’ understanding of their education. The findings emphasize the importance of conducting more studies to try to understand the inconsistent results of knowledge retention and determine ways to preserve the long-term effects of teach-back.

Studies have shown that a majority of patients remain confused about their health care plans after being discharged from the hospital.^1,2 Furthermore, most patients do not recognize their lack of comprehension.² A substantial proportion of medical information is forgotten immediately after discharge. Kessels found that when larger amounts of information were presented, less was recalled, and almost half of the recalled information was incorrect.³ Researchers also have found that health information that was focused on individual needs not only increased patients’ understanding of their health needs and improved their health literacy, but supported self-management and promoted health outcomes for adults with chronic illness.^4,5

Health literacy is the “capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”⁶ To read instructions on a prescription bottle, patients need an intermediate level of health literacy. Even for patients with such a level of health literacy, comprehending and managing a health care plan for a chronic disease can be challenging. About 35% of Americans had lower than an intermediate level of health literacy.⁷ Insufficient health literacy is associated with increased health system use and costs, health disparities, and poor health outcomes.⁸ As a result, it is crucial to gear oral instructions to patients’ health literacy levels to ensure that patients understand health information and instructions and perform self-care at home. The teach-back method, a technique for verifying patients’ understanding of their health information, has been recommended by the Agency for Healthcare Research and Quality (AHRQ) and the Institute for Healthcare Improvement (IHI) as a strategy for taking universal precautions for health literacy. Patients are asked to repeat the instructions they receive from their health care professionals (HCPs). HCPs should use caring and plain language in a shame-free environment during patient education. By using the teach-back method, HCPs can assess patients’ understanding, and reteach or modify teaching if comprehension is not demonstrated. Patients have an important role in their health and their ability to understand health information has a significant impact on their health behavior and outcomes.

In our systematic research, we examined the effectiveness of using the teach-back method to understand health education as well as the impact of this method on patients’ disease self-management and health outcomes.

Methods

In the teach-back method, patients explain health information in their own words.⁹ To gauge the use and effectiveness of this method, investigators have studied patient perceptions and acknowledgments of the method as well as the effects of the method on health interventions. According to Dorothea Orem’s self-care deficit nursing theory, disease self-management is an “executive ability” to “control, handle, direct or govern” self-care activities.¹⁰ We define disease self-management as disease knowledge and disease management changes that promote self-care activities. In addition, we define health outcomes as health changes that result from the teach-back method, such as changes in postdischarge readmission rates, patient satisfaction, and health behavior.

Inclusion Criteria

We systematically reviewed evidence regarding the teach-back method as an educational intervention for patients aged ≥ 18 years. We included articles if they reported the process and outcomes of using the method alone or in combination with other educational strategies. The literature search focused on English-language articles published in peer-reviewed journals. Included in the review were qualitative, randomized controlled trials (RCTs); quasi-experimental studies; cohort studies; and pretest–posttest studies on the effects of the teach-back method. As the method can be applied in any health care setting, we used studies conducted in a variety of settings, including primary care, inpatient, outpatient, emergency department (ED), and community, in any time frame. Study participants had heart failure, diabetes mellitus (DM), hypertension, asthma, or other chronic diseases.

Exclusion Criteria

Studies that used the teach-back method as an outcome measurement but not an intervention were excluded. For example, those that used the method to measure patients’ postintervention understanding were excluded. Also excluded were those that used the method to examine HCP training or to measure HCP outcomes (ie, studies that did not use the method for patient education or outcomes).

Literature Search

In September 2017, we searched 4 databases: Ovid Medline, PubMed, EBSCO (Elton B. Stephens Co), CINAHL (Cumulative Index to Nursing and Allied Health Literature), and ProQuest. Also included were relevant studies from cited reference searching (Figure).

This systematic review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline for searches and formatting results. The literature search was performed with the term teach-back and terms from the structured PICO (population, intervention, comparison, outcomes) statement. The study population consisted of patients who received the teach-back intervention as part of the patient education process in a medical care setting, and the comparator population consisted of patients who did not receive the intervention in their patient education. Target outcomes were disease self-management, self-care, patient satisfaction, patient perception and acknowledgment of the teach-back method, and other health outcomes.

Data Collection

Data collected included authors, publication date, and journal; purpose; study design; setting, sample, and population; intervention; and outcomes.

The methodologic quality of papers retrieved for review was determined with Critical Appraisals Skills Programme (CASP) guidelines (casp-uk.net/casp-tools-checklists). CASP randomised controlled trial, cohort study, case control study, and qualitative checklists were used. The authors assessed the full texts for eligibility. Disagreements were resolved through discussion.

The initial literature search found 112, 135, and 161 articles from EBSCO CINAHL, Ovid Medline, and PubMed, respectively. Five articles from ProQuest were identified through the EBSCO CINAHL search. After inclusion and exclusion criteria were applied, duplicate articles removed, a cited reference added, and CASP criteria assessed, 26 articles remained in the review. The 26 studies consisted of 15 cohort studies, 5 case–control studies, 5 RCTs, and 1 qualitative interview. Twenty-two of the articles were published in the US, the other 4 in Australia and Iran (2 each).^11-14 All 26 studies used the teach-back method with other educational interventions to reinforce learning (eg, the method was used after heart failure or DM education). Of the 26 studies, 10 used a pretest–posttest intervention design,^15-24 and 10 used a quasi-experimental or experimental design.^{11,13,14,25-31}

Results

The common outcome measures used in the 26 studies fall into 5 categories: patient satisfaction; postdischarge readmission; patient perception of teach-back method effectiveness; disease knowledge and disease management improvements; and intervention effects on health-related quality of life (HR-QOL). A summary of included articles, study setting, design, outcomes, and details is available from the author.

Patient Satisfaction

Ten studies examined the impact of the teach-back method on patient satisfaction.^{15,17,19,21,23,26,27,29,31,32} Of these 10 studies, 6 explored the influence of the method on Hospital Consumer Assessment of Healthcare Providers and Systems survey scores.^{15,17,19,21,22,26} All included studies indicated improved satisfaction with medication education, discharge information, and health management—except for the Silva study, who found an upward trend but not a statistically significant improvement in patient understanding of the purpose of a medication.²³

Grice and colleagues also found that community-dwelling seniors expressed satisfaction with using the teach-back method while being evaluated and assessed for health services at home.³² Improvement or a positive trend in teach-back groups was reported in a majority of the studies except for those by Hyrkas and Wiggins, and Griffey and colleagues.^27,29 Hyrkas and Wiggins found the method slightly improved patients’ medication confidence after hospital discharge, though patient satisfaction scores were associated with patient–nurse relationships, not with use of the teach-back method and a motivational interview.²⁷ Similarly, Griffey and colleagues found that patients who had limited health literacy and received a standard discharge with teach-back scored higher on medication comprehension, compared with patients who received only a standard discharge, but there was no difference in patient satisfaction after an ED visit.²⁹

Postdischarge Readmission

Results emphasized the importance of teach-back in reinforcing discharge instructions and improving postdischarge readmission rates. Of the 6 studies on the effect that teach-back with discharge summary had on readmission rates, 2 found statistically significant improvement for patients with heart failure at 12 months (teach-back, 59%; non-teach-back, 44%; P = .005) and patients with coronary artery bypass grafting (CABG) at 30 days (preintervention, 25%; postintervention, 12%; P = .02).^11,16 In addition, 3 of the 6 studies reported improvement but did not provide P values.^18,20,22 One study indicated improvement in other measured outcomes but found no significant difference for patients who received teach-back with their discharge summaries.²⁷ In all studies, teach-back was added to an intervention and used to confirm and promote knowledge and self-care management.

Patient Perception of Teach-Back Effectiveness

In 2 qualitative studies, patients indicated teach-back was an effective educational method.^16,33 For patients with CABG, Bates and colleagues added a scheduled cardiology follow-up appointment and teach-back patient education to their State Action on Avoidable Rehospitalizations interventions; 96% of participants rated teach-back effective or highly effective.¹⁶ In the other study, Samuels-Kalow and colleagues interviewed 51 patients and parents who received teach-back as part of the discharge process in 2 EDs; participants indicated teach-back helped them remember what they learned from their HCPs, and gave them the opportunity to connect with their HCPs, though some with lower health literacy expressed concerns about perceived judgment by HCPs.³³

Disease Knowledge and Management

Thirteen studies examined knowledge improvement after interventions that included teach-back. Study participants answered most questions correctly after receiving teach-back.^20,32,34,35 Slater and colleagues found ED patients who received discharge instructions with teach-back had significantly higher scores measuring knowledge of diagnosis (P < .001), signs and symptoms indicating a need to return to the ED (P < .001), and follow-up instructions (P = .03); scores measuring knowledge of medication were higher as well, but were not statistically different (P = .14).²⁴ In multiple studies, improvement was not always statistically significant in terms of knowledge retention.^{12,25,29-31,36} Studies that compared medication adherence found teach-back was more effective than motivational interviews (P = .56).²⁷

Teach-back has been widely used in primary care, inpatient, and ED settings. Two studies on the effect of teach-back in primary care sampled patients with DM.^28,36 Kandula and colleagues found that participants who answered questions incorrectly after watching a multimedia DM education program could significantly improve their DM knowledge by engaging in teach-back immediately after the intervention; however, knowledge retention was not improved at 2-week follow-up (phone call).²⁸ In contrast, Swavely and colleagues compared patients who completed a 13-hour DM education program with or without teach-back and found that teach-back patients demonstrated significantly improved DM knowledge and self-care activities at 3 months.³⁶

Effects of Interventions on HR-QOL

The teach-back method had been used with QOL improvement programs and other interventions. Ahmadidarrehsima and colleagues incorporated teach-back into their medical self-management program (8 to 11 sessions, each lasting 1.5 to 2 hours) for women with breast cancer and found that the mean happiness score increased to 62.9 from 37.2 (P < .001) in the intervention group, whereas the score for the usual-care group decreased from 41.4 to 29.8.¹³ Ghiasvand and colleagues compared QOL of postpartum mothers who received routine care with QOL of those who received routine care plus 2 sessions of postpartum self-care with teach-back; mean QOL scores were significantly (P < .001) higher for the teach-back group (124.73) than for the no teach-back group (115.03).¹⁴

Discussion

This review examined the use and effectiveness of the teach-back method in health education and its influence in patients’ disease self-management and health outcomes. Results showed positive effects of teach-back on patient satisfaction, patient perceptions and acknowledgments, postdischarge readmissions, disease self-management and knowledge, and HR-QOL.

The teach-back method has been widely used in inpatient, outpatient, ED, and community settings as part of health education programs and interventions. It has been paired with educational interventions ranging from short instructions to 20-hour programs. These differences reflect the broad application of the method in patient education. Many studies have found that teach-back improves disease knowledge and self-management, though their results are not always statistically significant. In an RCT of patients with low health literacy, Griffey and colleagues studied the effect of ED discharge education with and without teach-back and found teach-back did not increase post-ED comprehension of diagnoses, medical examinations, and treatments or perceived comprehension of treatment and care; however, compared with the no teach-back group, the teach-back group had significantly higher scores on comprehension of post-ED self-care (P < .02), follow-up (P < .0001), and medication (P = .054).²⁹ This finding indicates teach-back is an effective method for helping patients understand self-care and disease self-management at home.

Comprehending medical diagnoses, examinations, and treatments involves acquiring, analyzing, and comparing multiple pieces of health information. Because comprehension requires a level of abstract thinking usually present in patients with intermediate and proficient health literacy,improvements might be more difficult to see in patients with low health literacy.⁸ Press and colleagues found that asthma patients who repeated respiratory inhaler instructions with teach-back during discharge education had less misuse of (P = .01) metered-dose and Diskus (P = .05) inhalers and lower 30-day readmission rates (P = .02) compared with the misuse of patients who received only 1 set of oral and written instructions.³¹ Even though the Diskus result was not statistically significant, it demonstrated teach-back can be used to improve patient self-care and education.³¹

Most participants in the reviewed studies improved their disease knowledge with teach-back, though the evidence regarding improved health care knowledge retention was limited. For example, the 2 studies on use of teach-back in primary care clinics had contradictory knowledge retention results.^28,36 As both studies incorporated teach-back into existing interventions, these results could be associated with those interventions and not with the teach-back method.

Health literacy is achieved through a complicated process of obtaining, analyzing, choosing, and communicating health information. Even though its knowledge retention results are inconsistent, the teach-back method is recommended by the American Academy of Family Physicians at strength of recommendation taxonomy level C.⁸ Such a designation indicates that the recommendation is based on expert opinion, bench research, consensus guideline, usual practice, clinical experience, or a case series and is appropriate for assessment of patient comprehension.³⁷ Teach-back is also suggested by AHRQ and IHI for university precautions regarding health literacy and as such should remain a standard of practice. More study is needed to understand the inconsistent results of knowledge retention and the long-term effects of the teach-back method.

Limitations

Although this review did not limit the publication years of its articles, no pre-2011 articles were found. The teach-back method has been used to measure patients’ postintervention understanding and to educate HCPs on ways to improve patient communication. As this review did not include studies of teach-back as an outcome measurement or studies of training and adaptation of teach-back in HCP or nurse education, other study results may have a bearing on the current findings. Teach-back has been used to close communication gaps between patients and HCPs.

All articles included in this review used the teach-back method with other educational or organizational interventions. The outcomes found in this review may be associated with those interventions and not with teach-back itself. Data reported here have not demonstrated a definite association between teach-back and the measured outcomes; therefore, caution should be exercised when drawing conclusions based on these data. In addition, most of the studies considered in this review were cohort or case–control studies; only 5 RCTs were included. Other confounding factors, including patient health literacy levels, HCP types, HCP competencies in use of teach-back, and type and duration of interventions used before teach-back, may have contributed to this review’s findings.

Conclusion

Findings of this systematic review support use of the teach-back method as effective in reinforcing or confirming patient education. As none of the included studies reported harmful outcomes, the teach-back method poses little risk with respect to increasing patients’ understanding of their education. The findings emphasize the importance of conducting more studies to try to understand the inconsistent results of knowledge retention and determine ways to preserve the long-term effects of teach-back.

Providing consistent and high-quality services to gender diverse patients is a top priority for health care systems, including the Veterans Health Administration (VHA).¹ Over the past decade, awareness of transgender and gender nonconforming (TGNC) people in the US has increased. Gender identity refers to a person’s inner sense of where that person belongs on a continuum of masculine to androgynous to feminine traits. This identity range can additionally include nonbinary identifications, such as “gender fluid” or “genderqueer.” A goal of patient-centered care is for health care providers (HCPs) to refer to TGNC individuals, like their cisgender counterparts, according to their gender identity. Gender identity for TGNC individuals may be different from their birth sex. Birth sex, commonly referred to as “sex assigned at birth,” is the biologic and physiologic characteristics that are reflected on a person’s original birth certificate and described as male or female.

Background

In the electronic health record (EHR), birth sex is an important, structured variable that is used to facilitate effective patient care that is efficient, equitable, and patient-centered. Birth sex in an EHR often is used to cue automatic timely generation of health screens (eg, pap smears, prostate exams) and calculation of medication dosages and laboratory test ranges by adjusting for a person’s typical hormonal history and anatomy.

Gender identity fields are independently helpful to include in the EHR, because clinicians can use this information to ensure proper pronoun use and avoid misgendering a patient. Additionally, the gender identity field informs HCPs who may conduct more frequent or different health screenings to evaluate specific health risks that are more prevalent in gender minority (ie, lesbian, gay, bisexual) patients.^2,3

EHRs rely on structured data elements to standardize data about patients for clinical care, quality improvement, data sharing, and patient safety.^4,5 However, health care organizations are grappling with how to incorporate gender identity and birth sex information into EHRs.³ A 2011 Veterans Health Administration (VHA) directive required staff and providers to address and provide care to veterans based on their gender identity. Like other health systems, VHA had 1 demographic data field in the EHR to indicate birth sex, with no field for gender identity. A HCP could enter gender identity information into a progress note, but this addition might not be noticed by other HCPs. Consequently, staff and providers had no effective way of knowing a veteran’s gender identity from the EHR, which contributed to misgendering TGNC veterans.

With the singular demographic field of sex representing both birth sex and gender identity, some TGNC veterans chose to change their birth sex information to align with their gender identity. This change assured TGNC veterans that staff and providers would not misgender them because the birth sex field is easily observed and would allow providers to use respectful, gender-consistent pronouns when speaking with them. However, changing the birth sex field can misalign natal sex–based clinical reminders, medication dosages, and laboratory test values, which created potential patient safety risks. Thus, birth sex created potential hazards to quality and safety when used as a marker even with other variables—such as current anatomy, height, and weight—for health screenings, medication dosing, and other medical decisions.

In this article, we: (1) outline several patient safety issues that can arise with the birth sex field serving as an indicator for both birth sex and gender identity; (2) present case examples that illustrate the benefits of self-identified gender identity (SIGI) in an EHR; (3) describe the process of work-group development of patient-provider communication tools to improve patient safety; and (4) provide a brief overview of resources rolled out as a part of SIGI. This report serves as a guide for other federal organizations that wish to increase affirmative care and safe practices for transgender consumers. We will provide an overview of the tasks leading up to SIGI implementation, deliverables from the project, and lessons learned.

Veterans Affairs SIGI EHR Field

In 2016, the US Department of Veterans Affairs (VA) began implementing a SIGI demographic field across all EHRs, requiring administrative staff to ask enrolled and new veterans their gender identity (full implementation of SIGI has not yet occurred and will occur when a later EHR upgrade displays SIGI in the EHR). The initiation of SIGI did not change any information in the birth sex field, meaning that some veterans continue to have birth sex field information that results in problematic automatic medical reminders and dosing values. Consequently, the National Center for Patient Safety (NCPS) noted that this discrepancy may be a pertinent patient safety issue. The NCPS and Lesbian, Gay Bisexual, and Transgender (LGBT) Health national program offices worked to provide documentation to TGNC veterans to inform them of the clinical health care implications of having their birth sex demographic field reflect gender identity that is inconsistent with their natal sex (ie, original birth certificate record of sex).

Patient Safety Issues

Conversations between transgender patients and their HCPs about transition goals, necessary medical tests, and laboratory ranges based on their current anatomy and physiology can improve patient safety and satisfaction with medical care. Prior to the availability of the SIGI field, VA facilities varied in their documentation of gender identity in the patient chart. LGBT veteran care coordinators discussed diverse suggestions that ranged from informally documenting SIGI in each progress note to using flags to draw attention to use certain sections of local EHRs. These suggestions, though well intentioned, were not adequate for documenting gender identity at the national level because of regional variations in EHR customization options. Furthermore, the use of flags for drawing clinical attention to gender identity posed a potential for stigma toward patients, given that flags are typically reserved for behavioral or other risk concerns.

Several problems can emerge when HCPs are not equipped with accurate information about patient birth sex and SIGI. For instance, TGNC patients lack a way of being known from clinic to clinic by proper pronouns or self-labels. Providers may misgender veterans, which is a negative experience for TGNC veterans linked with increased barriers to care and decreased frequency of health care visits.⁴ Moreover, the quality and personalization of care across clinic locations in the facility’s system is variable without a consistent method of documenting birth sex and SIGI. For example, in clinics where the veteran is well known (eg, primary care), staff may be more affirming of the veteran’s gender identity than those in specialty care clinics that lack prior patient contact.

Furthermore, depending on hormone and surgical interventions, some health screenings may be irrelevant for TGNC patients. To determine appropriate health screens and assess potential risks associated with hormone therapy, providers must have access to current information regarding a patient’s physiologic anatomy.⁶ Health screenings and laboratory results in sophisticated EHRs (ie, EHRs that might autodetermine normative values) may populate incorrect treatment recommendations, such as sex-based medication dosages. Furthermore, laboratory test results could be incorrectly paired with a different assumed hormonal history, potentially putting the patient at risk.

Case Examples

An important element of EHRs facilitating the goal of patient-centered care is that patients have their EHR validate their sense of self, and their providers can use names and pronouns that correspond to the patient’s SIGI. Some patients have spent a great amount of effort altering their name and sex in legal records and may want their birth sex field to conform to their gender identity. To that end, patients may seek to alter their birth sex information so that it is congruent with how they see themselves to affirm their identity, despite patient safety risks. Several scenarios below demonstrate the potential costs and benefits to patients altering birth sex and SIGI in the EHR.

Case 1 Presentation

A young transman is working with his therapist on engaging in self-validating behaviors. This veteran has met with his PCP and informed the provider of his decision to alter the birth sex field in his EHR from female to male.

Ideally, the patient would begin to have regular conversations with his HCPs about his birth sex and gender identity, so that medical professionals can provide relevant screenings and affirm the patient’s gender identity while acknowledging his right to list his birth sex as he chooses. However, particular attention will need to be paid to assuring that natal sex–based health screenings (eg, pap smears, mammograms) are conducted on an appropriate schedule and that the veteran continues to discuss his current anatomy with providers.

Case 2 Presentation

A veteran has a male birth sex, identifies as a transwoman, and uses nongendered plural pronouns “they/them/theirs.” The word “they,” used as a singular pronoun may feel uncomfortable to some providers, but it validates the veteran’s sense of self and helps them feel welcome in the treatment environment. This patient communicated proactively with their HCPs about their transition goals and current hormone use.

They opted to have their birth sex field continue to indicate “male” because they, after a discussion with their PCP, are aware of the health implications of receiving an incorrect dose for their diabetes medication. They understand that having open communication and receiving input from their HCPs is part of good health care.

Case 3 Presentation

A patient with a sexual development disorder (intersex condition) identifies as a man (indicated as “male” in the SIGI field) and had his birth sex field changed to match his gender identity. He now seeks to change his birth sex field back to female, as he has complicated health considerations due to breast cancer.

The veteran thinks it is important that providers know about his intersex condition so that his breast cancer care is as seamless as possible. In particular, although this veteran is comfortable talking about his intersex condition and his identity with his PCP and oncologist, he wants to ensure that all people involved in his care (eg, pharmacists, radiologists) use the correct values in interpreting his medical data. Providers will need to use the female birth sex field for interpreting his medical data but use male pronouns when interacting with the veteran and documenting his care.

These case examples illustrate the need for HCPs to have patient-affirming education and appropriate clinical tools available when speaking to patients about birth sex, SIGI, and the implications of changing birth sex in the EHR. Moreover, these cases highlight that patient health needs may vary over time, due to factors such as perceived costs/benefits of a change in the sex field of the EHR as well as patient comfort with providers.

Current Status of SIGI and EHR

Although having separate fields for birth sex and SIGI in the EHR is ideal, the VHA does not yet have a fully functional SIGI field, and several TGNC veterans have changed their birth sex field to align with their gender identity. Roughly 9,700 patients have diagnostic codes related to transgender care in the VHA, meaning thousands of current patients would potentially benefit from SIGI implementation (John Blosnich, written communication, March 2018). A possible action that the VHA could take with the goal of enhancing patient safety would be to revert the birth sex field of patients who had previously changed the field back to the patient’s original birth sex. However, if this alteration to the EHR were done without the patient’s consent, numerous additional problems would result—including invalidating a veteran’s wishes—potentially driving patients away from receiving health care.

Moreover, in the absence of updated SIGI information (which only the veteran can provide), making a change in the EHR would perpetuate the misgendering of TGNC veterans who have already sought an administrative fix for this problem. Thus, the agency decided to engage patients in a discussion about their decision to keep the birth sex field consistent with their original birth certificate. In cases in which the field had been changed previously, the recommendation is for HCPs to gain patient consent to change the birth sex field back to what was on their original birth certificate. Thus, decisions about what should be listed in the EHR are made by the veteran using an informed decision-making model.

Patient Safety Education Workgroup

To begin the process of disentangling birth sex and SIGI fields in the EHR, 2 work groups were created: a technical work group (coding the patches for SIGI implementation) and a SIGI patient safety education work group. The patient safety education work group was committed to promoting affirmative VA policies that require validation of the gender identity of all veterans and pursuing best practices through clinical guidelines to promote effective, efficient, equitable, and safe veteran care. The patient safety education work group included representatives from all 3 branches of the VA (VHA, Veterans Benefits Administration, and National Cemetery Administration), including clinical media, patient safety, information technology, and education specialists. The group developed trainings for administrative staff about the appropriate ways to ask birth sex and SIGI questions, and how to record veteran-driven responses.

SIGI Fact Sheet

The patient safety education work group examined clinical literature and developed tools for staff and veterans to facilitate effective discussions about the importance and utility of documenting both birth sex and SIGI in the EHR. The patient safety education work group along with media and educational experts created basic key term definition documents to address the importance, purpose, and use of the SIGI field. The patient safety education work group developed 2 documents to facilitate communication between patients and providers.

A 1-page veteran-facing fact sheet was developed that described the differences between birth sex and SIGI fields and how these fields are used in the VA EHR system (Figure 1). In addition, a 1-page HCP-facing fact sheet was designed to inform HCPs that patients may have changed their birth sex in their EHR or might still wish to change their birth sex field, and to inform HCPs of the importance of patient-centered, gender-affirmative care (Figure 2). An additional goal of both documents was to educate veterans and HCPs on how the EHR automatically calculates laboratory results and screening notifications based on birth sex.

Review Process

As part of reviewing and finalizing the SIGI patient fact sheet, the patient safety education work group previewed the document’s content with veterans who provided feedback on drafts to improve comprehension, patient-centered care, and clinical accuracy. For instance, several patients commented that the document should address many gender identities, including intersex identities. As noted in one of the case presentations earlier, individuals who identify as intersex may have changed their birth sex to be consistent with their gender and might benefit from being informed about the EHR’s autocalculation feature. The patient safety education work group adjusted the SIGI patient fact sheet to include individuals who identify as intersex and instructed them to have a conversation with their HCP regarding potential birth sex changes in the EHR.

Much of the veteran feedback to the patient safety education work group reflected veteran concerns, more broadly, about implementation of SIGI. Many veterans were interested in how federal policy changes might affect their benefits package or clinical care within the VA. The SIGI patient fact sheet was a tool for communicating that Department of Defense (DoD) policies, specifically, do not have a bearing on VA care for LGBT veterans. Therefore, SIGI information does not affect service connection or benefits eligibility and is not shared with the DoD. Veterans found this information helpful to see reflected in the SIGI patient fact sheet.

The patient safety education work group also shared the SIGI provider fact sheet with VHA providers before finalizing the content. PCPs gave feedback to improve the specification of patient safety concerns and appropriate readership language. The patient safety education work group adjusted the SIGI provider fact sheet to be inclusive of relevant literature and an e-consultation link for assisting HCPs who are unsure how to proceed with a patient.

Implementation

The patient safety education work group also developed several materials to provide information about the birth sex and SIGI fields in the EHR. Because the SIGI demographic field is new and collected by clerical staff, training was necessary to explain the difference between birth sex and SIGI before implementation in the EHR. The training sessions educated staff about the difference between birth sex and SIGI, how to ask and respond to questions respectfully, and how to update these fields in the EHR. These trainings included a 20-minute video demonstrating best practices for asking about SIGI, a frequently asked questions document responding to 7 common questions about the new fields, and a quick reference guide for administrative staff to have handy at their desks.

Dissemination of the SIGI patient and provider fact sheets is planned to occur, ideally, several weeks before implementation of the new patches updating the EHR fields in spring 2020. Building on existing resources, the patient safety education work group plans to disseminate the patient fact sheets via e-mail lists for the national mental health facility leaders as well as through e-mail lists for VA PCPs, nursing and clerical staff, privacy officers, facility LGBT veteran care coordinators, VISN leads, transgender e-consultation, the Office of Connected Care, the LGBT external homepage for the VA, and the training website for VA employees. The goal is to target potential points of contact for veterans who may have already changed their birth sex and might benefit medically from altering birth sex to be consistent with their original birth certificate.

The SIGI provider fact sheet will be disseminated using internal e-mails, announcements on routine LGBT veteran care coordinator calls, weekly Ask LGBT Health teleconferences, and announcements at LGBT health training events both internally and externally. Several dissemination tools have already ensured that VA employees are aware of the SIGI field in the EHR. Leadership throughout the VA will be encouraged to share SIGI trainings with clerical staff. Additionally, broad-based e-mails summarizing changes to the EHR will be provided concurrent to the SIGI patch implementation to VA staff as well as links to the resources and training materials.

Challenges

One difficulty in the development process for both SIGI fact sheets was addressing the issue of patient safety for veterans who may be at different points in their gender transition process. It was challenging for the patient safety education work group to not sound alarmist in discussing the safety implications of birth sex changes in the EHR, as this is just one factor in clinical decision making. The goal was to educate veterans from a patient safety perspective about the implications of having a state-of-the-art, automated EHR. However, text can be perceived differently by different people, which is why the patient safety education work group asked veterans to preview the patient document and clinical providers to preview the provider document.

Both work groups encountered technologic challenges, including a delay in the implementation of the SIGI field due to a systemwide delay of EHR updates. Although it released training and educational materials to the VHA, the patient safety education work group understood that at some point in the future, VA programmers will update the EHR to change the information clerks and HCPs can see in the EHR. Coordination of the fact sheet release alongside information technology has been an important part of the SIGI rollout process.

Conclusion

HCPs have a complex role in providing treatment to TGNC patients in the VHA: They must affirm a patient’s gender identity through how they address them, while openly communicating the health risks inherent in having their birth sex field be incongruent with the sex recorded on their original birth certificate. Accomplishing these tasks simultaneously is difficult without invalidating the veteran’s identity or right to choose their EHR demographic birth sex label. Furthermore, patients may ask HCPs to write letters of support for either medical or surgical intervention or other documentation changes (eg, changes to a patient’s legal name, passport changes, or a safe passage letter for TGNC patients). Navigating the dialectic of safety and validation requires strong rapport, trust, and effective communication in the patient-provider relationship and great empathy by the provider.

A future task for the SIGI patient safety education work group is to continue to communicate with the technical work group and providers in the field about how demographic fields in the EHR are utilized to enable future EHR changes. This hurdle is not easy because EHR updates change the infrastructure through which demographic content is delivered and incorporated into a patient’s treatment. The VA HCPs are tasked with thoroughly examining the results that automated systems produce to ensure safe and accurate medical services are always provided to all patients. An integral part of patient-centered care is balancing any computer-guided recommendations with an understanding that actual patient needs may differ due to presence/absence of anatomy and other factors (eg, weight, current medications).

From a systems perspective, a benefit of adding the SIGI demographic field is systemic improvement in calculating the number of transgender veterans under VA care and evaluating health outcomes for this population. SIGI is particularly important for signaling gender pronouns for veterans, regardless of whether they are receiving care for a gender-related diagnosis. In terms of scope, the SIGI project potentially will apply to > 9 million enrolled veterans and nearly 400,000 VA employees.

Improvements could be made in the SIGI field of the new EHR, such as expanding the options for self-labels. Additionally, a text field could be used to enhance the quality of personalization provided to veterans self-identifying in the EHR, including pronoun specification. Moreover, adding new fields such as “preferred name” could improve the health care experience of not only TGNC veterans but all veterans who use something other than their full legal name (eg, a nickname). It will be good practice to notify providers and staff of a veteran’s requested name and pronouns when the patient checks in at an electronic kiosk so that all staff immediately know how to address the patient. The VHA can continue to adjust the options for the SIGI field once the new EHR system is operational. Ideally, this new EHR will display birth sex and SIGI to clinicians or clerks engaged in patient interactions.

Technology will continue to automate medical care, meaning that HCPs must be vigilant about how computer programming and the accuracy of prepopulated information affect patient care. The concerns discussed in this report relating to patient safety are relatively absent in the medical literature, even though substantial health risks exist to patients who have birth sex listed incorrectly for any reason.^6,7 Additionally, administrative burden can be reduced if patients who do not need certain screenings based on their current anatomy are not contacted for unnecessary screenings. Future EHR systems might incorporate anatomical considerations from an inventory to assist in automating patient care in safe and accessible ways.

Providing consistent and high-quality services to gender diverse patients is a top priority for health care systems, including the Veterans Health Administration (VHA).¹ Over the past decade, awareness of transgender and gender nonconforming (TGNC) people in the US has increased. Gender identity refers to a person’s inner sense of where that person belongs on a continuum of masculine to androgynous to feminine traits. This identity range can additionally include nonbinary identifications, such as “gender fluid” or “genderqueer.” A goal of patient-centered care is for health care providers (HCPs) to refer to TGNC individuals, like their cisgender counterparts, according to their gender identity. Gender identity for TGNC individuals may be different from their birth sex. Birth sex, commonly referred to as “sex assigned at birth,” is the biologic and physiologic characteristics that are reflected on a person’s original birth certificate and described as male or female.

Background

In the electronic health record (EHR), birth sex is an important, structured variable that is used to facilitate effective patient care that is efficient, equitable, and patient-centered. Birth sex in an EHR often is used to cue automatic timely generation of health screens (eg, pap smears, prostate exams) and calculation of medication dosages and laboratory test ranges by adjusting for a person’s typical hormonal history and anatomy.

Gender identity fields are independently helpful to include in the EHR, because clinicians can use this information to ensure proper pronoun use and avoid misgendering a patient. Additionally, the gender identity field informs HCPs who may conduct more frequent or different health screenings to evaluate specific health risks that are more prevalent in gender minority (ie, lesbian, gay, bisexual) patients.^2,3

EHRs rely on structured data elements to standardize data about patients for clinical care, quality improvement, data sharing, and patient safety.^4,5 However, health care organizations are grappling with how to incorporate gender identity and birth sex information into EHRs.³ A 2011 Veterans Health Administration (VHA) directive required staff and providers to address and provide care to veterans based on their gender identity. Like other health systems, VHA had 1 demographic data field in the EHR to indicate birth sex, with no field for gender identity. A HCP could enter gender identity information into a progress note, but this addition might not be noticed by other HCPs. Consequently, staff and providers had no effective way of knowing a veteran’s gender identity from the EHR, which contributed to misgendering TGNC veterans.

With the singular demographic field of sex representing both birth sex and gender identity, some TGNC veterans chose to change their birth sex information to align with their gender identity. This change assured TGNC veterans that staff and providers would not misgender them because the birth sex field is easily observed and would allow providers to use respectful, gender-consistent pronouns when speaking with them. However, changing the birth sex field can misalign natal sex–based clinical reminders, medication dosages, and laboratory test values, which created potential patient safety risks. Thus, birth sex created potential hazards to quality and safety when used as a marker even with other variables—such as current anatomy, height, and weight—for health screenings, medication dosing, and other medical decisions.

In this article, we: (1) outline several patient safety issues that can arise with the birth sex field serving as an indicator for both birth sex and gender identity; (2) present case examples that illustrate the benefits of self-identified gender identity (SIGI) in an EHR; (3) describe the process of work-group development of patient-provider communication tools to improve patient safety; and (4) provide a brief overview of resources rolled out as a part of SIGI. This report serves as a guide for other federal organizations that wish to increase affirmative care and safe practices for transgender consumers. We will provide an overview of the tasks leading up to SIGI implementation, deliverables from the project, and lessons learned.

Veterans Affairs SIGI EHR Field

In 2016, the US Department of Veterans Affairs (VA) began implementing a SIGI demographic field across all EHRs, requiring administrative staff to ask enrolled and new veterans their gender identity (full implementation of SIGI has not yet occurred and will occur when a later EHR upgrade displays SIGI in the EHR). The initiation of SIGI did not change any information in the birth sex field, meaning that some veterans continue to have birth sex field information that results in problematic automatic medical reminders and dosing values. Consequently, the National Center for Patient Safety (NCPS) noted that this discrepancy may be a pertinent patient safety issue. The NCPS and Lesbian, Gay Bisexual, and Transgender (LGBT) Health national program offices worked to provide documentation to TGNC veterans to inform them of the clinical health care implications of having their birth sex demographic field reflect gender identity that is inconsistent with their natal sex (ie, original birth certificate record of sex).

Patient Safety Issues

Conversations between transgender patients and their HCPs about transition goals, necessary medical tests, and laboratory ranges based on their current anatomy and physiology can improve patient safety and satisfaction with medical care. Prior to the availability of the SIGI field, VA facilities varied in their documentation of gender identity in the patient chart. LGBT veteran care coordinators discussed diverse suggestions that ranged from informally documenting SIGI in each progress note to using flags to draw attention to use certain sections of local EHRs. These suggestions, though well intentioned, were not adequate for documenting gender identity at the national level because of regional variations in EHR customization options. Furthermore, the use of flags for drawing clinical attention to gender identity posed a potential for stigma toward patients, given that flags are typically reserved for behavioral or other risk concerns.

Several problems can emerge when HCPs are not equipped with accurate information about patient birth sex and SIGI. For instance, TGNC patients lack a way of being known from clinic to clinic by proper pronouns or self-labels. Providers may misgender veterans, which is a negative experience for TGNC veterans linked with increased barriers to care and decreased frequency of health care visits.⁴ Moreover, the quality and personalization of care across clinic locations in the facility’s system is variable without a consistent method of documenting birth sex and SIGI. For example, in clinics where the veteran is well known (eg, primary care), staff may be more affirming of the veteran’s gender identity than those in specialty care clinics that lack prior patient contact.

Furthermore, depending on hormone and surgical interventions, some health screenings may be irrelevant for TGNC patients. To determine appropriate health screens and assess potential risks associated with hormone therapy, providers must have access to current information regarding a patient’s physiologic anatomy.⁶ Health screenings and laboratory results in sophisticated EHRs (ie, EHRs that might autodetermine normative values) may populate incorrect treatment recommendations, such as sex-based medication dosages. Furthermore, laboratory test results could be incorrectly paired with a different assumed hormonal history, potentially putting the patient at risk.

Case Examples

An important element of EHRs facilitating the goal of patient-centered care is that patients have their EHR validate their sense of self, and their providers can use names and pronouns that correspond to the patient’s SIGI. Some patients have spent a great amount of effort altering their name and sex in legal records and may want their birth sex field to conform to their gender identity. To that end, patients may seek to alter their birth sex information so that it is congruent with how they see themselves to affirm their identity, despite patient safety risks. Several scenarios below demonstrate the potential costs and benefits to patients altering birth sex and SIGI in the EHR.

Case 1 Presentation

A young transman is working with his therapist on engaging in self-validating behaviors. This veteran has met with his PCP and informed the provider of his decision to alter the birth sex field in his EHR from female to male.

Ideally, the patient would begin to have regular conversations with his HCPs about his birth sex and gender identity, so that medical professionals can provide relevant screenings and affirm the patient’s gender identity while acknowledging his right to list his birth sex as he chooses. However, particular attention will need to be paid to assuring that natal sex–based health screenings (eg, pap smears, mammograms) are conducted on an appropriate schedule and that the veteran continues to discuss his current anatomy with providers.

Case 2 Presentation

A veteran has a male birth sex, identifies as a transwoman, and uses nongendered plural pronouns “they/them/theirs.” The word “they,” used as a singular pronoun may feel uncomfortable to some providers, but it validates the veteran’s sense of self and helps them feel welcome in the treatment environment. This patient communicated proactively with their HCPs about their transition goals and current hormone use.

They opted to have their birth sex field continue to indicate “male” because they, after a discussion with their PCP, are aware of the health implications of receiving an incorrect dose for their diabetes medication. They understand that having open communication and receiving input from their HCPs is part of good health care.

Case 3 Presentation

A patient with a sexual development disorder (intersex condition) identifies as a man (indicated as “male” in the SIGI field) and had his birth sex field changed to match his gender identity. He now seeks to change his birth sex field back to female, as he has complicated health considerations due to breast cancer.

The veteran thinks it is important that providers know about his intersex condition so that his breast cancer care is as seamless as possible. In particular, although this veteran is comfortable talking about his intersex condition and his identity with his PCP and oncologist, he wants to ensure that all people involved in his care (eg, pharmacists, radiologists) use the correct values in interpreting his medical data. Providers will need to use the female birth sex field for interpreting his medical data but use male pronouns when interacting with the veteran and documenting his care.

These case examples illustrate the need for HCPs to have patient-affirming education and appropriate clinical tools available when speaking to patients about birth sex, SIGI, and the implications of changing birth sex in the EHR. Moreover, these cases highlight that patient health needs may vary over time, due to factors such as perceived costs/benefits of a change in the sex field of the EHR as well as patient comfort with providers.

Current Status of SIGI and EHR

Although having separate fields for birth sex and SIGI in the EHR is ideal, the VHA does not yet have a fully functional SIGI field, and several TGNC veterans have changed their birth sex field to align with their gender identity. Roughly 9,700 patients have diagnostic codes related to transgender care in the VHA, meaning thousands of current patients would potentially benefit from SIGI implementation (John Blosnich, written communication, March 2018). A possible action that the VHA could take with the goal of enhancing patient safety would be to revert the birth sex field of patients who had previously changed the field back to the patient’s original birth sex. However, if this alteration to the EHR were done without the patient’s consent, numerous additional problems would result—including invalidating a veteran’s wishes—potentially driving patients away from receiving health care.

Moreover, in the absence of updated SIGI information (which only the veteran can provide), making a change in the EHR would perpetuate the misgendering of TGNC veterans who have already sought an administrative fix for this problem. Thus, the agency decided to engage patients in a discussion about their decision to keep the birth sex field consistent with their original birth certificate. In cases in which the field had been changed previously, the recommendation is for HCPs to gain patient consent to change the birth sex field back to what was on their original birth certificate. Thus, decisions about what should be listed in the EHR are made by the veteran using an informed decision-making model.

Patient Safety Education Workgroup

To begin the process of disentangling birth sex and SIGI fields in the EHR, 2 work groups were created: a technical work group (coding the patches for SIGI implementation) and a SIGI patient safety education work group. The patient safety education work group was committed to promoting affirmative VA policies that require validation of the gender identity of all veterans and pursuing best practices through clinical guidelines to promote effective, efficient, equitable, and safe veteran care. The patient safety education work group included representatives from all 3 branches of the VA (VHA, Veterans Benefits Administration, and National Cemetery Administration), including clinical media, patient safety, information technology, and education specialists. The group developed trainings for administrative staff about the appropriate ways to ask birth sex and SIGI questions, and how to record veteran-driven responses.

SIGI Fact Sheet

The patient safety education work group examined clinical literature and developed tools for staff and veterans to facilitate effective discussions about the importance and utility of documenting both birth sex and SIGI in the EHR. The patient safety education work group along with media and educational experts created basic key term definition documents to address the importance, purpose, and use of the SIGI field. The patient safety education work group developed 2 documents to facilitate communication between patients and providers.

A 1-page veteran-facing fact sheet was developed that described the differences between birth sex and SIGI fields and how these fields are used in the VA EHR system (Figure 1). In addition, a 1-page HCP-facing fact sheet was designed to inform HCPs that patients may have changed their birth sex in their EHR or might still wish to change their birth sex field, and to inform HCPs of the importance of patient-centered, gender-affirmative care (Figure 2). An additional goal of both documents was to educate veterans and HCPs on how the EHR automatically calculates laboratory results and screening notifications based on birth sex.

Review Process

As part of reviewing and finalizing the SIGI patient fact sheet, the patient safety education work group previewed the document’s content with veterans who provided feedback on drafts to improve comprehension, patient-centered care, and clinical accuracy. For instance, several patients commented that the document should address many gender identities, including intersex identities. As noted in one of the case presentations earlier, individuals who identify as intersex may have changed their birth sex to be consistent with their gender and might benefit from being informed about the EHR’s autocalculation feature. The patient safety education work group adjusted the SIGI patient fact sheet to include individuals who identify as intersex and instructed them to have a conversation with their HCP regarding potential birth sex changes in the EHR.

Much of the veteran feedback to the patient safety education work group reflected veteran concerns, more broadly, about implementation of SIGI. Many veterans were interested in how federal policy changes might affect their benefits package or clinical care within the VA. The SIGI patient fact sheet was a tool for communicating that Department of Defense (DoD) policies, specifically, do not have a bearing on VA care for LGBT veterans. Therefore, SIGI information does not affect service connection or benefits eligibility and is not shared with the DoD. Veterans found this information helpful to see reflected in the SIGI patient fact sheet.

The patient safety education work group also shared the SIGI provider fact sheet with VHA providers before finalizing the content. PCPs gave feedback to improve the specification of patient safety concerns and appropriate readership language. The patient safety education work group adjusted the SIGI provider fact sheet to be inclusive of relevant literature and an e-consultation link for assisting HCPs who are unsure how to proceed with a patient.

Implementation

The patient safety education work group also developed several materials to provide information about the birth sex and SIGI fields in the EHR. Because the SIGI demographic field is new and collected by clerical staff, training was necessary to explain the difference between birth sex and SIGI before implementation in the EHR. The training sessions educated staff about the difference between birth sex and SIGI, how to ask and respond to questions respectfully, and how to update these fields in the EHR. These trainings included a 20-minute video demonstrating best practices for asking about SIGI, a frequently asked questions document responding to 7 common questions about the new fields, and a quick reference guide for administrative staff to have handy at their desks.

Dissemination of the SIGI patient and provider fact sheets is planned to occur, ideally, several weeks before implementation of the new patches updating the EHR fields in spring 2020. Building on existing resources, the patient safety education work group plans to disseminate the patient fact sheets via e-mail lists for the national mental health facility leaders as well as through e-mail lists for VA PCPs, nursing and clerical staff, privacy officers, facility LGBT veteran care coordinators, VISN leads, transgender e-consultation, the Office of Connected Care, the LGBT external homepage for the VA, and the training website for VA employees. The goal is to target potential points of contact for veterans who may have already changed their birth sex and might benefit medically from altering birth sex to be consistent with their original birth certificate.

The SIGI provider fact sheet will be disseminated using internal e-mails, announcements on routine LGBT veteran care coordinator calls, weekly Ask LGBT Health teleconferences, and announcements at LGBT health training events both internally and externally. Several dissemination tools have already ensured that VA employees are aware of the SIGI field in the EHR. Leadership throughout the VA will be encouraged to share SIGI trainings with clerical staff. Additionally, broad-based e-mails summarizing changes to the EHR will be provided concurrent to the SIGI patch implementation to VA staff as well as links to the resources and training materials.

Challenges

One difficulty in the development process for both SIGI fact sheets was addressing the issue of patient safety for veterans who may be at different points in their gender transition process. It was challenging for the patient safety education work group to not sound alarmist in discussing the safety implications of birth sex changes in the EHR, as this is just one factor in clinical decision making. The goal was to educate veterans from a patient safety perspective about the implications of having a state-of-the-art, automated EHR. However, text can be perceived differently by different people, which is why the patient safety education work group asked veterans to preview the patient document and clinical providers to preview the provider document.

Both work groups encountered technologic challenges, including a delay in the implementation of the SIGI field due to a systemwide delay of EHR updates. Although it released training and educational materials to the VHA, the patient safety education work group understood that at some point in the future, VA programmers will update the EHR to change the information clerks and HCPs can see in the EHR. Coordination of the fact sheet release alongside information technology has been an important part of the SIGI rollout process.

Conclusion

HCPs have a complex role in providing treatment to TGNC patients in the VHA: They must affirm a patient’s gender identity through how they address them, while openly communicating the health risks inherent in having their birth sex field be incongruent with the sex recorded on their original birth certificate. Accomplishing these tasks simultaneously is difficult without invalidating the veteran’s identity or right to choose their EHR demographic birth sex label. Furthermore, patients may ask HCPs to write letters of support for either medical or surgical intervention or other documentation changes (eg, changes to a patient’s legal name, passport changes, or a safe passage letter for TGNC patients). Navigating the dialectic of safety and validation requires strong rapport, trust, and effective communication in the patient-provider relationship and great empathy by the provider.

A future task for the SIGI patient safety education work group is to continue to communicate with the technical work group and providers in the field about how demographic fields in the EHR are utilized to enable future EHR changes. This hurdle is not easy because EHR updates change the infrastructure through which demographic content is delivered and incorporated into a patient’s treatment. The VA HCPs are tasked with thoroughly examining the results that automated systems produce to ensure safe and accurate medical services are always provided to all patients. An integral part of patient-centered care is balancing any computer-guided recommendations with an understanding that actual patient needs may differ due to presence/absence of anatomy and other factors (eg, weight, current medications).

From a systems perspective, a benefit of adding the SIGI demographic field is systemic improvement in calculating the number of transgender veterans under VA care and evaluating health outcomes for this population. SIGI is particularly important for signaling gender pronouns for veterans, regardless of whether they are receiving care for a gender-related diagnosis. In terms of scope, the SIGI project potentially will apply to > 9 million enrolled veterans and nearly 400,000 VA employees.

Improvements could be made in the SIGI field of the new EHR, such as expanding the options for self-labels. Additionally, a text field could be used to enhance the quality of personalization provided to veterans self-identifying in the EHR, including pronoun specification. Moreover, adding new fields such as “preferred name” could improve the health care experience of not only TGNC veterans but all veterans who use something other than their full legal name (eg, a nickname). It will be good practice to notify providers and staff of a veteran’s requested name and pronouns when the patient checks in at an electronic kiosk so that all staff immediately know how to address the patient. The VHA can continue to adjust the options for the SIGI field once the new EHR system is operational. Ideally, this new EHR will display birth sex and SIGI to clinicians or clerks engaged in patient interactions.

Technology will continue to automate medical care, meaning that HCPs must be vigilant about how computer programming and the accuracy of prepopulated information affect patient care. The concerns discussed in this report relating to patient safety are relatively absent in the medical literature, even though substantial health risks exist to patients who have birth sex listed incorrectly for any reason.^6,7 Additionally, administrative burden can be reduced if patients who do not need certain screenings based on their current anatomy are not contacted for unnecessary screenings. Future EHR systems might incorporate anatomical considerations from an inventory to assist in automating patient care in safe and accessible ways.

Providing consistent and high-quality services to gender diverse patients is a top priority for health care systems, including the Veterans Health Administration (VHA).¹ Over the past decade, awareness of transgender and gender nonconforming (TGNC) people in the US has increased. Gender identity refers to a person’s inner sense of where that person belongs on a continuum of masculine to androgynous to feminine traits. This identity range can additionally include nonbinary identifications, such as “gender fluid” or “genderqueer.” A goal of patient-centered care is for health care providers (HCPs) to refer to TGNC individuals, like their cisgender counterparts, according to their gender identity. Gender identity for TGNC individuals may be different from their birth sex. Birth sex, commonly referred to as “sex assigned at birth,” is the biologic and physiologic characteristics that are reflected on a person’s original birth certificate and described as male or female.

Background

In the electronic health record (EHR), birth sex is an important, structured variable that is used to facilitate effective patient care that is efficient, equitable, and patient-centered. Birth sex in an EHR often is used to cue automatic timely generation of health screens (eg, pap smears, prostate exams) and calculation of medication dosages and laboratory test ranges by adjusting for a person’s typical hormonal history and anatomy.

Gender identity fields are independently helpful to include in the EHR, because clinicians can use this information to ensure proper pronoun use and avoid misgendering a patient. Additionally, the gender identity field informs HCPs who may conduct more frequent or different health screenings to evaluate specific health risks that are more prevalent in gender minority (ie, lesbian, gay, bisexual) patients.^2,3

EHRs rely on structured data elements to standardize data about patients for clinical care, quality improvement, data sharing, and patient safety.^4,5 However, health care organizations are grappling with how to incorporate gender identity and birth sex information into EHRs.³ A 2011 Veterans Health Administration (VHA) directive required staff and providers to address and provide care to veterans based on their gender identity. Like other health systems, VHA had 1 demographic data field in the EHR to indicate birth sex, with no field for gender identity. A HCP could enter gender identity information into a progress note, but this addition might not be noticed by other HCPs. Consequently, staff and providers had no effective way of knowing a veteran’s gender identity from the EHR, which contributed to misgendering TGNC veterans.

With the singular demographic field of sex representing both birth sex and gender identity, some TGNC veterans chose to change their birth sex information to align with their gender identity. This change assured TGNC veterans that staff and providers would not misgender them because the birth sex field is easily observed and would allow providers to use respectful, gender-consistent pronouns when speaking with them. However, changing the birth sex field can misalign natal sex–based clinical reminders, medication dosages, and laboratory test values, which created potential patient safety risks. Thus, birth sex created potential hazards to quality and safety when used as a marker even with other variables—such as current anatomy, height, and weight—for health screenings, medication dosing, and other medical decisions.

In this article, we: (1) outline several patient safety issues that can arise with the birth sex field serving as an indicator for both birth sex and gender identity; (2) present case examples that illustrate the benefits of self-identified gender identity (SIGI) in an EHR; (3) describe the process of work-group development of patient-provider communication tools to improve patient safety; and (4) provide a brief overview of resources rolled out as a part of SIGI. This report serves as a guide for other federal organizations that wish to increase affirmative care and safe practices for transgender consumers. We will provide an overview of the tasks leading up to SIGI implementation, deliverables from the project, and lessons learned.

Veterans Affairs SIGI EHR Field

In 2016, the US Department of Veterans Affairs (VA) began implementing a SIGI demographic field across all EHRs, requiring administrative staff to ask enrolled and new veterans their gender identity (full implementation of SIGI has not yet occurred and will occur when a later EHR upgrade displays SIGI in the EHR). The initiation of SIGI did not change any information in the birth sex field, meaning that some veterans continue to have birth sex field information that results in problematic automatic medical reminders and dosing values. Consequently, the National Center for Patient Safety (NCPS) noted that this discrepancy may be a pertinent patient safety issue. The NCPS and Lesbian, Gay Bisexual, and Transgender (LGBT) Health national program offices worked to provide documentation to TGNC veterans to inform them of the clinical health care implications of having their birth sex demographic field reflect gender identity that is inconsistent with their natal sex (ie, original birth certificate record of sex).

Patient Safety Issues

Conversations between transgender patients and their HCPs about transition goals, necessary medical tests, and laboratory ranges based on their current anatomy and physiology can improve patient safety and satisfaction with medical care. Prior to the availability of the SIGI field, VA facilities varied in their documentation of gender identity in the patient chart. LGBT veteran care coordinators discussed diverse suggestions that ranged from informally documenting SIGI in each progress note to using flags to draw attention to use certain sections of local EHRs. These suggestions, though well intentioned, were not adequate for documenting gender identity at the national level because of regional variations in EHR customization options. Furthermore, the use of flags for drawing clinical attention to gender identity posed a potential for stigma toward patients, given that flags are typically reserved for behavioral or other risk concerns.

Several problems can emerge when HCPs are not equipped with accurate information about patient birth sex and SIGI. For instance, TGNC patients lack a way of being known from clinic to clinic by proper pronouns or self-labels. Providers may misgender veterans, which is a negative experience for TGNC veterans linked with increased barriers to care and decreased frequency of health care visits.⁴ Moreover, the quality and personalization of care across clinic locations in the facility’s system is variable without a consistent method of documenting birth sex and SIGI. For example, in clinics where the veteran is well known (eg, primary care), staff may be more affirming of the veteran’s gender identity than those in specialty care clinics that lack prior patient contact.

Furthermore, depending on hormone and surgical interventions, some health screenings may be irrelevant for TGNC patients. To determine appropriate health screens and assess potential risks associated with hormone therapy, providers must have access to current information regarding a patient’s physiologic anatomy.⁶ Health screenings and laboratory results in sophisticated EHRs (ie, EHRs that might autodetermine normative values) may populate incorrect treatment recommendations, such as sex-based medication dosages. Furthermore, laboratory test results could be incorrectly paired with a different assumed hormonal history, potentially putting the patient at risk.

Case Examples

An important element of EHRs facilitating the goal of patient-centered care is that patients have their EHR validate their sense of self, and their providers can use names and pronouns that correspond to the patient’s SIGI. Some patients have spent a great amount of effort altering their name and sex in legal records and may want their birth sex field to conform to their gender identity. To that end, patients may seek to alter their birth sex information so that it is congruent with how they see themselves to affirm their identity, despite patient safety risks. Several scenarios below demonstrate the potential costs and benefits to patients altering birth sex and SIGI in the EHR.

Case 1 Presentation

A young transman is working with his therapist on engaging in self-validating behaviors. This veteran has met with his PCP and informed the provider of his decision to alter the birth sex field in his EHR from female to male.

Ideally, the patient would begin to have regular conversations with his HCPs about his birth sex and gender identity, so that medical professionals can provide relevant screenings and affirm the patient’s gender identity while acknowledging his right to list his birth sex as he chooses. However, particular attention will need to be paid to assuring that natal sex–based health screenings (eg, pap smears, mammograms) are conducted on an appropriate schedule and that the veteran continues to discuss his current anatomy with providers.

Case 2 Presentation

A veteran has a male birth sex, identifies as a transwoman, and uses nongendered plural pronouns “they/them/theirs.” The word “they,” used as a singular pronoun may feel uncomfortable to some providers, but it validates the veteran’s sense of self and helps them feel welcome in the treatment environment. This patient communicated proactively with their HCPs about their transition goals and current hormone use.

They opted to have their birth sex field continue to indicate “male” because they, after a discussion with their PCP, are aware of the health implications of receiving an incorrect dose for their diabetes medication. They understand that having open communication and receiving input from their HCPs is part of good health care.

Case 3 Presentation

A patient with a sexual development disorder (intersex condition) identifies as a man (indicated as “male” in the SIGI field) and had his birth sex field changed to match his gender identity. He now seeks to change his birth sex field back to female, as he has complicated health considerations due to breast cancer.

The veteran thinks it is important that providers know about his intersex condition so that his breast cancer care is as seamless as possible. In particular, although this veteran is comfortable talking about his intersex condition and his identity with his PCP and oncologist, he wants to ensure that all people involved in his care (eg, pharmacists, radiologists) use the correct values in interpreting his medical data. Providers will need to use the female birth sex field for interpreting his medical data but use male pronouns when interacting with the veteran and documenting his care.

These case examples illustrate the need for HCPs to have patient-affirming education and appropriate clinical tools available when speaking to patients about birth sex, SIGI, and the implications of changing birth sex in the EHR. Moreover, these cases highlight that patient health needs may vary over time, due to factors such as perceived costs/benefits of a change in the sex field of the EHR as well as patient comfort with providers.

Current Status of SIGI and EHR

Although having separate fields for birth sex and SIGI in the EHR is ideal, the VHA does not yet have a fully functional SIGI field, and several TGNC veterans have changed their birth sex field to align with their gender identity. Roughly 9,700 patients have diagnostic codes related to transgender care in the VHA, meaning thousands of current patients would potentially benefit from SIGI implementation (John Blosnich, written communication, March 2018). A possible action that the VHA could take with the goal of enhancing patient safety would be to revert the birth sex field of patients who had previously changed the field back to the patient’s original birth sex. However, if this alteration to the EHR were done without the patient’s consent, numerous additional problems would result—including invalidating a veteran’s wishes—potentially driving patients away from receiving health care.

Moreover, in the absence of updated SIGI information (which only the veteran can provide), making a change in the EHR would perpetuate the misgendering of TGNC veterans who have already sought an administrative fix for this problem. Thus, the agency decided to engage patients in a discussion about their decision to keep the birth sex field consistent with their original birth certificate. In cases in which the field had been changed previously, the recommendation is for HCPs to gain patient consent to change the birth sex field back to what was on their original birth certificate. Thus, decisions about what should be listed in the EHR are made by the veteran using an informed decision-making model.

Patient Safety Education Workgroup

To begin the process of disentangling birth sex and SIGI fields in the EHR, 2 work groups were created: a technical work group (coding the patches for SIGI implementation) and a SIGI patient safety education work group. The patient safety education work group was committed to promoting affirmative VA policies that require validation of the gender identity of all veterans and pursuing best practices through clinical guidelines to promote effective, efficient, equitable, and safe veteran care. The patient safety education work group included representatives from all 3 branches of the VA (VHA, Veterans Benefits Administration, and National Cemetery Administration), including clinical media, patient safety, information technology, and education specialists. The group developed trainings for administrative staff about the appropriate ways to ask birth sex and SIGI questions, and how to record veteran-driven responses.

SIGI Fact Sheet

The patient safety education work group examined clinical literature and developed tools for staff and veterans to facilitate effective discussions about the importance and utility of documenting both birth sex and SIGI in the EHR. The patient safety education work group along with media and educational experts created basic key term definition documents to address the importance, purpose, and use of the SIGI field. The patient safety education work group developed 2 documents to facilitate communication between patients and providers.

A 1-page veteran-facing fact sheet was developed that described the differences between birth sex and SIGI fields and how these fields are used in the VA EHR system (Figure 1). In addition, a 1-page HCP-facing fact sheet was designed to inform HCPs that patients may have changed their birth sex in their EHR or might still wish to change their birth sex field, and to inform HCPs of the importance of patient-centered, gender-affirmative care (Figure 2). An additional goal of both documents was to educate veterans and HCPs on how the EHR automatically calculates laboratory results and screening notifications based on birth sex.

Review Process

As part of reviewing and finalizing the SIGI patient fact sheet, the patient safety education work group previewed the document’s content with veterans who provided feedback on drafts to improve comprehension, patient-centered care, and clinical accuracy. For instance, several patients commented that the document should address many gender identities, including intersex identities. As noted in one of the case presentations earlier, individuals who identify as intersex may have changed their birth sex to be consistent with their gender and might benefit from being informed about the EHR’s autocalculation feature. The patient safety education work group adjusted the SIGI patient fact sheet to include individuals who identify as intersex and instructed them to have a conversation with their HCP regarding potential birth sex changes in the EHR.

Much of the veteran feedback to the patient safety education work group reflected veteran concerns, more broadly, about implementation of SIGI. Many veterans were interested in how federal policy changes might affect their benefits package or clinical care within the VA. The SIGI patient fact sheet was a tool for communicating that Department of Defense (DoD) policies, specifically, do not have a bearing on VA care for LGBT veterans. Therefore, SIGI information does not affect service connection or benefits eligibility and is not shared with the DoD. Veterans found this information helpful to see reflected in the SIGI patient fact sheet.

The patient safety education work group also shared the SIGI provider fact sheet with VHA providers before finalizing the content. PCPs gave feedback to improve the specification of patient safety concerns and appropriate readership language. The patient safety education work group adjusted the SIGI provider fact sheet to be inclusive of relevant literature and an e-consultation link for assisting HCPs who are unsure how to proceed with a patient.

Implementation

The patient safety education work group also developed several materials to provide information about the birth sex and SIGI fields in the EHR. Because the SIGI demographic field is new and collected by clerical staff, training was necessary to explain the difference between birth sex and SIGI before implementation in the EHR. The training sessions educated staff about the difference between birth sex and SIGI, how to ask and respond to questions respectfully, and how to update these fields in the EHR. These trainings included a 20-minute video demonstrating best practices for asking about SIGI, a frequently asked questions document responding to 7 common questions about the new fields, and a quick reference guide for administrative staff to have handy at their desks.

Dissemination of the SIGI patient and provider fact sheets is planned to occur, ideally, several weeks before implementation of the new patches updating the EHR fields in spring 2020. Building on existing resources, the patient safety education work group plans to disseminate the patient fact sheets via e-mail lists for the national mental health facility leaders as well as through e-mail lists for VA PCPs, nursing and clerical staff, privacy officers, facility LGBT veteran care coordinators, VISN leads, transgender e-consultation, the Office of Connected Care, the LGBT external homepage for the VA, and the training website for VA employees. The goal is to target potential points of contact for veterans who may have already changed their birth sex and might benefit medically from altering birth sex to be consistent with their original birth certificate.

The SIGI provider fact sheet will be disseminated using internal e-mails, announcements on routine LGBT veteran care coordinator calls, weekly Ask LGBT Health teleconferences, and announcements at LGBT health training events both internally and externally. Several dissemination tools have already ensured that VA employees are aware of the SIGI field in the EHR. Leadership throughout the VA will be encouraged to share SIGI trainings with clerical staff. Additionally, broad-based e-mails summarizing changes to the EHR will be provided concurrent to the SIGI patch implementation to VA staff as well as links to the resources and training materials.

Challenges

One difficulty in the development process for both SIGI fact sheets was addressing the issue of patient safety for veterans who may be at different points in their gender transition process. It was challenging for the patient safety education work group to not sound alarmist in discussing the safety implications of birth sex changes in the EHR, as this is just one factor in clinical decision making. The goal was to educate veterans from a patient safety perspective about the implications of having a state-of-the-art, automated EHR. However, text can be perceived differently by different people, which is why the patient safety education work group asked veterans to preview the patient document and clinical providers to preview the provider document.

Both work groups encountered technologic challenges, including a delay in the implementation of the SIGI field due to a systemwide delay of EHR updates. Although it released training and educational materials to the VHA, the patient safety education work group understood that at some point in the future, VA programmers will update the EHR to change the information clerks and HCPs can see in the EHR. Coordination of the fact sheet release alongside information technology has been an important part of the SIGI rollout process.

Conclusion

HCPs have a complex role in providing treatment to TGNC patients in the VHA: They must affirm a patient’s gender identity through how they address them, while openly communicating the health risks inherent in having their birth sex field be incongruent with the sex recorded on their original birth certificate. Accomplishing these tasks simultaneously is difficult without invalidating the veteran’s identity or right to choose their EHR demographic birth sex label. Furthermore, patients may ask HCPs to write letters of support for either medical or surgical intervention or other documentation changes (eg, changes to a patient’s legal name, passport changes, or a safe passage letter for TGNC patients). Navigating the dialectic of safety and validation requires strong rapport, trust, and effective communication in the patient-provider relationship and great empathy by the provider.

A future task for the SIGI patient safety education work group is to continue to communicate with the technical work group and providers in the field about how demographic fields in the EHR are utilized to enable future EHR changes. This hurdle is not easy because EHR updates change the infrastructure through which demographic content is delivered and incorporated into a patient’s treatment. The VA HCPs are tasked with thoroughly examining the results that automated systems produce to ensure safe and accurate medical services are always provided to all patients. An integral part of patient-centered care is balancing any computer-guided recommendations with an understanding that actual patient needs may differ due to presence/absence of anatomy and other factors (eg, weight, current medications).

From a systems perspective, a benefit of adding the SIGI demographic field is systemic improvement in calculating the number of transgender veterans under VA care and evaluating health outcomes for this population. SIGI is particularly important for signaling gender pronouns for veterans, regardless of whether they are receiving care for a gender-related diagnosis. In terms of scope, the SIGI project potentially will apply to > 9 million enrolled veterans and nearly 400,000 VA employees.

Improvements could be made in the SIGI field of the new EHR, such as expanding the options for self-labels. Additionally, a text field could be used to enhance the quality of personalization provided to veterans self-identifying in the EHR, including pronoun specification. Moreover, adding new fields such as “preferred name” could improve the health care experience of not only TGNC veterans but all veterans who use something other than their full legal name (eg, a nickname). It will be good practice to notify providers and staff of a veteran’s requested name and pronouns when the patient checks in at an electronic kiosk so that all staff immediately know how to address the patient. The VHA can continue to adjust the options for the SIGI field once the new EHR system is operational. Ideally, this new EHR will display birth sex and SIGI to clinicians or clerks engaged in patient interactions.

Technology will continue to automate medical care, meaning that HCPs must be vigilant about how computer programming and the accuracy of prepopulated information affect patient care. The concerns discussed in this report relating to patient safety are relatively absent in the medical literature, even though substantial health risks exist to patients who have birth sex listed incorrectly for any reason.^6,7 Additionally, administrative burden can be reduced if patients who do not need certain screenings based on their current anatomy are not contacted for unnecessary screenings. Future EHR systems might incorporate anatomical considerations from an inventory to assist in automating patient care in safe and accessible ways.

Colorectal cancer (CRC) ranks among the most common causes of cancer and cancer-related death in the US. The US Multi-Society Task Force (USMSTF) on Colorectal Cancer thus strongly endorsed using several available screening options.¹ The published guidelines largely rely on age to define the target population (Table 1). For average-risk individuals, national and Veterans Health Administration (VHA) guidelines currently recommend CRC screening in individuals aged between 50 and 75 years with a life expectancy of > 5 years.¹

Although case-control studies also point to a potential benefit in persons aged > 75 years,^2,3 the USMSTF cited less convincing evidence and suggested an individualized approach that should consider relative cancer risk and comorbidity burden. Such an approach is supported by modeling studies, which suggest reduced benefit and increased risk of screening with increasing age. The reduced benefit also is significantly affected by comorbidity and relative cancer risk.⁴ The VHA has successfully implemented CRC screening, capturing the majority of eligible patients based on age criteria. A recent survey showed that more than three-quarters of veterans between age 50 and 75 years had undergone some screening test for CRC as part of routine preventive care. Colonoscopy clearly emerged as the dominant modality chosen for CRC screening and accounted for nearly 84% of these screening tests.⁵ Consistent with these data, a case-control study confirmed that the widespread implementation of colonoscopy as CRC screening method reduced cancer-related mortality in veterans for cases of left but not right-sided colon cancer.⁶

With calls to expand the age range of CRC screening beyond aged 75 years, we decided to assess survival rates of a cohort of veterans who underwent a screening or surveillance colonoscopy between 2008 and 2014.⁷ The goals were to characterize the portion of the cohort that had died, the time between a screening colonoscopy and death, the portion of deaths that were aged ≥ 80 years, and the causes of the deaths. In addition, we focused on a subgroup of the cohort, defined by death within 2 years after the index colonoscopy, to identify predictors of early death that were independent of age.

Methods

We queried the endoscopy reporting system (EndoWorks; Olympus America, Center Valley, PA) for all colonoscopies performed by 2 of 14 physicians at the George Wahlen VA Medical Center (GWVAMC) in Salt Lake City, Utah, who performed endoscopic procedures between January 1, 2008 and December 1, 2014. These physicians had focused their clinical practice exclusively on elective outpatient colonoscopies and accounted for 37.4% of the examinations at GWVAMC during the study period. All colonoscopy requests were triaged and assigned based on availability of open and appropriate procedure time slots without direct physician-specific referral, thus reducing the chance of skewing results. The reports were filtered through a text search to focus on examinations that listed screening or surveillance as indication. The central patient electronic health record was then reviewed to extract basic demographic data, survival status (as of August 1, 2018), and survival time in years after the index or subsequent colonoscopy. For deceased veterans, the age at the time of death, cause of death, and comorbidities were queried.

This study compared cases and control across the study. Cases were persons who clearly died early (defined as > 2 years following the index examination). They were matched with controls who lived for ≥ 5 years after their colonoscopy. These periods were selected because the USMSTF recommended that CRC screening or surveillance colonoscopy should be discontinued in persons with a life expectancy of < 5 years, and most study patients underwent their index procedure ≥ 5 years before August 2018. Cases and controls underwent a colonoscopy in the same year and were matched for age, sex, and presence of underlying inflammatory bowel disease (IBD). For cases and controls, we identified the ordering health care provider specialty, (ie, primary care, gastroenterology, or other).

In addition, we reviewed the encounter linked to the order and abstracted relevant comorbidities listed at that time, noted the use of anticoagulants, opioid analgesics, and benzodiazepines. The comorbidity burden was quantified using the Charlson Comorbidity Index.⁸ In addition, we denoted the presence of psychiatric problems (eg, anxiety, depression, bipolar disease, psychosis, substance abuse), the diagnosis of atrial fibrillation (AF) or other cardiac arrhythmias, and whether the patient had previously been treated for a malignancy that was in apparent clinical remission. Finally, we searched for routine laboratory tests at the time of this visit or, when not obtained, within 6 months of the encounter, and abstracted serum creatinine, hemoglobin (Hgb), platelet number, serum protein, and albumin. In clinical practice, cutoff values of test results are often more helpful in decision making. We, therefore, dichotomized results for Hgb (cutoff: 10 g/dL), creatinine (cutoff: 2 mg/dL), and albumin (cutoff: 3.2 mg/dL).

Descriptive and analytical statistics were obtained with Stata Version 14.1 (College Station, TX). Unless indicated otherwise, continuous data are shown as mean with 95% CIs. For dichotomous data, we used percentages with their 95% CIs. Analytic statistics were performed with the t test for continuous variables and the 2-tailed test for proportions. A P < .05 was considered a significant difference. To determine independent predictors of early death, we performed a logistic regression analysis with results being expressed as odds ratio with 95% CIs. Survival status was chosen as a dependent variable, and we entered variables that significantly correlated with survival in the bivariate analysis as independent variables.

The study was designed and conducted as a quality improvement project to assess colonoscopy performance and outcomes with the Salt Lake City Specialty Care Center of Innovation (COI), one of 5 regional COIs with an operational mission to improve health care access, utilization, and quality. Our work related to colonoscopy and access within the COI region, including Salt Lake City, has been reviewed and acknowledged by the GWVAMC Institutional Review Board as quality improvement. Andrew Gawron has an operational appointment in the GWVAMC COI, which is part of a US Department of Veterans Affairs (VA) central office initiative established in 2015. The COIs are charged with identifying best practices within the VA and applying those practices throughout the COI region. This local project to identify practice patterns and outcomes locally was sponsored by the GWVAMC COI with a focus to generate information to improve colonoscopy referral quality in patients at Salt Lake City and inform regional and national efforts in this domain.

Results

During the study period, 4,879 veterans (96.9% male) underwent at least 1 colonoscopy for screening or surveillance by 1 of the 2 providers. A total of 306 persons (6.3%) were aged > 80 years. The indication for surveillance colonoscopies included IBD in 78 (1.6%) veterans 2 of whom were women. The mean (SD) follow-up period between the index colonoscopy and study closure or death was 7.4 years (1.7). During the study time, 1,439 persons underwent a repeat examination for surveillance. The percentage of veterans with at least 1 additional colonoscopy after the index test was significantly higher in patients with known IBD compared with those without IBD (78.2% vs 28.7%; P < .01).

Between the index colonoscopy and August 2018, 974 patients (20.0%) died (Figure). The mean (SD) time between the colonoscopy and recorded year of death was 4.4 years (4.1). The fraction of women in the cohort that died (n = 18) was lower compared with 132 for the group of persons still alive (1.8% vs 3.4%; P < .05). The fraction of veterans with IBD who died by August 2018 did not differ from that of patients with IBD in the cohort of individuals who survived (19.2% vs 20.0%; P = .87). The cohort of veterans who died before study closure included 107 persons who were aged > 80 years at the time of their index colonoscopy, which is significantly more than in the cohort of persons still alive (11.0% vs 5.1%; P < .01).

Cause of Death

In 209 of the 974 (21.5%) veteran deaths a cause was recorded. Malignancies accounted for 88 of the deaths (42.1%), and CRCs were responsible for 14 (6.7%) deaths (Table 2). In 8 of these patients, the cancer had been identified at an advanced stage, not allowing for curative therapy. One patient had been asked to return for a repeat test as residual fecal matter did not allow proper visualization. He died 1 year later due to complications of sepsis after colonic perforation caused by a proximal colon cancer. Five patients underwent surgery with curative intent but suffered recurrences. In addition to malignancies, advanced diseases, such as cardiovascular, bronchopulmonary illnesses, and infections, were other commonly listed causes of death.

We also abstracted comorbidities that were known at the time of death or the most recent encounter within the VHA system. Hypertension was most commonly listed (549) followed by a current or prior diagnosis of malignancies (355) and diabetes mellitus (DM) (Table 3). Prostate cancer was the most commonly diagnosed malignancy (80), 17 of whom had a second malignancy. CRC accounted for 54 of the malignancies, 1 of which developed in a patient with long-standing ulcerative colitis, 2 were a manifestation of a known hereditary cancer syndrome (Lynch syndrome), and the remaining 51 cases were various cancers without known predisposition. The diagnosis of CRC was made during the study period in 29 veterans. In the remaining 25 patients, the colonoscopy was performed as a surveillance examination after previous surgery for CRC.

Potential Predictors of Early Death

To better define potential predictors of early death, we focused on the 258 persons (5.3%) who died within 2 years after the index procedure and paired them with matched controls. One patient underwent a colonoscopy for surveillance of previously treated cancer and was excluded due to very advanced age, as no matched control could be identified. The mean (SD) age of this male-predominant cohort was 68.2 (9.6) vs 67.9 (9.4) years for cases and controls, respectively. At the time of referral for the test, 29 persons (11.3%) were aged > 80 years, which is significantly more than seen for the overall cohort with 306 (6.3%; P < .001). While primary care providers accounted for most referrals in cases (85.2%) and controls (93.0%), the fraction of veterans referred by gastroenterologists or other specialty care providers was significantly higher in the case group compared with that in the controls (14.8% vs 7.0%; P < .05).

In our age-matched analysis, we examined other potential factors that could influence survival. The burden of comorbid conditions summarized in the Charlson Comorbidity Index significantly correlated with survival status (Table 4). As this composite index does not include psychiatric conditions, we separately examined the impact of anxiety, depression, bipolar disease, psychotic disorders, and substance abuse. The diagnoses of depression and substance use disorders (SUDs) were associated with higher rates of early death. Considering concerns about SUDs, we also assessed the association between prescription for opioids or benzodiazepines and survival status, which showed a marginal correlation. Anticoagulant use, a likely surrogate for cardiovascular disorders, were more commonly listed in the cases than they were in the controls.

Looking at specific comorbid conditions, significant problems affecting key organ systems from heart to lung, liver, kidneys, or brain (dementia) were all predictors of poor outcome. Similarly, DM with secondary complication correlated with early death after the index procedure. In contrast, a history of prior myocardial infarction, prior cancer treatment without evidence of persistent or recurrent disease, or prior peptic ulcer disease did not differ between cases and controls. Focusing on routine blood tests, we noted marginal, but statistically different results for Hgb, serum creatinine, and albumin in cases compared with controls.

Next we performed a logistic regression to identify independent predictors of survival status. The referring provider specialty, Charlson Comorbidity Index, the diagnosis of a SUD, current benzodiazepine use, and significant anemia or hypoalbuminemia independently predicted death within 2 years of the index examination (Table 5). Considering the composite nature of the Charlson Comorbidity Index, we separately examined the relative importance of different comorbid conditions using a logistic regression analysis. Consistent with the univariate analyses, a known malignancy; severe liver, lung, or kidney disease; and DM with secondary complications were associated with poor outcome. Only arrhythmias other than AF were independent marginal predictors of early death, whereas other variables related to cardiac performance did not reach the level of significance (Table 6). As was true for our analysis examining the composite comorbidity index, the diagnosis of a SUD remained significant as a predictor of death within 2 years of the index colonoscopy.

Discussion

This retrospective analysis followed patients for a mean time of 7 years after a colonoscopy for CRC screening or polyp surveillance. We noted a high rate of all-cause mortality, with 20% of the cohort dying within the period studied. Malignancies, cardiovascular diseases, and advanced lung diseases were most commonly listed causes of death. As expected, CRC was among the 3 most common malignancies and was the cause of death in 6.7% of the group with sufficiently detailed information. While these results fall within the expected range for the mortality related to CRC,⁹ the results do not allow us to assess the impact of screening, which has been shown to decrease cancer-related mortality in veterans.⁶ This was limited because the sample size was too small to assess the impact of screening and the cause of death was ascertained for a small percentage of the sample.

Although our findings are limited to a subset of patients seen in a single center, they suggest the importance of appropriate eligibility criteria for screening tests, as also defined in national guidelines.¹ As a key anchoring point that describes the target population, age contributed to the rate of relatively early death after the index procedure. Consistent with previously published data, we saw a significant impact of comorbid diseases.^10,11 However, our findings go beyond prior reports and show the important impact of psychiatric disease burden, most important the role of SUDs. The predictive value of a summary score, such as the Charlson Comorbidity Index, supports the idea of a cumulative impact, with an increasing disease burden decreasing life expectancy.^10-14 It is important to consider the ongoing impact of such coexisting illnesses. Our analysis shows, the mere history of prior problems did not independently predict survival status in our cohort.

Although age is the key anchoring point that defines the target population for CRC screening programs, the benefit of earlier cancer detection or, in the context of colonoscopy with polypectomy, cancer prevention comes with a delay. Thus, cancer risk, procedural risk, and life expectancy should all be weighed when discussing and deciding on the appropriateness of CRC screening. When we disregard inherited cancer syndromes, CRC is clearly a disease of the second half of life with the incidence increasing with age.¹⁵ However, other disease burdens rise, which may affects the risk of screening and treatment should cancer be found.

Using our understanding of disease development, researchers have introduced the concept of time to benefit or lag time to help decisions about screening strategies. The period defines the likely time for a precursor or early form of cancer potentially detected by screening to manifest as a clinically relevant lesion. This lag time becomes an especially important consideration in screening of older and/or chronically ill adults with life expectancies that may be close to or even less than the time to benefit.¹⁶ Modeling studies suggest that 1,000 flexible sigmoidoscopy screenings are needed to prevent 1 cancer that would manifest about 10 years after the index examination.^17,18 The mean life expectancy of a healthy person aged 75 years exceeds 10 years but drops with comorbidity burden. Consistent with these considerations, an analysis of Medicare claims data concluded that individuals with ≥ 3 significant comorbidities do not derive any benefit from screening colonoscopy.¹⁴ Looking at the impact of comorbidities, mathematical models concluded that colorectal cancer screening should not be continued in persons with moderate or severe comorbid conditions aged 66 years and 72 years, respectively.¹⁹ In contrast, modeling results suggest a benefit of continued screening up to and even above the age of 80 years if persons have an increased cancer risk and if there are no confounding comorbidities.⁴

Life expectancy and time to benefit describe probabilities. Although such probabilities are relevant in public policy decision, providers and patients may struggle with probabilistic thinking when faced with decisions that involve probabilities of individual health care vs population health care. Both are concerned about the seemingly gloomy or pessimistic undertone of discussing life expectancy and the inherent uncertainty of prognostic tools.^20,21 Prior research indicates that this reluctance translates into clinical practice. When faced with vignettes, most clinicians would offer CRC screening to healthy persons aged 80 years with rates falling when the description included a significant comorbid burden; however, more than 40% would still consider screening in octogenarians with poor health.²²

Consistent with these responses to theoretical scenarios, CRC screening of veterans dropped with age but was still continued in persons with significant comorbidity.²³ Large studies of the veteran population suggest that about 10% of veterans aged > 70 years have chronic medical problems that limit their life expectancy to < 5 years; nonetheless, more than 40% of this cohort underwent colonoscopies for CRC screening.^24,25 Interestingly, more illness burden and more clinical encounters translated into more screening examinations in older sick veterans compared with that of the cohort of healthier older persons, suggesting an impact of clinical reminders and the key role of age as the main anchoring variable.²³

Ongoing screening despite limited or even no benefit is not unique to CRC. Using validated tools, Pollock and colleagues showed comparable screening rates for breast and prostate cancer when they examined cohorts at either high or low risk of early mortality.²⁶ Similar results have been reported in veterans with about one-third of elderly males with poor life expectancy still undergoing prostate cancer screening.²⁷ Interestingly, inappropriate screening is more common in nonacademic centers and influenced by provider characteristics: nurse practitioner, physician assistants, older attending physicians and male physicians were more likely to order such tests.^27,28

Limitations

In this study, we examined a cohort of veterans enrolled in CRC screening within a single institution and obtained survival data for a mean follow-up of > 7 years. We also restricted our study to patients undergoing examinations that explicitly listed screening as indication or polyp surveillance for the test. However, inclusion was based on the indication listed in the report, which may differ from the intent of the ordering provider. Reporting systems often come with default settings, which may skew data. Comorbidities for the entire cohort of veterans who died within the time frame of the study were extracted from the chart without controlling for time-dependent changes, which may more appropriately describe the comorbidity burden at the time of the test. Using a case-control design, we addressed this potential caveat and included only illnesses recorded in the encounter linked to the colonoscopy order. Despite these limitations, our results highlight the importance to more effectively define and target appropriate candidates for CRC screening.

Conclusion

This study shows that age is a simple but not sufficiently accurate criterion to define potential candidates for CRC screening. As automated reminders often prompt discussions about and referral to screening examinations, we should develop algorithms that estimate the individual cancer risk and/or integrate an automatically calculated comorbidity index with these alerts or insert such a tool into order-sets. In addition, providers and patients need to be educated about the rationale and need for a more comprehensive approach to CRC screening that considers anticipated life expectancy. On an individual and health system level, our goal should be to reduce overall mortality rather than only cancer-specific death rates.

Colorectal cancer (CRC) ranks among the most common causes of cancer and cancer-related death in the US. The US Multi-Society Task Force (USMSTF) on Colorectal Cancer thus strongly endorsed using several available screening options.¹ The published guidelines largely rely on age to define the target population (Table 1). For average-risk individuals, national and Veterans Health Administration (VHA) guidelines currently recommend CRC screening in individuals aged between 50 and 75 years with a life expectancy of > 5 years.¹

Although case-control studies also point to a potential benefit in persons aged > 75 years,^2,3 the USMSTF cited less convincing evidence and suggested an individualized approach that should consider relative cancer risk and comorbidity burden. Such an approach is supported by modeling studies, which suggest reduced benefit and increased risk of screening with increasing age. The reduced benefit also is significantly affected by comorbidity and relative cancer risk.⁴ The VHA has successfully implemented CRC screening, capturing the majority of eligible patients based on age criteria. A recent survey showed that more than three-quarters of veterans between age 50 and 75 years had undergone some screening test for CRC as part of routine preventive care. Colonoscopy clearly emerged as the dominant modality chosen for CRC screening and accounted for nearly 84% of these screening tests.⁵ Consistent with these data, a case-control study confirmed that the widespread implementation of colonoscopy as CRC screening method reduced cancer-related mortality in veterans for cases of left but not right-sided colon cancer.⁶

With calls to expand the age range of CRC screening beyond aged 75 years, we decided to assess survival rates of a cohort of veterans who underwent a screening or surveillance colonoscopy between 2008 and 2014.⁷ The goals were to characterize the portion of the cohort that had died, the time between a screening colonoscopy and death, the portion of deaths that were aged ≥ 80 years, and the causes of the deaths. In addition, we focused on a subgroup of the cohort, defined by death within 2 years after the index colonoscopy, to identify predictors of early death that were independent of age.

Methods

We queried the endoscopy reporting system (EndoWorks; Olympus America, Center Valley, PA) for all colonoscopies performed by 2 of 14 physicians at the George Wahlen VA Medical Center (GWVAMC) in Salt Lake City, Utah, who performed endoscopic procedures between January 1, 2008 and December 1, 2014. These physicians had focused their clinical practice exclusively on elective outpatient colonoscopies and accounted for 37.4% of the examinations at GWVAMC during the study period. All colonoscopy requests were triaged and assigned based on availability of open and appropriate procedure time slots without direct physician-specific referral, thus reducing the chance of skewing results. The reports were filtered through a text search to focus on examinations that listed screening or surveillance as indication. The central patient electronic health record was then reviewed to extract basic demographic data, survival status (as of August 1, 2018), and survival time in years after the index or subsequent colonoscopy. For deceased veterans, the age at the time of death, cause of death, and comorbidities were queried.

This study compared cases and control across the study. Cases were persons who clearly died early (defined as > 2 years following the index examination). They were matched with controls who lived for ≥ 5 years after their colonoscopy. These periods were selected because the USMSTF recommended that CRC screening or surveillance colonoscopy should be discontinued in persons with a life expectancy of < 5 years, and most study patients underwent their index procedure ≥ 5 years before August 2018. Cases and controls underwent a colonoscopy in the same year and were matched for age, sex, and presence of underlying inflammatory bowel disease (IBD). For cases and controls, we identified the ordering health care provider specialty, (ie, primary care, gastroenterology, or other).

In addition, we reviewed the encounter linked to the order and abstracted relevant comorbidities listed at that time, noted the use of anticoagulants, opioid analgesics, and benzodiazepines. The comorbidity burden was quantified using the Charlson Comorbidity Index.⁸ In addition, we denoted the presence of psychiatric problems (eg, anxiety, depression, bipolar disease, psychosis, substance abuse), the diagnosis of atrial fibrillation (AF) or other cardiac arrhythmias, and whether the patient had previously been treated for a malignancy that was in apparent clinical remission. Finally, we searched for routine laboratory tests at the time of this visit or, when not obtained, within 6 months of the encounter, and abstracted serum creatinine, hemoglobin (Hgb), platelet number, serum protein, and albumin. In clinical practice, cutoff values of test results are often more helpful in decision making. We, therefore, dichotomized results for Hgb (cutoff: 10 g/dL), creatinine (cutoff: 2 mg/dL), and albumin (cutoff: 3.2 mg/dL).

Descriptive and analytical statistics were obtained with Stata Version 14.1 (College Station, TX). Unless indicated otherwise, continuous data are shown as mean with 95% CIs. For dichotomous data, we used percentages with their 95% CIs. Analytic statistics were performed with the t test for continuous variables and the 2-tailed test for proportions. A P < .05 was considered a significant difference. To determine independent predictors of early death, we performed a logistic regression analysis with results being expressed as odds ratio with 95% CIs. Survival status was chosen as a dependent variable, and we entered variables that significantly correlated with survival in the bivariate analysis as independent variables.

The study was designed and conducted as a quality improvement project to assess colonoscopy performance and outcomes with the Salt Lake City Specialty Care Center of Innovation (COI), one of 5 regional COIs with an operational mission to improve health care access, utilization, and quality. Our work related to colonoscopy and access within the COI region, including Salt Lake City, has been reviewed and acknowledged by the GWVAMC Institutional Review Board as quality improvement. Andrew Gawron has an operational appointment in the GWVAMC COI, which is part of a US Department of Veterans Affairs (VA) central office initiative established in 2015. The COIs are charged with identifying best practices within the VA and applying those practices throughout the COI region. This local project to identify practice patterns and outcomes locally was sponsored by the GWVAMC COI with a focus to generate information to improve colonoscopy referral quality in patients at Salt Lake City and inform regional and national efforts in this domain.

Results

During the study period, 4,879 veterans (96.9% male) underwent at least 1 colonoscopy for screening or surveillance by 1 of the 2 providers. A total of 306 persons (6.3%) were aged > 80 years. The indication for surveillance colonoscopies included IBD in 78 (1.6%) veterans 2 of whom were women. The mean (SD) follow-up period between the index colonoscopy and study closure or death was 7.4 years (1.7). During the study time, 1,439 persons underwent a repeat examination for surveillance. The percentage of veterans with at least 1 additional colonoscopy after the index test was significantly higher in patients with known IBD compared with those without IBD (78.2% vs 28.7%; P < .01).

Between the index colonoscopy and August 2018, 974 patients (20.0%) died (Figure). The mean (SD) time between the colonoscopy and recorded year of death was 4.4 years (4.1). The fraction of women in the cohort that died (n = 18) was lower compared with 132 for the group of persons still alive (1.8% vs 3.4%; P < .05). The fraction of veterans with IBD who died by August 2018 did not differ from that of patients with IBD in the cohort of individuals who survived (19.2% vs 20.0%; P = .87). The cohort of veterans who died before study closure included 107 persons who were aged > 80 years at the time of their index colonoscopy, which is significantly more than in the cohort of persons still alive (11.0% vs 5.1%; P < .01).

Cause of Death

In 209 of the 974 (21.5%) veteran deaths a cause was recorded. Malignancies accounted for 88 of the deaths (42.1%), and CRCs were responsible for 14 (6.7%) deaths (Table 2). In 8 of these patients, the cancer had been identified at an advanced stage, not allowing for curative therapy. One patient had been asked to return for a repeat test as residual fecal matter did not allow proper visualization. He died 1 year later due to complications of sepsis after colonic perforation caused by a proximal colon cancer. Five patients underwent surgery with curative intent but suffered recurrences. In addition to malignancies, advanced diseases, such as cardiovascular, bronchopulmonary illnesses, and infections, were other commonly listed causes of death.

We also abstracted comorbidities that were known at the time of death or the most recent encounter within the VHA system. Hypertension was most commonly listed (549) followed by a current or prior diagnosis of malignancies (355) and diabetes mellitus (DM) (Table 3). Prostate cancer was the most commonly diagnosed malignancy (80), 17 of whom had a second malignancy. CRC accounted for 54 of the malignancies, 1 of which developed in a patient with long-standing ulcerative colitis, 2 were a manifestation of a known hereditary cancer syndrome (Lynch syndrome), and the remaining 51 cases were various cancers without known predisposition. The diagnosis of CRC was made during the study period in 29 veterans. In the remaining 25 patients, the colonoscopy was performed as a surveillance examination after previous surgery for CRC.

Potential Predictors of Early Death

To better define potential predictors of early death, we focused on the 258 persons (5.3%) who died within 2 years after the index procedure and paired them with matched controls. One patient underwent a colonoscopy for surveillance of previously treated cancer and was excluded due to very advanced age, as no matched control could be identified. The mean (SD) age of this male-predominant cohort was 68.2 (9.6) vs 67.9 (9.4) years for cases and controls, respectively. At the time of referral for the test, 29 persons (11.3%) were aged > 80 years, which is significantly more than seen for the overall cohort with 306 (6.3%; P < .001). While primary care providers accounted for most referrals in cases (85.2%) and controls (93.0%), the fraction of veterans referred by gastroenterologists or other specialty care providers was significantly higher in the case group compared with that in the controls (14.8% vs 7.0%; P < .05).

In our age-matched analysis, we examined other potential factors that could influence survival. The burden of comorbid conditions summarized in the Charlson Comorbidity Index significantly correlated with survival status (Table 4). As this composite index does not include psychiatric conditions, we separately examined the impact of anxiety, depression, bipolar disease, psychotic disorders, and substance abuse. The diagnoses of depression and substance use disorders (SUDs) were associated with higher rates of early death. Considering concerns about SUDs, we also assessed the association between prescription for opioids or benzodiazepines and survival status, which showed a marginal correlation. Anticoagulant use, a likely surrogate for cardiovascular disorders, were more commonly listed in the cases than they were in the controls.

Looking at specific comorbid conditions, significant problems affecting key organ systems from heart to lung, liver, kidneys, or brain (dementia) were all predictors of poor outcome. Similarly, DM with secondary complication correlated with early death after the index procedure. In contrast, a history of prior myocardial infarction, prior cancer treatment without evidence of persistent or recurrent disease, or prior peptic ulcer disease did not differ between cases and controls. Focusing on routine blood tests, we noted marginal, but statistically different results for Hgb, serum creatinine, and albumin in cases compared with controls.

Next we performed a logistic regression to identify independent predictors of survival status. The referring provider specialty, Charlson Comorbidity Index, the diagnosis of a SUD, current benzodiazepine use, and significant anemia or hypoalbuminemia independently predicted death within 2 years of the index examination (Table 5). Considering the composite nature of the Charlson Comorbidity Index, we separately examined the relative importance of different comorbid conditions using a logistic regression analysis. Consistent with the univariate analyses, a known malignancy; severe liver, lung, or kidney disease; and DM with secondary complications were associated with poor outcome. Only arrhythmias other than AF were independent marginal predictors of early death, whereas other variables related to cardiac performance did not reach the level of significance (Table 6). As was true for our analysis examining the composite comorbidity index, the diagnosis of a SUD remained significant as a predictor of death within 2 years of the index colonoscopy.

Discussion

This retrospective analysis followed patients for a mean time of 7 years after a colonoscopy for CRC screening or polyp surveillance. We noted a high rate of all-cause mortality, with 20% of the cohort dying within the period studied. Malignancies, cardiovascular diseases, and advanced lung diseases were most commonly listed causes of death. As expected, CRC was among the 3 most common malignancies and was the cause of death in 6.7% of the group with sufficiently detailed information. While these results fall within the expected range for the mortality related to CRC,⁹ the results do not allow us to assess the impact of screening, which has been shown to decrease cancer-related mortality in veterans.⁶ This was limited because the sample size was too small to assess the impact of screening and the cause of death was ascertained for a small percentage of the sample.

Although our findings are limited to a subset of patients seen in a single center, they suggest the importance of appropriate eligibility criteria for screening tests, as also defined in national guidelines.¹ As a key anchoring point that describes the target population, age contributed to the rate of relatively early death after the index procedure. Consistent with previously published data, we saw a significant impact of comorbid diseases.^10,11 However, our findings go beyond prior reports and show the important impact of psychiatric disease burden, most important the role of SUDs. The predictive value of a summary score, such as the Charlson Comorbidity Index, supports the idea of a cumulative impact, with an increasing disease burden decreasing life expectancy.^10-14 It is important to consider the ongoing impact of such coexisting illnesses. Our analysis shows, the mere history of prior problems did not independently predict survival status in our cohort.

Although age is the key anchoring point that defines the target population for CRC screening programs, the benefit of earlier cancer detection or, in the context of colonoscopy with polypectomy, cancer prevention comes with a delay. Thus, cancer risk, procedural risk, and life expectancy should all be weighed when discussing and deciding on the appropriateness of CRC screening. When we disregard inherited cancer syndromes, CRC is clearly a disease of the second half of life with the incidence increasing with age.¹⁵ However, other disease burdens rise, which may affects the risk of screening and treatment should cancer be found.

Using our understanding of disease development, researchers have introduced the concept of time to benefit or lag time to help decisions about screening strategies. The period defines the likely time for a precursor or early form of cancer potentially detected by screening to manifest as a clinically relevant lesion. This lag time becomes an especially important consideration in screening of older and/or chronically ill adults with life expectancies that may be close to or even less than the time to benefit.¹⁶ Modeling studies suggest that 1,000 flexible sigmoidoscopy screenings are needed to prevent 1 cancer that would manifest about 10 years after the index examination.^17,18 The mean life expectancy of a healthy person aged 75 years exceeds 10 years but drops with comorbidity burden. Consistent with these considerations, an analysis of Medicare claims data concluded that individuals with ≥ 3 significant comorbidities do not derive any benefit from screening colonoscopy.¹⁴ Looking at the impact of comorbidities, mathematical models concluded that colorectal cancer screening should not be continued in persons with moderate or severe comorbid conditions aged 66 years and 72 years, respectively.¹⁹ In contrast, modeling results suggest a benefit of continued screening up to and even above the age of 80 years if persons have an increased cancer risk and if there are no confounding comorbidities.⁴

Life expectancy and time to benefit describe probabilities. Although such probabilities are relevant in public policy decision, providers and patients may struggle with probabilistic thinking when faced with decisions that involve probabilities of individual health care vs population health care. Both are concerned about the seemingly gloomy or pessimistic undertone of discussing life expectancy and the inherent uncertainty of prognostic tools.^20,21 Prior research indicates that this reluctance translates into clinical practice. When faced with vignettes, most clinicians would offer CRC screening to healthy persons aged 80 years with rates falling when the description included a significant comorbid burden; however, more than 40% would still consider screening in octogenarians with poor health.²²

Consistent with these responses to theoretical scenarios, CRC screening of veterans dropped with age but was still continued in persons with significant comorbidity.²³ Large studies of the veteran population suggest that about 10% of veterans aged > 70 years have chronic medical problems that limit their life expectancy to < 5 years; nonetheless, more than 40% of this cohort underwent colonoscopies for CRC screening.^24,25 Interestingly, more illness burden and more clinical encounters translated into more screening examinations in older sick veterans compared with that of the cohort of healthier older persons, suggesting an impact of clinical reminders and the key role of age as the main anchoring variable.²³

Ongoing screening despite limited or even no benefit is not unique to CRC. Using validated tools, Pollock and colleagues showed comparable screening rates for breast and prostate cancer when they examined cohorts at either high or low risk of early mortality.²⁶ Similar results have been reported in veterans with about one-third of elderly males with poor life expectancy still undergoing prostate cancer screening.²⁷ Interestingly, inappropriate screening is more common in nonacademic centers and influenced by provider characteristics: nurse practitioner, physician assistants, older attending physicians and male physicians were more likely to order such tests.^27,28

Limitations

In this study, we examined a cohort of veterans enrolled in CRC screening within a single institution and obtained survival data for a mean follow-up of > 7 years. We also restricted our study to patients undergoing examinations that explicitly listed screening as indication or polyp surveillance for the test. However, inclusion was based on the indication listed in the report, which may differ from the intent of the ordering provider. Reporting systems often come with default settings, which may skew data. Comorbidities for the entire cohort of veterans who died within the time frame of the study were extracted from the chart without controlling for time-dependent changes, which may more appropriately describe the comorbidity burden at the time of the test. Using a case-control design, we addressed this potential caveat and included only illnesses recorded in the encounter linked to the colonoscopy order. Despite these limitations, our results highlight the importance to more effectively define and target appropriate candidates for CRC screening.

Conclusion

This study shows that age is a simple but not sufficiently accurate criterion to define potential candidates for CRC screening. As automated reminders often prompt discussions about and referral to screening examinations, we should develop algorithms that estimate the individual cancer risk and/or integrate an automatically calculated comorbidity index with these alerts or insert such a tool into order-sets. In addition, providers and patients need to be educated about the rationale and need for a more comprehensive approach to CRC screening that considers anticipated life expectancy. On an individual and health system level, our goal should be to reduce overall mortality rather than only cancer-specific death rates.

Colorectal cancer (CRC) ranks among the most common causes of cancer and cancer-related death in the US. The US Multi-Society Task Force (USMSTF) on Colorectal Cancer thus strongly endorsed using several available screening options.¹ The published guidelines largely rely on age to define the target population (Table 1). For average-risk individuals, national and Veterans Health Administration (VHA) guidelines currently recommend CRC screening in individuals aged between 50 and 75 years with a life expectancy of > 5 years.¹

Although case-control studies also point to a potential benefit in persons aged > 75 years,^2,3 the USMSTF cited less convincing evidence and suggested an individualized approach that should consider relative cancer risk and comorbidity burden. Such an approach is supported by modeling studies, which suggest reduced benefit and increased risk of screening with increasing age. The reduced benefit also is significantly affected by comorbidity and relative cancer risk.⁴ The VHA has successfully implemented CRC screening, capturing the majority of eligible patients based on age criteria. A recent survey showed that more than three-quarters of veterans between age 50 and 75 years had undergone some screening test for CRC as part of routine preventive care. Colonoscopy clearly emerged as the dominant modality chosen for CRC screening and accounted for nearly 84% of these screening tests.⁵ Consistent with these data, a case-control study confirmed that the widespread implementation of colonoscopy as CRC screening method reduced cancer-related mortality in veterans for cases of left but not right-sided colon cancer.⁶

With calls to expand the age range of CRC screening beyond aged 75 years, we decided to assess survival rates of a cohort of veterans who underwent a screening or surveillance colonoscopy between 2008 and 2014.⁷ The goals were to characterize the portion of the cohort that had died, the time between a screening colonoscopy and death, the portion of deaths that were aged ≥ 80 years, and the causes of the deaths. In addition, we focused on a subgroup of the cohort, defined by death within 2 years after the index colonoscopy, to identify predictors of early death that were independent of age.

Methods

We queried the endoscopy reporting system (EndoWorks; Olympus America, Center Valley, PA) for all colonoscopies performed by 2 of 14 physicians at the George Wahlen VA Medical Center (GWVAMC) in Salt Lake City, Utah, who performed endoscopic procedures between January 1, 2008 and December 1, 2014. These physicians had focused their clinical practice exclusively on elective outpatient colonoscopies and accounted for 37.4% of the examinations at GWVAMC during the study period. All colonoscopy requests were triaged and assigned based on availability of open and appropriate procedure time slots without direct physician-specific referral, thus reducing the chance of skewing results. The reports were filtered through a text search to focus on examinations that listed screening or surveillance as indication. The central patient electronic health record was then reviewed to extract basic demographic data, survival status (as of August 1, 2018), and survival time in years after the index or subsequent colonoscopy. For deceased veterans, the age at the time of death, cause of death, and comorbidities were queried.

This study compared cases and control across the study. Cases were persons who clearly died early (defined as > 2 years following the index examination). They were matched with controls who lived for ≥ 5 years after their colonoscopy. These periods were selected because the USMSTF recommended that CRC screening or surveillance colonoscopy should be discontinued in persons with a life expectancy of < 5 years, and most study patients underwent their index procedure ≥ 5 years before August 2018. Cases and controls underwent a colonoscopy in the same year and were matched for age, sex, and presence of underlying inflammatory bowel disease (IBD). For cases and controls, we identified the ordering health care provider specialty, (ie, primary care, gastroenterology, or other).

In addition, we reviewed the encounter linked to the order and abstracted relevant comorbidities listed at that time, noted the use of anticoagulants, opioid analgesics, and benzodiazepines. The comorbidity burden was quantified using the Charlson Comorbidity Index.⁸ In addition, we denoted the presence of psychiatric problems (eg, anxiety, depression, bipolar disease, psychosis, substance abuse), the diagnosis of atrial fibrillation (AF) or other cardiac arrhythmias, and whether the patient had previously been treated for a malignancy that was in apparent clinical remission. Finally, we searched for routine laboratory tests at the time of this visit or, when not obtained, within 6 months of the encounter, and abstracted serum creatinine, hemoglobin (Hgb), platelet number, serum protein, and albumin. In clinical practice, cutoff values of test results are often more helpful in decision making. We, therefore, dichotomized results for Hgb (cutoff: 10 g/dL), creatinine (cutoff: 2 mg/dL), and albumin (cutoff: 3.2 mg/dL).

Descriptive and analytical statistics were obtained with Stata Version 14.1 (College Station, TX). Unless indicated otherwise, continuous data are shown as mean with 95% CIs. For dichotomous data, we used percentages with their 95% CIs. Analytic statistics were performed with the t test for continuous variables and the 2-tailed test for proportions. A P < .05 was considered a significant difference. To determine independent predictors of early death, we performed a logistic regression analysis with results being expressed as odds ratio with 95% CIs. Survival status was chosen as a dependent variable, and we entered variables that significantly correlated with survival in the bivariate analysis as independent variables.

The study was designed and conducted as a quality improvement project to assess colonoscopy performance and outcomes with the Salt Lake City Specialty Care Center of Innovation (COI), one of 5 regional COIs with an operational mission to improve health care access, utilization, and quality. Our work related to colonoscopy and access within the COI region, including Salt Lake City, has been reviewed and acknowledged by the GWVAMC Institutional Review Board as quality improvement. Andrew Gawron has an operational appointment in the GWVAMC COI, which is part of a US Department of Veterans Affairs (VA) central office initiative established in 2015. The COIs are charged with identifying best practices within the VA and applying those practices throughout the COI region. This local project to identify practice patterns and outcomes locally was sponsored by the GWVAMC COI with a focus to generate information to improve colonoscopy referral quality in patients at Salt Lake City and inform regional and national efforts in this domain.

Results

During the study period, 4,879 veterans (96.9% male) underwent at least 1 colonoscopy for screening or surveillance by 1 of the 2 providers. A total of 306 persons (6.3%) were aged > 80 years. The indication for surveillance colonoscopies included IBD in 78 (1.6%) veterans 2 of whom were women. The mean (SD) follow-up period between the index colonoscopy and study closure or death was 7.4 years (1.7). During the study time, 1,439 persons underwent a repeat examination for surveillance. The percentage of veterans with at least 1 additional colonoscopy after the index test was significantly higher in patients with known IBD compared with those without IBD (78.2% vs 28.7%; P < .01).

Between the index colonoscopy and August 2018, 974 patients (20.0%) died (Figure). The mean (SD) time between the colonoscopy and recorded year of death was 4.4 years (4.1). The fraction of women in the cohort that died (n = 18) was lower compared with 132 for the group of persons still alive (1.8% vs 3.4%; P < .05). The fraction of veterans with IBD who died by August 2018 did not differ from that of patients with IBD in the cohort of individuals who survived (19.2% vs 20.0%; P = .87). The cohort of veterans who died before study closure included 107 persons who were aged > 80 years at the time of their index colonoscopy, which is significantly more than in the cohort of persons still alive (11.0% vs 5.1%; P < .01).

Cause of Death

In 209 of the 974 (21.5%) veteran deaths a cause was recorded. Malignancies accounted for 88 of the deaths (42.1%), and CRCs were responsible for 14 (6.7%) deaths (Table 2). In 8 of these patients, the cancer had been identified at an advanced stage, not allowing for curative therapy. One patient had been asked to return for a repeat test as residual fecal matter did not allow proper visualization. He died 1 year later due to complications of sepsis after colonic perforation caused by a proximal colon cancer. Five patients underwent surgery with curative intent but suffered recurrences. In addition to malignancies, advanced diseases, such as cardiovascular, bronchopulmonary illnesses, and infections, were other commonly listed causes of death.

We also abstracted comorbidities that were known at the time of death or the most recent encounter within the VHA system. Hypertension was most commonly listed (549) followed by a current or prior diagnosis of malignancies (355) and diabetes mellitus (DM) (Table 3). Prostate cancer was the most commonly diagnosed malignancy (80), 17 of whom had a second malignancy. CRC accounted for 54 of the malignancies, 1 of which developed in a patient with long-standing ulcerative colitis, 2 were a manifestation of a known hereditary cancer syndrome (Lynch syndrome), and the remaining 51 cases were various cancers without known predisposition. The diagnosis of CRC was made during the study period in 29 veterans. In the remaining 25 patients, the colonoscopy was performed as a surveillance examination after previous surgery for CRC.

Potential Predictors of Early Death

To better define potential predictors of early death, we focused on the 258 persons (5.3%) who died within 2 years after the index procedure and paired them with matched controls. One patient underwent a colonoscopy for surveillance of previously treated cancer and was excluded due to very advanced age, as no matched control could be identified. The mean (SD) age of this male-predominant cohort was 68.2 (9.6) vs 67.9 (9.4) years for cases and controls, respectively. At the time of referral for the test, 29 persons (11.3%) were aged > 80 years, which is significantly more than seen for the overall cohort with 306 (6.3%; P < .001). While primary care providers accounted for most referrals in cases (85.2%) and controls (93.0%), the fraction of veterans referred by gastroenterologists or other specialty care providers was significantly higher in the case group compared with that in the controls (14.8% vs 7.0%; P < .05).

In our age-matched analysis, we examined other potential factors that could influence survival. The burden of comorbid conditions summarized in the Charlson Comorbidity Index significantly correlated with survival status (Table 4). As this composite index does not include psychiatric conditions, we separately examined the impact of anxiety, depression, bipolar disease, psychotic disorders, and substance abuse. The diagnoses of depression and substance use disorders (SUDs) were associated with higher rates of early death. Considering concerns about SUDs, we also assessed the association between prescription for opioids or benzodiazepines and survival status, which showed a marginal correlation. Anticoagulant use, a likely surrogate for cardiovascular disorders, were more commonly listed in the cases than they were in the controls.

Looking at specific comorbid conditions, significant problems affecting key organ systems from heart to lung, liver, kidneys, or brain (dementia) were all predictors of poor outcome. Similarly, DM with secondary complication correlated with early death after the index procedure. In contrast, a history of prior myocardial infarction, prior cancer treatment without evidence of persistent or recurrent disease, or prior peptic ulcer disease did not differ between cases and controls. Focusing on routine blood tests, we noted marginal, but statistically different results for Hgb, serum creatinine, and albumin in cases compared with controls.

Next we performed a logistic regression to identify independent predictors of survival status. The referring provider specialty, Charlson Comorbidity Index, the diagnosis of a SUD, current benzodiazepine use, and significant anemia or hypoalbuminemia independently predicted death within 2 years of the index examination (Table 5). Considering the composite nature of the Charlson Comorbidity Index, we separately examined the relative importance of different comorbid conditions using a logistic regression analysis. Consistent with the univariate analyses, a known malignancy; severe liver, lung, or kidney disease; and DM with secondary complications were associated with poor outcome. Only arrhythmias other than AF were independent marginal predictors of early death, whereas other variables related to cardiac performance did not reach the level of significance (Table 6). As was true for our analysis examining the composite comorbidity index, the diagnosis of a SUD remained significant as a predictor of death within 2 years of the index colonoscopy.

Discussion

This retrospective analysis followed patients for a mean time of 7 years after a colonoscopy for CRC screening or polyp surveillance. We noted a high rate of all-cause mortality, with 20% of the cohort dying within the period studied. Malignancies, cardiovascular diseases, and advanced lung diseases were most commonly listed causes of death. As expected, CRC was among the 3 most common malignancies and was the cause of death in 6.7% of the group with sufficiently detailed information. While these results fall within the expected range for the mortality related to CRC,⁹ the results do not allow us to assess the impact of screening, which has been shown to decrease cancer-related mortality in veterans.⁶ This was limited because the sample size was too small to assess the impact of screening and the cause of death was ascertained for a small percentage of the sample.

Although our findings are limited to a subset of patients seen in a single center, they suggest the importance of appropriate eligibility criteria for screening tests, as also defined in national guidelines.¹ As a key anchoring point that describes the target population, age contributed to the rate of relatively early death after the index procedure. Consistent with previously published data, we saw a significant impact of comorbid diseases.^10,11 However, our findings go beyond prior reports and show the important impact of psychiatric disease burden, most important the role of SUDs. The predictive value of a summary score, such as the Charlson Comorbidity Index, supports the idea of a cumulative impact, with an increasing disease burden decreasing life expectancy.^10-14 It is important to consider the ongoing impact of such coexisting illnesses. Our analysis shows, the mere history of prior problems did not independently predict survival status in our cohort.

Although age is the key anchoring point that defines the target population for CRC screening programs, the benefit of earlier cancer detection or, in the context of colonoscopy with polypectomy, cancer prevention comes with a delay. Thus, cancer risk, procedural risk, and life expectancy should all be weighed when discussing and deciding on the appropriateness of CRC screening. When we disregard inherited cancer syndromes, CRC is clearly a disease of the second half of life with the incidence increasing with age.¹⁵ However, other disease burdens rise, which may affects the risk of screening and treatment should cancer be found.

Using our understanding of disease development, researchers have introduced the concept of time to benefit or lag time to help decisions about screening strategies. The period defines the likely time for a precursor or early form of cancer potentially detected by screening to manifest as a clinically relevant lesion. This lag time becomes an especially important consideration in screening of older and/or chronically ill adults with life expectancies that may be close to or even less than the time to benefit.¹⁶ Modeling studies suggest that 1,000 flexible sigmoidoscopy screenings are needed to prevent 1 cancer that would manifest about 10 years after the index examination.^17,18 The mean life expectancy of a healthy person aged 75 years exceeds 10 years but drops with comorbidity burden. Consistent with these considerations, an analysis of Medicare claims data concluded that individuals with ≥ 3 significant comorbidities do not derive any benefit from screening colonoscopy.¹⁴ Looking at the impact of comorbidities, mathematical models concluded that colorectal cancer screening should not be continued in persons with moderate or severe comorbid conditions aged 66 years and 72 years, respectively.¹⁹ In contrast, modeling results suggest a benefit of continued screening up to and even above the age of 80 years if persons have an increased cancer risk and if there are no confounding comorbidities.⁴

Life expectancy and time to benefit describe probabilities. Although such probabilities are relevant in public policy decision, providers and patients may struggle with probabilistic thinking when faced with decisions that involve probabilities of individual health care vs population health care. Both are concerned about the seemingly gloomy or pessimistic undertone of discussing life expectancy and the inherent uncertainty of prognostic tools.^20,21 Prior research indicates that this reluctance translates into clinical practice. When faced with vignettes, most clinicians would offer CRC screening to healthy persons aged 80 years with rates falling when the description included a significant comorbid burden; however, more than 40% would still consider screening in octogenarians with poor health.²²

Consistent with these responses to theoretical scenarios, CRC screening of veterans dropped with age but was still continued in persons with significant comorbidity.²³ Large studies of the veteran population suggest that about 10% of veterans aged > 70 years have chronic medical problems that limit their life expectancy to < 5 years; nonetheless, more than 40% of this cohort underwent colonoscopies for CRC screening.^24,25 Interestingly, more illness burden and more clinical encounters translated into more screening examinations in older sick veterans compared with that of the cohort of healthier older persons, suggesting an impact of clinical reminders and the key role of age as the main anchoring variable.²³

Ongoing screening despite limited or even no benefit is not unique to CRC. Using validated tools, Pollock and colleagues showed comparable screening rates for breast and prostate cancer when they examined cohorts at either high or low risk of early mortality.²⁶ Similar results have been reported in veterans with about one-third of elderly males with poor life expectancy still undergoing prostate cancer screening.²⁷ Interestingly, inappropriate screening is more common in nonacademic centers and influenced by provider characteristics: nurse practitioner, physician assistants, older attending physicians and male physicians were more likely to order such tests.^27,28

Limitations

In this study, we examined a cohort of veterans enrolled in CRC screening within a single institution and obtained survival data for a mean follow-up of > 7 years. We also restricted our study to patients undergoing examinations that explicitly listed screening as indication or polyp surveillance for the test. However, inclusion was based on the indication listed in the report, which may differ from the intent of the ordering provider. Reporting systems often come with default settings, which may skew data. Comorbidities for the entire cohort of veterans who died within the time frame of the study were extracted from the chart without controlling for time-dependent changes, which may more appropriately describe the comorbidity burden at the time of the test. Using a case-control design, we addressed this potential caveat and included only illnesses recorded in the encounter linked to the colonoscopy order. Despite these limitations, our results highlight the importance to more effectively define and target appropriate candidates for CRC screening.

Conclusion

This study shows that age is a simple but not sufficiently accurate criterion to define potential candidates for CRC screening. As automated reminders often prompt discussions about and referral to screening examinations, we should develop algorithms that estimate the individual cancer risk and/or integrate an automatically calculated comorbidity index with these alerts or insert such a tool into order-sets. In addition, providers and patients need to be educated about the rationale and need for a more comprehensive approach to CRC screening that considers anticipated life expectancy. On an individual and health system level, our goal should be to reduce overall mortality rather than only cancer-specific death rates.

Effective communication between physicians and nurses is an essential element of any healthcare system. Numerous studies have highlighted the benefits of high quality physician–nurse (MD–RN) communication, including improved patient outcomes,¹ higher patient satisfaction,² and better nurse job satisfaction and retention rates.^3-5 Having physicians and nurses round together (bedside interdisciplinary rounding) has been shown to improve the perception of teamwork,^6,7 reduce the number of pages for the physician team,^6,8 better involve the patients in developing the plan of care,⁸ and even decrease the length and cost of stay.⁹

Being physically in the same space at the same time is the first and nonnegotiable requirement of bedside interdisciplinary rounding. However, precise and objective data regarding the extent to which physicians and nurses overlap at the patient bedside are lacking. Studies that examine the face-to-face component of MD–RN communication have generally relied on either qualitative methods, such as focus groups and surveys,^10,11 or quantitative methods that are subjective, such as validated scales.¹² In addition, the few studies that report quantitative data usually rely on manual observation methods that can be affected by various forms of observer bias.^10,13,14 There is also a paucity of data on how bedside overlap changes over the work week or as a function of room location.

Recently, real-time locator systems using radio frequency identification (RFID) have allowed measurement of staff and equipment movement in a precise and quantitative manner.^9,15 Although there have been previous studies using RFID locators to create time-motion maps of various hospital staff, no study has used RFID to measure and analyze the workflow of both physicians and nurses simultaneously.^16-18 The purpose of our investigation was to utilize our hospital-wide RFID staff locator technology to accurately and quantitatively assess physician and nurse rounding habits. Understanding the current rate of overlap is an important first step to establishing bedside interdisciplinary rounding.

Setting and Participants

The investigation was conducted at a single quaternary-care academic center. The study is exempt per our Institutional Review Board. Data were gathered from three adjacent medical-surgical acute care wards. The layout for each ward was the same: 19 single- or double-occupancy patient rooms arranged in a linear hallway, with a nursing station located at the center of the ward.

The study utilized wearable RFID tags (manufactured by Hill-Rom Holdings, Inc) that located specific staff within the hospital in real time. The RFID tags were checked at Hill-Rom graphical stations to ensure that their locations were tracked accurately. The investigators also wore them and walked around the wards in a prescripted manner to ensure validity. In addition, the locator accuracy was audited by participating attendings once per week and cross-checked with the generated data. Attending physicians on the University Hospitalist inpatient medicine teams were then given their uniquely-tagged RFIDs at the beginning of this study. Nurses already wear individual RFID tags as part of their normal standard-of-care workflow.

The attending hospitalists wore their RFID tags when they were on service for the entirety of the shift. They were encouraged to include nurses at the bedside, but this was not mandatory. The rounding team also included residents and medical students. Rounding usually begins at a prespecified time, but the route taken varies daily depending on patient location. Afternoon rounds were done as needed, depending on patient acuity. The attending physicians’ participation in this study was not disclosed to the patient. The patient care activities and daily routines of both nurses and physicians were otherwise unaltered.

Study Design and Data Collection

Data were collected on the three wards for 90 consecutive days, including nights and weekends. As physicians and nurses moved throughout the ward to conduct their usual patient care activities, the temporal-spatial data associated with their unique RFIDs were automatically collected in real time by the Hill-Rom receivers built into each patient room. Every day, a spreadsheet detailing the activity of all participating nurses and physicians for the past 24 hours was generated for the investigators.

A rounding event was defined as any episode in which a physician was in a patient room for more than 10 seconds. Incidences in which a physician entered and left a room multiple times over a short time span (with less than five minutes in between each event) were classified as a single rounding event. A physician and a nurse were defined as having overlapped if their RFID data showed that they were in the same patient room for a minimum of 10 seconds at the same time. For the purposes of this study, data generated from other RFID-wearing professionals, such as nursing assistants or unit secretaries, as well as data collected from the hallways, were excluded.

Statistical Analysis

All statistical analyses were conducted using GraphPad Prism (GraphPad Software, San Diego, California). Rounding and overlap lengths were rounded to the nearest minute (minimum one minute). Mean lengths are expressed along with the standard error. Comparisons of the average lengths of MD rounding events between wards was conducted using two-tailed Student t-test or one-way ANOVA. Comparisons of the frequency of MD–RN overlap between wards and across different days of the week were performed using a Chi-squared test. The analysis of correlation between the frequency of MD–RN overlap and distance between patient room and nursing station was conducted by calculating Pearson’s correlation. A P value of less than .05 was considered statistically significant.

Baseline Rounding Characteristics

Over the study period of 90 consecutive days, 739 MD rounding events were captured, for an average of 8.2 events per day. The mean length of all MD rounding events was 7.31 minutes (±0.27, ranging from one to 70 minutes). Of these 739 MD rounding events, we separately examined the 267 events that took place in single-bed patient rooms, to control for false-positive physician and nurse interactions (for example, if the MD and RN were caring for two separate roommates). The average rounding length of single-bed rooms was 6.93 (±0.27) minutes (Figure 1). For the three individual wards, the average rounding lengths were 6.40 ± 0.73, 7.48 ± 0.94, and 7.02 ± 0.54 minutes, respectively (no statistically significant difference).

Frequency of MD–RN Overlap

Of the 267 MD rounding events observed in single-bed rooms, a nurse was present in the room for 80 events (30.0%). The frequencies of MD–RN overlap in patient rooms were 37.0% (30/81), 28.0% (14/50), and 26.5% (36/136) for the three individual wards (P > .05), respectively.

The durations of MD–RN overlap, when these events did occur, were 3.43 ± 0.38, 3.00 ± 0.70, and 3.69 ± 0.92 minutes, respectively (P > .05). The overall mean length of MD–RN overlap for all single rooms was 3.48 ± 0.45 minutes.

Rounding Characteristics over the Course of the Week

To assess how rounding characteristics differed over the work week, we partitioned our data into the individual days of the week. The length of each MD rounding event (time spent in each patient room) did not vary significantly over the course of the week (Figure 2a). When the data for the individual days were aggregated into “weekdays” (Monday through Friday) and “weekends” (Saturday and Sunday), the mean lengths of MD rounds were 7.26 ± 0.32 minutes on weekdays and 7.47 ± 0.52 minutes on weekends (P > .05).

In addition, there was no difference in how frequently physicians and nurses overlapped at the patient bedside between weekdays and weekends. Of the 565 weekday MD rounding events, 238 had a nurse at bedside (42.1%), and of the 173 weekend MD rounding events, 73 had a nurse at bedside (42.2%; Figure 2b).

Effect of a Bedside Nurse on the Length of Rounds

Next, the data on the length of MD rounds were partitioned based on whether there was a bedside nurse present during rounds. The mean length of rounds with only MDs (without a bedside nurse) was 5.68 ± 0.24 minutes. By comparison, the mean length of rounds with both a nurse and a physician at the patient bedside was 9.56 ± 0.53 minutes (Figure 3). This difference was statistically significant (P < .001).

Association between Patient Room Location and the Likelihood of MD–RN Overlap

All three wards in this study have a linear layout, consisting of 19 patient rooms in a row (Figure 4a). The nursing station is located in a central position within each ward, across from the 10^th patient room. The frequency of MD–RN overlap was calculated for each room, and each room was ranked according to its relative distance from the nursing station. For each individual ward, there was no statistically significant trend in MD–RN overlap frequency as a function of the distance to the nursing station (data not shown). However, when the data from all three wards were aggregated, there was a statistically significant trend (P < .05) with a negative Pearson correlation (r = –0.670; Figure 4b). The slope of the best fit line was 1.94, suggesting that for each additional room farther away from the nursing station, the likelihood of interdisciplinary rounds (with both physicians and nurses together at the bedside) decreases by almost 2%.

To the best of our knowledge, this is the first time-motion study of MD–RN overlap using real-time, RFID-based location technology to capture the rounding activity of both nurses and physicians. Our primary interest was to examine the extent of MD–RN overlap at the patient bedside. This is an important metric that can pave the way for bedside interdisciplinary rounds. Although the exact nature of nurse-physician communication was not measured using the methodology in this study, understanding the length of time physicians spend in patient rooms, across different wards and throughout the work week, provides insights on the current workflow and potential areas of improvement. For example, we found that 30.0% of MD rounds overlapped with a nurse at the bedside. This baseline data highlight one potential barrier to institution-wide bedside interdisciplinary rounds. Workflow changes, such as better co-localization of patients by service lines or utilization of technologies to augment the visibility of rounding physicians, may improve this overlap frequency.

Data in the literature regarding how much interaction physicians and nurses have, especially at the bedside, are sparse and vary widely. In a recent study using medical students as observers by Stickrath et al., 807 MD rounding events led by medicine attendings were observed over 90 days. The frequency of rounding events that included “communication with nurse” was only 12%.¹⁹ Furthermore, only 64.9% of these communications were at the bedside, for an effective prevalence of bedside MD–RN communication of 7.8%. This number is low compared to our observed frequency of 30.0%. On the other extreme, a study from a hospital that intentionally institutes multidisciplinary rounding (explicitly defined as involving a physician and a nurse at a bedside) reported a frequency range of 63% to 81%.⁷ A follow-up study by the same group again demonstrated a high frequency of multidisciplinary rounds (74%) across a variety of ward and specialty types (range 35% to 97%.).¹¹ However, because of the selection bias of this particular setting, the high prevalence does not reflect a generalizable frequency of bedside MD–RN overlap at most hospitals.

The length of time spent by physicians at the patient bedside balances the competing demands of patient care and rapport-building with maintaining efficiency and progressing to other important tasks. In our study, physicians spent an average of 7.31 minutes at the bedside per patient. A previously published multiinstitutional observational study, which included our hospital, reported that the average length of rounds at bedside was 4.8 minutes.¹³ A second study reported that 8.0 minutes were spent at the bedside per patient.⁷ All three studies examined the same setting of internal medicine rounds at academic university-based hospitals, led by an attending physician with junior and senior residents present. However, the methodologies to measure the length of physician rounds were different: Priest et al. involved observers, Gonzalos et al. used E-mail-based surveys, and we utilized RFID-based locators. Additional institutional, individual, and patient-based factors also influence the length of rounds and are challenging to directly measure.

Furthermore, the discovery that the length of rounds and the frequency of MD–RN overlap did not statistically differ between weekdays and weekends (P > .05) was unexpected. Given the general trend of reduced physician staffing on weekends and the practice of cross-covering larger patient censuses, we would have expected shorter rounds and less frequent MD–RN overlap on the weekends.^7,20 The remarkable similarity between weekday and weekend metrics suggests that our workflow and rounding habits are not compromised on the weekends.

In addition, we found that MD rounds with a nurse at bedside took longer than rounds without a nurse, and that patient rooms located farther away from the central nursing station had a lower frequency of MD–RN overlap. However, we want to emphasize that these findings are merely associative, and not causal. For example, sicker patients usually take longer to round on than stable patients, and it is also the sicker patients who are more likely to have their nurses at the bedside, independent of physician rounding activity. Furthermore, even if rounding with nurses takes more time, it may ultimately result in fewer pages and overall time savings for both physicians and nurses.⁶

With regards to the association between room location and frequency of MD–RN overlap, the data can be interpreted in two ways. On the one hand, if the distance between the patient room and the nursing station does, in fact, reduce the frequency of overlap by almost 2% per room (Figure 4b), these data can be informative for future workflow development, quality improvement projects, or even hospital design. On the other hand, many wards might intentionally place more stable, less acute patients farther away from the nursing station because they do not need to be watched as closely. In that case, these data confirm their expectations and no action is needed.

There are several limitations to our study. The principal limitation, as discussed above, is that while our RFID system can generate large quantities of precise data on MD–RN overlap, we do not know the qualitative nature of the overlap. Just because a nurse and a physician are in the same room at the same time does not mean that they are communicating with each other. Second, we defined “rounding” as lasting a minimum of 10 seconds at the bedside. We believe that at least 10 seconds is needed to engage in any meaningful interaction between the physician and the patient, or the physician and the nurse. Reducing the time cutoff below 10 seconds risks capturing more “noise,” (decreasing specificity) whereas increasing the time cutoff above 10 seconds risks losing out on encounters that actually had substantial communication (decreasing sensitivity). Even if the communications can be classified as pure “social check-ins,” we believe these are important data to capture, as social check-ins are an important part of the patient’s care and experience. Third, several studies have commented on the modest accuracy of RFID technology as a locator system.^15,21 To address this, we both validated the accuracy of our RFID tags prior to the study and restricted our measurements to only inside patient rooms, which has less signal noise than hallways.

Future directions include expanding this study to include housestaff and physicians from other specialities, which may reveal different patterns and metrics of patient and nurse interactions.

RFID technology is a high-throughput method of generating precise, quantitative, and objective data on physician and nurse rounding habits. This tool can be widely applied to generate baseline rounding and overlap data for a variety of wards and settings, especially for institutions that are interested in comparing their metrics and performance to other peer wards or hospitals. Furthermore, this method can generate the necessary pre- and postintervention data for countless quality improvement endeavors, including efforts to enhance bedside interdisciplinary rounding.

Acknowledgments

The authors would like to thank the attending hospitalists who piloted wearing the RFID tags. This study would not be possible without your participation. The authors also wish to extend their appreciation to Gretchen Brown, MSN RN NEA-BC, for her support. Finally, the authors would like to thank Dr. Laurence Katznelson, Thi Dinh La, and the Resident Safety Council at Stanford, as well as the Stanford GME Office.

Disclosures

The authors have nothing to disclose.  

Effective communication between physicians and nurses is an essential element of any healthcare system. Numerous studies have highlighted the benefits of high quality physician–nurse (MD–RN) communication, including improved patient outcomes,¹ higher patient satisfaction,² and better nurse job satisfaction and retention rates.^3-5 Having physicians and nurses round together (bedside interdisciplinary rounding) has been shown to improve the perception of teamwork,^6,7 reduce the number of pages for the physician team,^6,8 better involve the patients in developing the plan of care,⁸ and even decrease the length and cost of stay.⁹

Being physically in the same space at the same time is the first and nonnegotiable requirement of bedside interdisciplinary rounding. However, precise and objective data regarding the extent to which physicians and nurses overlap at the patient bedside are lacking. Studies that examine the face-to-face component of MD–RN communication have generally relied on either qualitative methods, such as focus groups and surveys,^10,11 or quantitative methods that are subjective, such as validated scales.¹² In addition, the few studies that report quantitative data usually rely on manual observation methods that can be affected by various forms of observer bias.^10,13,14 There is also a paucity of data on how bedside overlap changes over the work week or as a function of room location.

Recently, real-time locator systems using radio frequency identification (RFID) have allowed measurement of staff and equipment movement in a precise and quantitative manner.^9,15 Although there have been previous studies using RFID locators to create time-motion maps of various hospital staff, no study has used RFID to measure and analyze the workflow of both physicians and nurses simultaneously.^16-18 The purpose of our investigation was to utilize our hospital-wide RFID staff locator technology to accurately and quantitatively assess physician and nurse rounding habits. Understanding the current rate of overlap is an important first step to establishing bedside interdisciplinary rounding.

Setting and Participants

The investigation was conducted at a single quaternary-care academic center. The study is exempt per our Institutional Review Board. Data were gathered from three adjacent medical-surgical acute care wards. The layout for each ward was the same: 19 single- or double-occupancy patient rooms arranged in a linear hallway, with a nursing station located at the center of the ward.

The study utilized wearable RFID tags (manufactured by Hill-Rom Holdings, Inc) that located specific staff within the hospital in real time. The RFID tags were checked at Hill-Rom graphical stations to ensure that their locations were tracked accurately. The investigators also wore them and walked around the wards in a prescripted manner to ensure validity. In addition, the locator accuracy was audited by participating attendings once per week and cross-checked with the generated data. Attending physicians on the University Hospitalist inpatient medicine teams were then given their uniquely-tagged RFIDs at the beginning of this study. Nurses already wear individual RFID tags as part of their normal standard-of-care workflow.

The attending hospitalists wore their RFID tags when they were on service for the entirety of the shift. They were encouraged to include nurses at the bedside, but this was not mandatory. The rounding team also included residents and medical students. Rounding usually begins at a prespecified time, but the route taken varies daily depending on patient location. Afternoon rounds were done as needed, depending on patient acuity. The attending physicians’ participation in this study was not disclosed to the patient. The patient care activities and daily routines of both nurses and physicians were otherwise unaltered.

Study Design and Data Collection

Data were collected on the three wards for 90 consecutive days, including nights and weekends. As physicians and nurses moved throughout the ward to conduct their usual patient care activities, the temporal-spatial data associated with their unique RFIDs were automatically collected in real time by the Hill-Rom receivers built into each patient room. Every day, a spreadsheet detailing the activity of all participating nurses and physicians for the past 24 hours was generated for the investigators.

A rounding event was defined as any episode in which a physician was in a patient room for more than 10 seconds. Incidences in which a physician entered and left a room multiple times over a short time span (with less than five minutes in between each event) were classified as a single rounding event. A physician and a nurse were defined as having overlapped if their RFID data showed that they were in the same patient room for a minimum of 10 seconds at the same time. For the purposes of this study, data generated from other RFID-wearing professionals, such as nursing assistants or unit secretaries, as well as data collected from the hallways, were excluded.

Statistical Analysis

All statistical analyses were conducted using GraphPad Prism (GraphPad Software, San Diego, California). Rounding and overlap lengths were rounded to the nearest minute (minimum one minute). Mean lengths are expressed along with the standard error. Comparisons of the average lengths of MD rounding events between wards was conducted using two-tailed Student t-test or one-way ANOVA. Comparisons of the frequency of MD–RN overlap between wards and across different days of the week were performed using a Chi-squared test. The analysis of correlation between the frequency of MD–RN overlap and distance between patient room and nursing station was conducted by calculating Pearson’s correlation. A P value of less than .05 was considered statistically significant.

Baseline Rounding Characteristics

Over the study period of 90 consecutive days, 739 MD rounding events were captured, for an average of 8.2 events per day. The mean length of all MD rounding events was 7.31 minutes (±0.27, ranging from one to 70 minutes). Of these 739 MD rounding events, we separately examined the 267 events that took place in single-bed patient rooms, to control for false-positive physician and nurse interactions (for example, if the MD and RN were caring for two separate roommates). The average rounding length of single-bed rooms was 6.93 (±0.27) minutes (Figure 1). For the three individual wards, the average rounding lengths were 6.40 ± 0.73, 7.48 ± 0.94, and 7.02 ± 0.54 minutes, respectively (no statistically significant difference).

Frequency of MD–RN Overlap

Of the 267 MD rounding events observed in single-bed rooms, a nurse was present in the room for 80 events (30.0%). The frequencies of MD–RN overlap in patient rooms were 37.0% (30/81), 28.0% (14/50), and 26.5% (36/136) for the three individual wards (P > .05), respectively.

The durations of MD–RN overlap, when these events did occur, were 3.43 ± 0.38, 3.00 ± 0.70, and 3.69 ± 0.92 minutes, respectively (P > .05). The overall mean length of MD–RN overlap for all single rooms was 3.48 ± 0.45 minutes.

Rounding Characteristics over the Course of the Week

To assess how rounding characteristics differed over the work week, we partitioned our data into the individual days of the week. The length of each MD rounding event (time spent in each patient room) did not vary significantly over the course of the week (Figure 2a). When the data for the individual days were aggregated into “weekdays” (Monday through Friday) and “weekends” (Saturday and Sunday), the mean lengths of MD rounds were 7.26 ± 0.32 minutes on weekdays and 7.47 ± 0.52 minutes on weekends (P > .05).

In addition, there was no difference in how frequently physicians and nurses overlapped at the patient bedside between weekdays and weekends. Of the 565 weekday MD rounding events, 238 had a nurse at bedside (42.1%), and of the 173 weekend MD rounding events, 73 had a nurse at bedside (42.2%; Figure 2b).

Effect of a Bedside Nurse on the Length of Rounds

Next, the data on the length of MD rounds were partitioned based on whether there was a bedside nurse present during rounds. The mean length of rounds with only MDs (without a bedside nurse) was 5.68 ± 0.24 minutes. By comparison, the mean length of rounds with both a nurse and a physician at the patient bedside was 9.56 ± 0.53 minutes (Figure 3). This difference was statistically significant (P < .001).

Association between Patient Room Location and the Likelihood of MD–RN Overlap

All three wards in this study have a linear layout, consisting of 19 patient rooms in a row (Figure 4a). The nursing station is located in a central position within each ward, across from the 10^th patient room. The frequency of MD–RN overlap was calculated for each room, and each room was ranked according to its relative distance from the nursing station. For each individual ward, there was no statistically significant trend in MD–RN overlap frequency as a function of the distance to the nursing station (data not shown). However, when the data from all three wards were aggregated, there was a statistically significant trend (P < .05) with a negative Pearson correlation (r = –0.670; Figure 4b). The slope of the best fit line was 1.94, suggesting that for each additional room farther away from the nursing station, the likelihood of interdisciplinary rounds (with both physicians and nurses together at the bedside) decreases by almost 2%.

To the best of our knowledge, this is the first time-motion study of MD–RN overlap using real-time, RFID-based location technology to capture the rounding activity of both nurses and physicians. Our primary interest was to examine the extent of MD–RN overlap at the patient bedside. This is an important metric that can pave the way for bedside interdisciplinary rounds. Although the exact nature of nurse-physician communication was not measured using the methodology in this study, understanding the length of time physicians spend in patient rooms, across different wards and throughout the work week, provides insights on the current workflow and potential areas of improvement. For example, we found that 30.0% of MD rounds overlapped with a nurse at the bedside. This baseline data highlight one potential barrier to institution-wide bedside interdisciplinary rounds. Workflow changes, such as better co-localization of patients by service lines or utilization of technologies to augment the visibility of rounding physicians, may improve this overlap frequency.

Data in the literature regarding how much interaction physicians and nurses have, especially at the bedside, are sparse and vary widely. In a recent study using medical students as observers by Stickrath et al., 807 MD rounding events led by medicine attendings were observed over 90 days. The frequency of rounding events that included “communication with nurse” was only 12%.¹⁹ Furthermore, only 64.9% of these communications were at the bedside, for an effective prevalence of bedside MD–RN communication of 7.8%. This number is low compared to our observed frequency of 30.0%. On the other extreme, a study from a hospital that intentionally institutes multidisciplinary rounding (explicitly defined as involving a physician and a nurse at a bedside) reported a frequency range of 63% to 81%.⁷ A follow-up study by the same group again demonstrated a high frequency of multidisciplinary rounds (74%) across a variety of ward and specialty types (range 35% to 97%.).¹¹ However, because of the selection bias of this particular setting, the high prevalence does not reflect a generalizable frequency of bedside MD–RN overlap at most hospitals.

The length of time spent by physicians at the patient bedside balances the competing demands of patient care and rapport-building with maintaining efficiency and progressing to other important tasks. In our study, physicians spent an average of 7.31 minutes at the bedside per patient. A previously published multiinstitutional observational study, which included our hospital, reported that the average length of rounds at bedside was 4.8 minutes.¹³ A second study reported that 8.0 minutes were spent at the bedside per patient.⁷ All three studies examined the same setting of internal medicine rounds at academic university-based hospitals, led by an attending physician with junior and senior residents present. However, the methodologies to measure the length of physician rounds were different: Priest et al. involved observers, Gonzalos et al. used E-mail-based surveys, and we utilized RFID-based locators. Additional institutional, individual, and patient-based factors also influence the length of rounds and are challenging to directly measure.

Furthermore, the discovery that the length of rounds and the frequency of MD–RN overlap did not statistically differ between weekdays and weekends (P > .05) was unexpected. Given the general trend of reduced physician staffing on weekends and the practice of cross-covering larger patient censuses, we would have expected shorter rounds and less frequent MD–RN overlap on the weekends.^7,20 The remarkable similarity between weekday and weekend metrics suggests that our workflow and rounding habits are not compromised on the weekends.

In addition, we found that MD rounds with a nurse at bedside took longer than rounds without a nurse, and that patient rooms located farther away from the central nursing station had a lower frequency of MD–RN overlap. However, we want to emphasize that these findings are merely associative, and not causal. For example, sicker patients usually take longer to round on than stable patients, and it is also the sicker patients who are more likely to have their nurses at the bedside, independent of physician rounding activity. Furthermore, even if rounding with nurses takes more time, it may ultimately result in fewer pages and overall time savings for both physicians and nurses.⁶

With regards to the association between room location and frequency of MD–RN overlap, the data can be interpreted in two ways. On the one hand, if the distance between the patient room and the nursing station does, in fact, reduce the frequency of overlap by almost 2% per room (Figure 4b), these data can be informative for future workflow development, quality improvement projects, or even hospital design. On the other hand, many wards might intentionally place more stable, less acute patients farther away from the nursing station because they do not need to be watched as closely. In that case, these data confirm their expectations and no action is needed.

There are several limitations to our study. The principal limitation, as discussed above, is that while our RFID system can generate large quantities of precise data on MD–RN overlap, we do not know the qualitative nature of the overlap. Just because a nurse and a physician are in the same room at the same time does not mean that they are communicating with each other. Second, we defined “rounding” as lasting a minimum of 10 seconds at the bedside. We believe that at least 10 seconds is needed to engage in any meaningful interaction between the physician and the patient, or the physician and the nurse. Reducing the time cutoff below 10 seconds risks capturing more “noise,” (decreasing specificity) whereas increasing the time cutoff above 10 seconds risks losing out on encounters that actually had substantial communication (decreasing sensitivity). Even if the communications can be classified as pure “social check-ins,” we believe these are important data to capture, as social check-ins are an important part of the patient’s care and experience. Third, several studies have commented on the modest accuracy of RFID technology as a locator system.^15,21 To address this, we both validated the accuracy of our RFID tags prior to the study and restricted our measurements to only inside patient rooms, which has less signal noise than hallways.

Future directions include expanding this study to include housestaff and physicians from other specialities, which may reveal different patterns and metrics of patient and nurse interactions.

RFID technology is a high-throughput method of generating precise, quantitative, and objective data on physician and nurse rounding habits. This tool can be widely applied to generate baseline rounding and overlap data for a variety of wards and settings, especially for institutions that are interested in comparing their metrics and performance to other peer wards or hospitals. Furthermore, this method can generate the necessary pre- and postintervention data for countless quality improvement endeavors, including efforts to enhance bedside interdisciplinary rounding.

Acknowledgments

The authors would like to thank the attending hospitalists who piloted wearing the RFID tags. This study would not be possible without your participation. The authors also wish to extend their appreciation to Gretchen Brown, MSN RN NEA-BC, for her support. Finally, the authors would like to thank Dr. Laurence Katznelson, Thi Dinh La, and the Resident Safety Council at Stanford, as well as the Stanford GME Office.

Disclosures

The authors have nothing to disclose.  

Effective communication between physicians and nurses is an essential element of any healthcare system. Numerous studies have highlighted the benefits of high quality physician–nurse (MD–RN) communication, including improved patient outcomes,¹ higher patient satisfaction,² and better nurse job satisfaction and retention rates.^3-5 Having physicians and nurses round together (bedside interdisciplinary rounding) has been shown to improve the perception of teamwork,^6,7 reduce the number of pages for the physician team,^6,8 better involve the patients in developing the plan of care,⁸ and even decrease the length and cost of stay.⁹

Being physically in the same space at the same time is the first and nonnegotiable requirement of bedside interdisciplinary rounding. However, precise and objective data regarding the extent to which physicians and nurses overlap at the patient bedside are lacking. Studies that examine the face-to-face component of MD–RN communication have generally relied on either qualitative methods, such as focus groups and surveys,^10,11 or quantitative methods that are subjective, such as validated scales.¹² In addition, the few studies that report quantitative data usually rely on manual observation methods that can be affected by various forms of observer bias.^10,13,14 There is also a paucity of data on how bedside overlap changes over the work week or as a function of room location.

Recently, real-time locator systems using radio frequency identification (RFID) have allowed measurement of staff and equipment movement in a precise and quantitative manner.^9,15 Although there have been previous studies using RFID locators to create time-motion maps of various hospital staff, no study has used RFID to measure and analyze the workflow of both physicians and nurses simultaneously.^16-18 The purpose of our investigation was to utilize our hospital-wide RFID staff locator technology to accurately and quantitatively assess physician and nurse rounding habits. Understanding the current rate of overlap is an important first step to establishing bedside interdisciplinary rounding.

Setting and Participants

The investigation was conducted at a single quaternary-care academic center. The study is exempt per our Institutional Review Board. Data were gathered from three adjacent medical-surgical acute care wards. The layout for each ward was the same: 19 single- or double-occupancy patient rooms arranged in a linear hallway, with a nursing station located at the center of the ward.

The study utilized wearable RFID tags (manufactured by Hill-Rom Holdings, Inc) that located specific staff within the hospital in real time. The RFID tags were checked at Hill-Rom graphical stations to ensure that their locations were tracked accurately. The investigators also wore them and walked around the wards in a prescripted manner to ensure validity. In addition, the locator accuracy was audited by participating attendings once per week and cross-checked with the generated data. Attending physicians on the University Hospitalist inpatient medicine teams were then given their uniquely-tagged RFIDs at the beginning of this study. Nurses already wear individual RFID tags as part of their normal standard-of-care workflow.

The attending hospitalists wore their RFID tags when they were on service for the entirety of the shift. They were encouraged to include nurses at the bedside, but this was not mandatory. The rounding team also included residents and medical students. Rounding usually begins at a prespecified time, but the route taken varies daily depending on patient location. Afternoon rounds were done as needed, depending on patient acuity. The attending physicians’ participation in this study was not disclosed to the patient. The patient care activities and daily routines of both nurses and physicians were otherwise unaltered.

Study Design and Data Collection

Data were collected on the three wards for 90 consecutive days, including nights and weekends. As physicians and nurses moved throughout the ward to conduct their usual patient care activities, the temporal-spatial data associated with their unique RFIDs were automatically collected in real time by the Hill-Rom receivers built into each patient room. Every day, a spreadsheet detailing the activity of all participating nurses and physicians for the past 24 hours was generated for the investigators.

A rounding event was defined as any episode in which a physician was in a patient room for more than 10 seconds. Incidences in which a physician entered and left a room multiple times over a short time span (with less than five minutes in between each event) were classified as a single rounding event. A physician and a nurse were defined as having overlapped if their RFID data showed that they were in the same patient room for a minimum of 10 seconds at the same time. For the purposes of this study, data generated from other RFID-wearing professionals, such as nursing assistants or unit secretaries, as well as data collected from the hallways, were excluded.

Statistical Analysis

All statistical analyses were conducted using GraphPad Prism (GraphPad Software, San Diego, California). Rounding and overlap lengths were rounded to the nearest minute (minimum one minute). Mean lengths are expressed along with the standard error. Comparisons of the average lengths of MD rounding events between wards was conducted using two-tailed Student t-test or one-way ANOVA. Comparisons of the frequency of MD–RN overlap between wards and across different days of the week were performed using a Chi-squared test. The analysis of correlation between the frequency of MD–RN overlap and distance between patient room and nursing station was conducted by calculating Pearson’s correlation. A P value of less than .05 was considered statistically significant.

Baseline Rounding Characteristics

Over the study period of 90 consecutive days, 739 MD rounding events were captured, for an average of 8.2 events per day. The mean length of all MD rounding events was 7.31 minutes (±0.27, ranging from one to 70 minutes). Of these 739 MD rounding events, we separately examined the 267 events that took place in single-bed patient rooms, to control for false-positive physician and nurse interactions (for example, if the MD and RN were caring for two separate roommates). The average rounding length of single-bed rooms was 6.93 (±0.27) minutes (Figure 1). For the three individual wards, the average rounding lengths were 6.40 ± 0.73, 7.48 ± 0.94, and 7.02 ± 0.54 minutes, respectively (no statistically significant difference).

Frequency of MD–RN Overlap

Of the 267 MD rounding events observed in single-bed rooms, a nurse was present in the room for 80 events (30.0%). The frequencies of MD–RN overlap in patient rooms were 37.0% (30/81), 28.0% (14/50), and 26.5% (36/136) for the three individual wards (P > .05), respectively.

The durations of MD–RN overlap, when these events did occur, were 3.43 ± 0.38, 3.00 ± 0.70, and 3.69 ± 0.92 minutes, respectively (P > .05). The overall mean length of MD–RN overlap for all single rooms was 3.48 ± 0.45 minutes.

Rounding Characteristics over the Course of the Week

To assess how rounding characteristics differed over the work week, we partitioned our data into the individual days of the week. The length of each MD rounding event (time spent in each patient room) did not vary significantly over the course of the week (Figure 2a). When the data for the individual days were aggregated into “weekdays” (Monday through Friday) and “weekends” (Saturday and Sunday), the mean lengths of MD rounds were 7.26 ± 0.32 minutes on weekdays and 7.47 ± 0.52 minutes on weekends (P > .05).

In addition, there was no difference in how frequently physicians and nurses overlapped at the patient bedside between weekdays and weekends. Of the 565 weekday MD rounding events, 238 had a nurse at bedside (42.1%), and of the 173 weekend MD rounding events, 73 had a nurse at bedside (42.2%; Figure 2b).

Effect of a Bedside Nurse on the Length of Rounds

Next, the data on the length of MD rounds were partitioned based on whether there was a bedside nurse present during rounds. The mean length of rounds with only MDs (without a bedside nurse) was 5.68 ± 0.24 minutes. By comparison, the mean length of rounds with both a nurse and a physician at the patient bedside was 9.56 ± 0.53 minutes (Figure 3). This difference was statistically significant (P < .001).

Association between Patient Room Location and the Likelihood of MD–RN Overlap

All three wards in this study have a linear layout, consisting of 19 patient rooms in a row (Figure 4a). The nursing station is located in a central position within each ward, across from the 10^th patient room. The frequency of MD–RN overlap was calculated for each room, and each room was ranked according to its relative distance from the nursing station. For each individual ward, there was no statistically significant trend in MD–RN overlap frequency as a function of the distance to the nursing station (data not shown). However, when the data from all three wards were aggregated, there was a statistically significant trend (P < .05) with a negative Pearson correlation (r = –0.670; Figure 4b). The slope of the best fit line was 1.94, suggesting that for each additional room farther away from the nursing station, the likelihood of interdisciplinary rounds (with both physicians and nurses together at the bedside) decreases by almost 2%.

To the best of our knowledge, this is the first time-motion study of MD–RN overlap using real-time, RFID-based location technology to capture the rounding activity of both nurses and physicians. Our primary interest was to examine the extent of MD–RN overlap at the patient bedside. This is an important metric that can pave the way for bedside interdisciplinary rounds. Although the exact nature of nurse-physician communication was not measured using the methodology in this study, understanding the length of time physicians spend in patient rooms, across different wards and throughout the work week, provides insights on the current workflow and potential areas of improvement. For example, we found that 30.0% of MD rounds overlapped with a nurse at the bedside. This baseline data highlight one potential barrier to institution-wide bedside interdisciplinary rounds. Workflow changes, such as better co-localization of patients by service lines or utilization of technologies to augment the visibility of rounding physicians, may improve this overlap frequency.

Data in the literature regarding how much interaction physicians and nurses have, especially at the bedside, are sparse and vary widely. In a recent study using medical students as observers by Stickrath et al., 807 MD rounding events led by medicine attendings were observed over 90 days. The frequency of rounding events that included “communication with nurse” was only 12%.¹⁹ Furthermore, only 64.9% of these communications were at the bedside, for an effective prevalence of bedside MD–RN communication of 7.8%. This number is low compared to our observed frequency of 30.0%. On the other extreme, a study from a hospital that intentionally institutes multidisciplinary rounding (explicitly defined as involving a physician and a nurse at a bedside) reported a frequency range of 63% to 81%.⁷ A follow-up study by the same group again demonstrated a high frequency of multidisciplinary rounds (74%) across a variety of ward and specialty types (range 35% to 97%.).¹¹ However, because of the selection bias of this particular setting, the high prevalence does not reflect a generalizable frequency of bedside MD–RN overlap at most hospitals.

The length of time spent by physicians at the patient bedside balances the competing demands of patient care and rapport-building with maintaining efficiency and progressing to other important tasks. In our study, physicians spent an average of 7.31 minutes at the bedside per patient. A previously published multiinstitutional observational study, which included our hospital, reported that the average length of rounds at bedside was 4.8 minutes.¹³ A second study reported that 8.0 minutes were spent at the bedside per patient.⁷ All three studies examined the same setting of internal medicine rounds at academic university-based hospitals, led by an attending physician with junior and senior residents present. However, the methodologies to measure the length of physician rounds were different: Priest et al. involved observers, Gonzalos et al. used E-mail-based surveys, and we utilized RFID-based locators. Additional institutional, individual, and patient-based factors also influence the length of rounds and are challenging to directly measure.

Furthermore, the discovery that the length of rounds and the frequency of MD–RN overlap did not statistically differ between weekdays and weekends (P > .05) was unexpected. Given the general trend of reduced physician staffing on weekends and the practice of cross-covering larger patient censuses, we would have expected shorter rounds and less frequent MD–RN overlap on the weekends.^7,20 The remarkable similarity between weekday and weekend metrics suggests that our workflow and rounding habits are not compromised on the weekends.

In addition, we found that MD rounds with a nurse at bedside took longer than rounds without a nurse, and that patient rooms located farther away from the central nursing station had a lower frequency of MD–RN overlap. However, we want to emphasize that these findings are merely associative, and not causal. For example, sicker patients usually take longer to round on than stable patients, and it is also the sicker patients who are more likely to have their nurses at the bedside, independent of physician rounding activity. Furthermore, even if rounding with nurses takes more time, it may ultimately result in fewer pages and overall time savings for both physicians and nurses.⁶

With regards to the association between room location and frequency of MD–RN overlap, the data can be interpreted in two ways. On the one hand, if the distance between the patient room and the nursing station does, in fact, reduce the frequency of overlap by almost 2% per room (Figure 4b), these data can be informative for future workflow development, quality improvement projects, or even hospital design. On the other hand, many wards might intentionally place more stable, less acute patients farther away from the nursing station because they do not need to be watched as closely. In that case, these data confirm their expectations and no action is needed.

There are several limitations to our study. The principal limitation, as discussed above, is that while our RFID system can generate large quantities of precise data on MD–RN overlap, we do not know the qualitative nature of the overlap. Just because a nurse and a physician are in the same room at the same time does not mean that they are communicating with each other. Second, we defined “rounding” as lasting a minimum of 10 seconds at the bedside. We believe that at least 10 seconds is needed to engage in any meaningful interaction between the physician and the patient, or the physician and the nurse. Reducing the time cutoff below 10 seconds risks capturing more “noise,” (decreasing specificity) whereas increasing the time cutoff above 10 seconds risks losing out on encounters that actually had substantial communication (decreasing sensitivity). Even if the communications can be classified as pure “social check-ins,” we believe these are important data to capture, as social check-ins are an important part of the patient’s care and experience. Third, several studies have commented on the modest accuracy of RFID technology as a locator system.^15,21 To address this, we both validated the accuracy of our RFID tags prior to the study and restricted our measurements to only inside patient rooms, which has less signal noise than hallways.

Future directions include expanding this study to include housestaff and physicians from other specialities, which may reveal different patterns and metrics of patient and nurse interactions.

RFID technology is a high-throughput method of generating precise, quantitative, and objective data on physician and nurse rounding habits. This tool can be widely applied to generate baseline rounding and overlap data for a variety of wards and settings, especially for institutions that are interested in comparing their metrics and performance to other peer wards or hospitals. Furthermore, this method can generate the necessary pre- and postintervention data for countless quality improvement endeavors, including efforts to enhance bedside interdisciplinary rounding.

Acknowledgments

The authors would like to thank the attending hospitalists who piloted wearing the RFID tags. This study would not be possible without your participation. The authors also wish to extend their appreciation to Gretchen Brown, MSN RN NEA-BC, for her support. Finally, the authors would like to thank Dr. Laurence Katznelson, Thi Dinh La, and the Resident Safety Council at Stanford, as well as the Stanford GME Office.

Disclosures

The authors have nothing to disclose.  

According to the National Institute for Occupational Safety and Health (NIOSH), stress is a major problem for more than 18 million US health care workers (HCWs).¹ Increases in technology, high patient acuity, and new demands for meeting institutional benchmarks create stressful clinical work environments. HCWs at the US Department of Veterans Affairs (VA) are perhaps at particular risk of experiencing burnout due to the unique needs of VA patients and bureaucratic demands.² Stress may lead to depression, decreased job satisfaction, and other psychological distress among HCWs.³ This, in turn, affects the delivery of care. High levels of burnout have been associated with increased medication errors, lower quality of care, and lower patient satisfaction scores.^4-10

A Cochrane Review found that mental and physical relaxation reduce stress in HCWs.¹¹ Among these, meditative interventions (eg, mindfulness, meditation, yoga) have demonstrated promise.^12-14 Results from a systematic meta-analysis of meditative interventions for HCWs indicated small-to-moderate improvements in emotional exhaustion, sense of personal accomplishment, and life satisfaction. Additional research is needed to determine effects of meditative interventions on burnout and caregiver burden.¹⁵

Unfortunately, many meditative intervention programs are lengthy and require a significant investment of time. They also require some form of sitting meditation every day, placing additional demands on busy HCWs. There remains a need for practical strategies to reduce HCW stress that are easier to master and practice.

Background

We developed, implemented, evaluated, and modified an evidence-based meditative intervention called the Mantram Repetition Program (MRP) to address workplace stress and burnout. The MRP is a mind-body, spiritually enhanced intervention that offers benefits similar to other types of meditative interventions.¹⁶ MRP is composed of 3 primary components: (1) silently repeating a self-selected, meaningful word or phrase (here called a mantram); (2) intentionally slowing down thoughts and behaviors; and (3) developing the ability to focus on a single task at a time (ie, one-pointed attention). The MRP does not require participants to set aside a specific place to practice, and mantram repetition can be initiated intermittently and privately throughout the day (eg, between tasks, while walking or waiting). Examples of 4 sessions (eg, Mantram 1, 2, 3, and 4) can be found on the PsychArmor Institute website (www.psycharmor.org; San Diego, CA).

Initially, the MRP was offered in a group format, in 6 or 8 weekly, 90-minute face-to-face sessions to both patient and nonpatient populations. Studies in veterans with chronic diseases demonstrated improvements in perceived stress, anxiety, and anger, and increased levels of spiritual well-being and quality of life (QOL).^17-19 Veterans with posttraumatic stress disorder (PTSD) reported improvements in PTSD symptoms, QOL, and spiritual well-being.^20-23 Family caregivers of veterans with dementia reported significant reductions in caregiver burden, depression, and anxiety after participating in the MRP.²⁴

Similar results have substantiated the effects of the MRP among HCWs, including reductions in perceived stress, stress of conscience (ie, the conflict that results from competing values and behaviors in the workplace), and burnout.^25-27 HCWs also reported improvements in mindfulness and spiritual well-being.²⁸ In a randomized controlled trial, South Korean nurse managers who completed the MRP demonstrated significant improvements in psychosocial and spiritual well-being and leadership practice and experienced reductions in burnout compared with that of the control group.²⁷ In a qualitative study, the most frequently reported benefits of the MRP were improvements in managing symptoms of stress, anxiety, and feeling out of control.¹⁸

HCWs reported they found it difficult to attend the 8-week MRP face-to-face group classes. Therefore, we developed a shorter online version of the MRP consisting of six 1-hour educational sessions: 4 online self-learning modules, and 2 live meeting webinars with the course facilitator.²⁸ VA employees were invited to enroll in the program from June 2013 through 2016 through group e-mails and announcements in the VA Employee Education Service newsletters. Those eligible to participate could earn up to 6 hours of continuing education.

Although the program was generally well accepted, feedback from HCWs indicated that providers still lacked enough time to participate fully. We therefore condensed the MRP into one 90-minute, videotaped webinar entitled “Mind-Body-Spiritual Strategies for a Healthy Workforce: The Mantram Repetition Program.” The webinar was delivered in real time in June 2013 and archived for viewing later. This condensed course provided an overview of the development, theory, and practice of MRP core components. Specific instructions included how to choose and use a mantram; the importance of acting slowly with intention to avoid mistakes; and ways of developing single-pointed attention. Participants were invited to complete a standard course evaluation using an online survey.

This article presents results from qualitative analyses of participant feedback for the condensed MRP in a nationwide sample of more than 1,700 HCWs within the VA. We used template summary analysis to identify themes in participants’ responses to 2 open-ended questions: “What about this learning activity was most useful to you?” and “What about this learning activity was least useful to you?” These results have implications for reducing HCW stress and developing training programs for HCWs.

Analysis

Responses to the what was most useful question were downloaded to a spreadsheet file for analyses. Investigators chose summary template analysis, a rapid qualitative analytic technique, as the best strategy for analyzing these textual data. This technique is often used in health services research when it is unrealistic to use more time-consuming qualitative methods, such as coding.²⁹

To begin, the analyst, a PhD-level anthropologist, read through the feedback to identify similar words, phrases, and/or concepts (ie, themes). Once the analyst gained a sense of general themes, she developed category labels using verbatim words and/or phrases in the feedback (similar to developing in vivo codes.³⁰ She listed these categories at the top of a summary template document, providing a definition for each to ensure analytic rigor.

Next, each category was listed down the left side of the template. Participant feedback was copied and pasted from the spreadsheet form into the appropriate category for each of 200 responses. The investigator identified subthemes within each category. After analysis was completed for the first 200 course participants, the analyst grouped similar categories together into broader domains to further organize the data. She then read through the feedback from the remaining 917 course participants to identify negative cases (ie, dissimilarities in feedback). An additional researcher familiar with the condensed MRP training then examined the categories and domains. Together, they discussed and resolved any inconsistencies in interpretation of the data.

To get a better sense of the full range of perspectives about the training, the analyst then read through the written feedback for the what was least useful question. She scanned the feedback for negative cases that contradicted template findings and noted these in a document. A more balanced evaluation of the course emerged through this secondary analysis.

Results

Online surveys were completed by 1,117 participants, of which three-quarters (841) were female. Two hundred eleven (19%) viewed the condensed MRP in real time. The remaining participants viewed an online video of the course. Anonymous course evaluations captured only gender and professional classification of participants. Participants represented a wide range of professional roles. The majority (63%) held clinical positions with direct patient care. The next largest category included administrative or health information personnel (21%). There were also students and trainees among these categories.

Qualitative Findings

Feedback about the course was organized into categories during analysis: (1) instructional format; (2) mode of delivery; (3) course content; (4) professional and personal empowerment; (5) religion and spirituality; and (6) ease of mantram practice. These categories represented 2 broad domains: feedback about the course and feedback about the intervention.

Instructional Format

HCWs often reported that the most useful aspect of the course was the instructional format. Most cited the ease with which they could understand the materials and helpfulness of the examples of mantram practice. The option to download course materials for later reference was also useful. Some HCWs indicated that the course could have been improved by incorporating an experiential component in which participants paused to practice a mantram.

Mode of Delivery

Delivery mode including the convenience of the training and the flexibility of having the course available at both work and home was mentioned in the feedback. Some HCWs reported that the most useful aspect of the training was the on-demand feature, which allowed them to stop and restart the program as needed. A few, however, referenced technical difficulties with the webinar.

Content

HCWs also indicated that general information about mantram repetition and information regarding the benefits of the intervention (eg, stress reduction) were useful. The scientific basis of mantram was described as useful by some, though others reported it as least useful. Practical guidance regarding the appropriate time and place to practice a mantram as well as concrete information regarding how to select a mantram was mentioned as the most useful by other participants.

Professional and Personal Empowerment

Professional and personal empowerment was referenced in evaluations. Professional development, such as learning a strategy for enhancing work performance, was reported as positive. HCWs also reported that learning a new strategy for self-care and coping with stress was useful. Some described having experienced a sense of validation by participating in the course that was empowering. Finally, some HCWs indicated the personal growth experienced as the most useful.

Religion and Spirituality

General statements regarding the utility of having learned a spiritually-based practice that crossed religious boundaries as well as general references to the power of prayer were listed in the feedback. Other HCWs indicated the usefulness of having learned that a mantram could be secular.

Ease of Mantram

HCWs referenced the ease with which a mantram can be learned and/or practiced. Course participants described the simplicity of mantram repetition and referenced its portability (ie, it can be practiced in many different settings). Finally, the overall flexibility of mantram practice of where and when it can be performed was also described as useful.

Discussion

Qualitative feedback from participant evaluations of a 90-minute, virtual online MRP course suggests that HCWs representing all areas of care are interested in learning practical strategies for managing workplace stress. Participants overwhelmingly perceived mantram practice as feasible to implement, with the portability of mantram repetition described as particularly useful. This aspect of mantram repetition represents a distinct advantage over meditative interventions that require a dedicated space and time in which to practice (eg, yoga postures, sitting meditation).

These preliminary findings also suggest that mantram practice is acceptable to HCWs representing a variety of roles. Participants indicated that they valued learning a meditative practice that can be interpreted as spiritual or secular, depending on the word or phrase chosen. Only 1 participant reported that the practice of mantram conflicted with his/her personal beliefs. A small minority of participants who found the discussion of spirituality disconcerting nevertheless indicated that the intervention was acceptable to them.

The finding that even a 90-minute course was challenging for some HCWs to accommodate speaks to the importance of developing short-duration stress-reduction programs. The standardized Mindfulness Based Stress Reduction (MBSR) program consists of 8 weekly 2.5-hour sessions and a full-day retreat for an overall commitment of 29 to 33 hours.³¹ Additionally, a systematic review of meditative interventions for informal and professional caregivers found that programs ranged from 4 to 8 weeks.¹⁵ These lengthier programs are likely more challenging than the condensed MRP.

These results also suggest the importance of general guidelines for meditative intervention courses for reducing HCW stress. The mode of delivery should be as flexible as possible, allowing course participants to start, stop, and restart the program as needed and to participate from a location most convenient to them. Although presenting evidence for clinical effectiveness is critical for establishing credibility, statistical data should be briefly summarized. An experiential component in which participants are encouraged to practice the intervention will enhance learning and ensure the translation of knowledge into practice. Finally, framing meditative practices as compatible with many different faiths and/or secular will enhance their acceptability.

Three recommended components of an overall strategy for reducing occupational burnout in health care settings include modifying the organizational structure and work processes, improving the fit between the organization and HCWs, and promoting and allowing time for individuals to learn strategies for coping with work-related stress.³² This 90-minute online MRP course represents an aspect of an overall strategy to reduce HCW stress and burnout. Providing opportunities for HCWs to learn strategies for managing stress could enhance the quality of care and improve patient outcomes. Future pragmatic trials could determine whether mantram practice impacts clinical care at the VA and elsewhere.

Limitations

All participants were self-selected; therefore, the findings may be biased favorably toward the intervention. These qualitative analyses are not generalizable. HCWs in other, non-VA settings might have different needs and/or stressors that should be considered in future program development. If this intervention is offered to a wider audience, then other formats ought to be offered, such as print, at-home recordings, live meeting, and face-to-face.

Conclusion

Course participants reported that the condensed 90-minute virtual MRP was convenient to complete. They described the intervention as flexible and easy to learn. Participants indicated that they intended to implement what they learned in the course to reduce work-related stress. This feedback can be used to recommend guidelines for developing meditative interventions aimed at reducing stress in HCWs.

Acknowledgments
This material is based on work supported by the US Department of Veterans Affairs (VA), VA Employee Education Service and with resources from the VA San Diego Healthcare System and the VA Center for Mental Healthcare & Outcomes Research, South Central Mental Illness Research, Education, and Clinical Center at the Central Arkansas Veterans Healthcare System.

According to the National Institute for Occupational Safety and Health (NIOSH), stress is a major problem for more than 18 million US health care workers (HCWs).¹ Increases in technology, high patient acuity, and new demands for meeting institutional benchmarks create stressful clinical work environments. HCWs at the US Department of Veterans Affairs (VA) are perhaps at particular risk of experiencing burnout due to the unique needs of VA patients and bureaucratic demands.² Stress may lead to depression, decreased job satisfaction, and other psychological distress among HCWs.³ This, in turn, affects the delivery of care. High levels of burnout have been associated with increased medication errors, lower quality of care, and lower patient satisfaction scores.^4-10

A Cochrane Review found that mental and physical relaxation reduce stress in HCWs.¹¹ Among these, meditative interventions (eg, mindfulness, meditation, yoga) have demonstrated promise.^12-14 Results from a systematic meta-analysis of meditative interventions for HCWs indicated small-to-moderate improvements in emotional exhaustion, sense of personal accomplishment, and life satisfaction. Additional research is needed to determine effects of meditative interventions on burnout and caregiver burden.¹⁵

Unfortunately, many meditative intervention programs are lengthy and require a significant investment of time. They also require some form of sitting meditation every day, placing additional demands on busy HCWs. There remains a need for practical strategies to reduce HCW stress that are easier to master and practice.

Background

We developed, implemented, evaluated, and modified an evidence-based meditative intervention called the Mantram Repetition Program (MRP) to address workplace stress and burnout. The MRP is a mind-body, spiritually enhanced intervention that offers benefits similar to other types of meditative interventions.¹⁶ MRP is composed of 3 primary components: (1) silently repeating a self-selected, meaningful word or phrase (here called a mantram); (2) intentionally slowing down thoughts and behaviors; and (3) developing the ability to focus on a single task at a time (ie, one-pointed attention). The MRP does not require participants to set aside a specific place to practice, and mantram repetition can be initiated intermittently and privately throughout the day (eg, between tasks, while walking or waiting). Examples of 4 sessions (eg, Mantram 1, 2, 3, and 4) can be found on the PsychArmor Institute website (www.psycharmor.org; San Diego, CA).

Initially, the MRP was offered in a group format, in 6 or 8 weekly, 90-minute face-to-face sessions to both patient and nonpatient populations. Studies in veterans with chronic diseases demonstrated improvements in perceived stress, anxiety, and anger, and increased levels of spiritual well-being and quality of life (QOL).^17-19 Veterans with posttraumatic stress disorder (PTSD) reported improvements in PTSD symptoms, QOL, and spiritual well-being.^20-23 Family caregivers of veterans with dementia reported significant reductions in caregiver burden, depression, and anxiety after participating in the MRP.²⁴

Similar results have substantiated the effects of the MRP among HCWs, including reductions in perceived stress, stress of conscience (ie, the conflict that results from competing values and behaviors in the workplace), and burnout.^25-27 HCWs also reported improvements in mindfulness and spiritual well-being.²⁸ In a randomized controlled trial, South Korean nurse managers who completed the MRP demonstrated significant improvements in psychosocial and spiritual well-being and leadership practice and experienced reductions in burnout compared with that of the control group.²⁷ In a qualitative study, the most frequently reported benefits of the MRP were improvements in managing symptoms of stress, anxiety, and feeling out of control.¹⁸

HCWs reported they found it difficult to attend the 8-week MRP face-to-face group classes. Therefore, we developed a shorter online version of the MRP consisting of six 1-hour educational sessions: 4 online self-learning modules, and 2 live meeting webinars with the course facilitator.²⁸ VA employees were invited to enroll in the program from June 2013 through 2016 through group e-mails and announcements in the VA Employee Education Service newsletters. Those eligible to participate could earn up to 6 hours of continuing education.

Although the program was generally well accepted, feedback from HCWs indicated that providers still lacked enough time to participate fully. We therefore condensed the MRP into one 90-minute, videotaped webinar entitled “Mind-Body-Spiritual Strategies for a Healthy Workforce: The Mantram Repetition Program.” The webinar was delivered in real time in June 2013 and archived for viewing later. This condensed course provided an overview of the development, theory, and practice of MRP core components. Specific instructions included how to choose and use a mantram; the importance of acting slowly with intention to avoid mistakes; and ways of developing single-pointed attention. Participants were invited to complete a standard course evaluation using an online survey.

This article presents results from qualitative analyses of participant feedback for the condensed MRP in a nationwide sample of more than 1,700 HCWs within the VA. We used template summary analysis to identify themes in participants’ responses to 2 open-ended questions: “What about this learning activity was most useful to you?” and “What about this learning activity was least useful to you?” These results have implications for reducing HCW stress and developing training programs for HCWs.

Analysis

Responses to the what was most useful question were downloaded to a spreadsheet file for analyses. Investigators chose summary template analysis, a rapid qualitative analytic technique, as the best strategy for analyzing these textual data. This technique is often used in health services research when it is unrealistic to use more time-consuming qualitative methods, such as coding.²⁹

To begin, the analyst, a PhD-level anthropologist, read through the feedback to identify similar words, phrases, and/or concepts (ie, themes). Once the analyst gained a sense of general themes, she developed category labels using verbatim words and/or phrases in the feedback (similar to developing in vivo codes.³⁰ She listed these categories at the top of a summary template document, providing a definition for each to ensure analytic rigor.

Next, each category was listed down the left side of the template. Participant feedback was copied and pasted from the spreadsheet form into the appropriate category for each of 200 responses. The investigator identified subthemes within each category. After analysis was completed for the first 200 course participants, the analyst grouped similar categories together into broader domains to further organize the data. She then read through the feedback from the remaining 917 course participants to identify negative cases (ie, dissimilarities in feedback). An additional researcher familiar with the condensed MRP training then examined the categories and domains. Together, they discussed and resolved any inconsistencies in interpretation of the data.

To get a better sense of the full range of perspectives about the training, the analyst then read through the written feedback for the what was least useful question. She scanned the feedback for negative cases that contradicted template findings and noted these in a document. A more balanced evaluation of the course emerged through this secondary analysis.

Results

Online surveys were completed by 1,117 participants, of which three-quarters (841) were female. Two hundred eleven (19%) viewed the condensed MRP in real time. The remaining participants viewed an online video of the course. Anonymous course evaluations captured only gender and professional classification of participants. Participants represented a wide range of professional roles. The majority (63%) held clinical positions with direct patient care. The next largest category included administrative or health information personnel (21%). There were also students and trainees among these categories.

Qualitative Findings

Feedback about the course was organized into categories during analysis: (1) instructional format; (2) mode of delivery; (3) course content; (4) professional and personal empowerment; (5) religion and spirituality; and (6) ease of mantram practice. These categories represented 2 broad domains: feedback about the course and feedback about the intervention.

Instructional Format

HCWs often reported that the most useful aspect of the course was the instructional format. Most cited the ease with which they could understand the materials and helpfulness of the examples of mantram practice. The option to download course materials for later reference was also useful. Some HCWs indicated that the course could have been improved by incorporating an experiential component in which participants paused to practice a mantram.

Mode of Delivery

Delivery mode including the convenience of the training and the flexibility of having the course available at both work and home was mentioned in the feedback. Some HCWs reported that the most useful aspect of the training was the on-demand feature, which allowed them to stop and restart the program as needed. A few, however, referenced technical difficulties with the webinar.

Content

HCWs also indicated that general information about mantram repetition and information regarding the benefits of the intervention (eg, stress reduction) were useful. The scientific basis of mantram was described as useful by some, though others reported it as least useful. Practical guidance regarding the appropriate time and place to practice a mantram as well as concrete information regarding how to select a mantram was mentioned as the most useful by other participants.

Professional and Personal Empowerment

Professional and personal empowerment was referenced in evaluations. Professional development, such as learning a strategy for enhancing work performance, was reported as positive. HCWs also reported that learning a new strategy for self-care and coping with stress was useful. Some described having experienced a sense of validation by participating in the course that was empowering. Finally, some HCWs indicated the personal growth experienced as the most useful.

Religion and Spirituality

General statements regarding the utility of having learned a spiritually-based practice that crossed religious boundaries as well as general references to the power of prayer were listed in the feedback. Other HCWs indicated the usefulness of having learned that a mantram could be secular.

Ease of Mantram

HCWs referenced the ease with which a mantram can be learned and/or practiced. Course participants described the simplicity of mantram repetition and referenced its portability (ie, it can be practiced in many different settings). Finally, the overall flexibility of mantram practice of where and when it can be performed was also described as useful.

Discussion

Qualitative feedback from participant evaluations of a 90-minute, virtual online MRP course suggests that HCWs representing all areas of care are interested in learning practical strategies for managing workplace stress. Participants overwhelmingly perceived mantram practice as feasible to implement, with the portability of mantram repetition described as particularly useful. This aspect of mantram repetition represents a distinct advantage over meditative interventions that require a dedicated space and time in which to practice (eg, yoga postures, sitting meditation).

These preliminary findings also suggest that mantram practice is acceptable to HCWs representing a variety of roles. Participants indicated that they valued learning a meditative practice that can be interpreted as spiritual or secular, depending on the word or phrase chosen. Only 1 participant reported that the practice of mantram conflicted with his/her personal beliefs. A small minority of participants who found the discussion of spirituality disconcerting nevertheless indicated that the intervention was acceptable to them.

The finding that even a 90-minute course was challenging for some HCWs to accommodate speaks to the importance of developing short-duration stress-reduction programs. The standardized Mindfulness Based Stress Reduction (MBSR) program consists of 8 weekly 2.5-hour sessions and a full-day retreat for an overall commitment of 29 to 33 hours.³¹ Additionally, a systematic review of meditative interventions for informal and professional caregivers found that programs ranged from 4 to 8 weeks.¹⁵ These lengthier programs are likely more challenging than the condensed MRP.

These results also suggest the importance of general guidelines for meditative intervention courses for reducing HCW stress. The mode of delivery should be as flexible as possible, allowing course participants to start, stop, and restart the program as needed and to participate from a location most convenient to them. Although presenting evidence for clinical effectiveness is critical for establishing credibility, statistical data should be briefly summarized. An experiential component in which participants are encouraged to practice the intervention will enhance learning and ensure the translation of knowledge into practice. Finally, framing meditative practices as compatible with many different faiths and/or secular will enhance their acceptability.

Three recommended components of an overall strategy for reducing occupational burnout in health care settings include modifying the organizational structure and work processes, improving the fit between the organization and HCWs, and promoting and allowing time for individuals to learn strategies for coping with work-related stress.³² This 90-minute online MRP course represents an aspect of an overall strategy to reduce HCW stress and burnout. Providing opportunities for HCWs to learn strategies for managing stress could enhance the quality of care and improve patient outcomes. Future pragmatic trials could determine whether mantram practice impacts clinical care at the VA and elsewhere.

Limitations

All participants were self-selected; therefore, the findings may be biased favorably toward the intervention. These qualitative analyses are not generalizable. HCWs in other, non-VA settings might have different needs and/or stressors that should be considered in future program development. If this intervention is offered to a wider audience, then other formats ought to be offered, such as print, at-home recordings, live meeting, and face-to-face.

Conclusion

Course participants reported that the condensed 90-minute virtual MRP was convenient to complete. They described the intervention as flexible and easy to learn. Participants indicated that they intended to implement what they learned in the course to reduce work-related stress. This feedback can be used to recommend guidelines for developing meditative interventions aimed at reducing stress in HCWs.

Acknowledgments
This material is based on work supported by the US Department of Veterans Affairs (VA), VA Employee Education Service and with resources from the VA San Diego Healthcare System and the VA Center for Mental Healthcare & Outcomes Research, South Central Mental Illness Research, Education, and Clinical Center at the Central Arkansas Veterans Healthcare System.

According to the National Institute for Occupational Safety and Health (NIOSH), stress is a major problem for more than 18 million US health care workers (HCWs).¹ Increases in technology, high patient acuity, and new demands for meeting institutional benchmarks create stressful clinical work environments. HCWs at the US Department of Veterans Affairs (VA) are perhaps at particular risk of experiencing burnout due to the unique needs of VA patients and bureaucratic demands.² Stress may lead to depression, decreased job satisfaction, and other psychological distress among HCWs.³ This, in turn, affects the delivery of care. High levels of burnout have been associated with increased medication errors, lower quality of care, and lower patient satisfaction scores.^4-10

A Cochrane Review found that mental and physical relaxation reduce stress in HCWs.¹¹ Among these, meditative interventions (eg, mindfulness, meditation, yoga) have demonstrated promise.^12-14 Results from a systematic meta-analysis of meditative interventions for HCWs indicated small-to-moderate improvements in emotional exhaustion, sense of personal accomplishment, and life satisfaction. Additional research is needed to determine effects of meditative interventions on burnout and caregiver burden.¹⁵

Unfortunately, many meditative intervention programs are lengthy and require a significant investment of time. They also require some form of sitting meditation every day, placing additional demands on busy HCWs. There remains a need for practical strategies to reduce HCW stress that are easier to master and practice.

Background

We developed, implemented, evaluated, and modified an evidence-based meditative intervention called the Mantram Repetition Program (MRP) to address workplace stress and burnout. The MRP is a mind-body, spiritually enhanced intervention that offers benefits similar to other types of meditative interventions.¹⁶ MRP is composed of 3 primary components: (1) silently repeating a self-selected, meaningful word or phrase (here called a mantram); (2) intentionally slowing down thoughts and behaviors; and (3) developing the ability to focus on a single task at a time (ie, one-pointed attention). The MRP does not require participants to set aside a specific place to practice, and mantram repetition can be initiated intermittently and privately throughout the day (eg, between tasks, while walking or waiting). Examples of 4 sessions (eg, Mantram 1, 2, 3, and 4) can be found on the PsychArmor Institute website (www.psycharmor.org; San Diego, CA).

Initially, the MRP was offered in a group format, in 6 or 8 weekly, 90-minute face-to-face sessions to both patient and nonpatient populations. Studies in veterans with chronic diseases demonstrated improvements in perceived stress, anxiety, and anger, and increased levels of spiritual well-being and quality of life (QOL).^17-19 Veterans with posttraumatic stress disorder (PTSD) reported improvements in PTSD symptoms, QOL, and spiritual well-being.^20-23 Family caregivers of veterans with dementia reported significant reductions in caregiver burden, depression, and anxiety after participating in the MRP.²⁴

Similar results have substantiated the effects of the MRP among HCWs, including reductions in perceived stress, stress of conscience (ie, the conflict that results from competing values and behaviors in the workplace), and burnout.^25-27 HCWs also reported improvements in mindfulness and spiritual well-being.²⁸ In a randomized controlled trial, South Korean nurse managers who completed the MRP demonstrated significant improvements in psychosocial and spiritual well-being and leadership practice and experienced reductions in burnout compared with that of the control group.²⁷ In a qualitative study, the most frequently reported benefits of the MRP were improvements in managing symptoms of stress, anxiety, and feeling out of control.¹⁸

HCWs reported they found it difficult to attend the 8-week MRP face-to-face group classes. Therefore, we developed a shorter online version of the MRP consisting of six 1-hour educational sessions: 4 online self-learning modules, and 2 live meeting webinars with the course facilitator.²⁸ VA employees were invited to enroll in the program from June 2013 through 2016 through group e-mails and announcements in the VA Employee Education Service newsletters. Those eligible to participate could earn up to 6 hours of continuing education.

Although the program was generally well accepted, feedback from HCWs indicated that providers still lacked enough time to participate fully. We therefore condensed the MRP into one 90-minute, videotaped webinar entitled “Mind-Body-Spiritual Strategies for a Healthy Workforce: The Mantram Repetition Program.” The webinar was delivered in real time in June 2013 and archived for viewing later. This condensed course provided an overview of the development, theory, and practice of MRP core components. Specific instructions included how to choose and use a mantram; the importance of acting slowly with intention to avoid mistakes; and ways of developing single-pointed attention. Participants were invited to complete a standard course evaluation using an online survey.

This article presents results from qualitative analyses of participant feedback for the condensed MRP in a nationwide sample of more than 1,700 HCWs within the VA. We used template summary analysis to identify themes in participants’ responses to 2 open-ended questions: “What about this learning activity was most useful to you?” and “What about this learning activity was least useful to you?” These results have implications for reducing HCW stress and developing training programs for HCWs.

Analysis

Responses to the what was most useful question were downloaded to a spreadsheet file for analyses. Investigators chose summary template analysis, a rapid qualitative analytic technique, as the best strategy for analyzing these textual data. This technique is often used in health services research when it is unrealistic to use more time-consuming qualitative methods, such as coding.²⁹

To begin, the analyst, a PhD-level anthropologist, read through the feedback to identify similar words, phrases, and/or concepts (ie, themes). Once the analyst gained a sense of general themes, she developed category labels using verbatim words and/or phrases in the feedback (similar to developing in vivo codes.³⁰ She listed these categories at the top of a summary template document, providing a definition for each to ensure analytic rigor.

Next, each category was listed down the left side of the template. Participant feedback was copied and pasted from the spreadsheet form into the appropriate category for each of 200 responses. The investigator identified subthemes within each category. After analysis was completed for the first 200 course participants, the analyst grouped similar categories together into broader domains to further organize the data. She then read through the feedback from the remaining 917 course participants to identify negative cases (ie, dissimilarities in feedback). An additional researcher familiar with the condensed MRP training then examined the categories and domains. Together, they discussed and resolved any inconsistencies in interpretation of the data.

To get a better sense of the full range of perspectives about the training, the analyst then read through the written feedback for the what was least useful question. She scanned the feedback for negative cases that contradicted template findings and noted these in a document. A more balanced evaluation of the course emerged through this secondary analysis.

Results

Online surveys were completed by 1,117 participants, of which three-quarters (841) were female. Two hundred eleven (19%) viewed the condensed MRP in real time. The remaining participants viewed an online video of the course. Anonymous course evaluations captured only gender and professional classification of participants. Participants represented a wide range of professional roles. The majority (63%) held clinical positions with direct patient care. The next largest category included administrative or health information personnel (21%). There were also students and trainees among these categories.

Qualitative Findings

Feedback about the course was organized into categories during analysis: (1) instructional format; (2) mode of delivery; (3) course content; (4) professional and personal empowerment; (5) religion and spirituality; and (6) ease of mantram practice. These categories represented 2 broad domains: feedback about the course and feedback about the intervention.

Instructional Format

HCWs often reported that the most useful aspect of the course was the instructional format. Most cited the ease with which they could understand the materials and helpfulness of the examples of mantram practice. The option to download course materials for later reference was also useful. Some HCWs indicated that the course could have been improved by incorporating an experiential component in which participants paused to practice a mantram.

Mode of Delivery

Delivery mode including the convenience of the training and the flexibility of having the course available at both work and home was mentioned in the feedback. Some HCWs reported that the most useful aspect of the training was the on-demand feature, which allowed them to stop and restart the program as needed. A few, however, referenced technical difficulties with the webinar.

Content

HCWs also indicated that general information about mantram repetition and information regarding the benefits of the intervention (eg, stress reduction) were useful. The scientific basis of mantram was described as useful by some, though others reported it as least useful. Practical guidance regarding the appropriate time and place to practice a mantram as well as concrete information regarding how to select a mantram was mentioned as the most useful by other participants.

Professional and Personal Empowerment

Professional and personal empowerment was referenced in evaluations. Professional development, such as learning a strategy for enhancing work performance, was reported as positive. HCWs also reported that learning a new strategy for self-care and coping with stress was useful. Some described having experienced a sense of validation by participating in the course that was empowering. Finally, some HCWs indicated the personal growth experienced as the most useful.

Religion and Spirituality

General statements regarding the utility of having learned a spiritually-based practice that crossed religious boundaries as well as general references to the power of prayer were listed in the feedback. Other HCWs indicated the usefulness of having learned that a mantram could be secular.

Ease of Mantram

HCWs referenced the ease with which a mantram can be learned and/or practiced. Course participants described the simplicity of mantram repetition and referenced its portability (ie, it can be practiced in many different settings). Finally, the overall flexibility of mantram practice of where and when it can be performed was also described as useful.

Discussion

Qualitative feedback from participant evaluations of a 90-minute, virtual online MRP course suggests that HCWs representing all areas of care are interested in learning practical strategies for managing workplace stress. Participants overwhelmingly perceived mantram practice as feasible to implement, with the portability of mantram repetition described as particularly useful. This aspect of mantram repetition represents a distinct advantage over meditative interventions that require a dedicated space and time in which to practice (eg, yoga postures, sitting meditation).

These preliminary findings also suggest that mantram practice is acceptable to HCWs representing a variety of roles. Participants indicated that they valued learning a meditative practice that can be interpreted as spiritual or secular, depending on the word or phrase chosen. Only 1 participant reported that the practice of mantram conflicted with his/her personal beliefs. A small minority of participants who found the discussion of spirituality disconcerting nevertheless indicated that the intervention was acceptable to them.

The finding that even a 90-minute course was challenging for some HCWs to accommodate speaks to the importance of developing short-duration stress-reduction programs. The standardized Mindfulness Based Stress Reduction (MBSR) program consists of 8 weekly 2.5-hour sessions and a full-day retreat for an overall commitment of 29 to 33 hours.³¹ Additionally, a systematic review of meditative interventions for informal and professional caregivers found that programs ranged from 4 to 8 weeks.¹⁵ These lengthier programs are likely more challenging than the condensed MRP.

These results also suggest the importance of general guidelines for meditative intervention courses for reducing HCW stress. The mode of delivery should be as flexible as possible, allowing course participants to start, stop, and restart the program as needed and to participate from a location most convenient to them. Although presenting evidence for clinical effectiveness is critical for establishing credibility, statistical data should be briefly summarized. An experiential component in which participants are encouraged to practice the intervention will enhance learning and ensure the translation of knowledge into practice. Finally, framing meditative practices as compatible with many different faiths and/or secular will enhance their acceptability.

Three recommended components of an overall strategy for reducing occupational burnout in health care settings include modifying the organizational structure and work processes, improving the fit between the organization and HCWs, and promoting and allowing time for individuals to learn strategies for coping with work-related stress.³² This 90-minute online MRP course represents an aspect of an overall strategy to reduce HCW stress and burnout. Providing opportunities for HCWs to learn strategies for managing stress could enhance the quality of care and improve patient outcomes. Future pragmatic trials could determine whether mantram practice impacts clinical care at the VA and elsewhere.

Limitations

All participants were self-selected; therefore, the findings may be biased favorably toward the intervention. These qualitative analyses are not generalizable. HCWs in other, non-VA settings might have different needs and/or stressors that should be considered in future program development. If this intervention is offered to a wider audience, then other formats ought to be offered, such as print, at-home recordings, live meeting, and face-to-face.

Conclusion

Course participants reported that the condensed 90-minute virtual MRP was convenient to complete. They described the intervention as flexible and easy to learn. Participants indicated that they intended to implement what they learned in the course to reduce work-related stress. This feedback can be used to recommend guidelines for developing meditative interventions aimed at reducing stress in HCWs.

Acknowledgments
This material is based on work supported by the US Department of Veterans Affairs (VA), VA Employee Education Service and with resources from the VA San Diego Healthcare System and the VA Center for Mental Healthcare & Outcomes Research, South Central Mental Illness Research, Education, and Clinical Center at the Central Arkansas Veterans Healthcare System.