Original Research

Obesity is associated with a significant increase in mortality. It increases the risk of type 2 diabetes mellitus (T2DM), hypertension, hyperlipidemia, and coronary artery disease.¹ T2DM is strongly associated with obesity in all ethnic groups.

Medical nutrition therapy and weight loss are very important for DM management.² This includes providing education about diet modification, increased physical activity, daily calorie intake evaluation, and consistent carbohydrate intake. For patients with T2DM, health care providers (HCPs) should emphasize lowering caloric intake and inducing weight loss for those who are overweight (body mass index [BMI] between 25 and 29.9) and obese (BMI ≥ 30). This can improve glycemic control by decreasing insulin resistance. Initial recommendations for weight loss and physical activity are to lose between 5% and 10% of initial body weight and to accumulate at least 30 minutes of moderate physical activity over the course of most days of the week.^3,4

Several formulas are available to estimate baseline caloric intake for weight maintenance. For weight loss of 1 to 2 pounds per week, lowering 500 to 1,000 calories from daily weight maintenance calories serves the goal. The American Diabetes Association (ADA) also suggests that HCPs recommend diet, physical activity, and behavioral therapy designed to achieve > 5% weight loss to overweight and obese patients with T2DM.⁵

Recognizing the clinical benefits of achieving weight loss in overweight or obese patients with T2DM, we aimed to increase the number of visits in the Endocrine Clinic at Central Arkansas Veterans Healthcare System (CAVHS) in Little Rock that addressed obesity, documented calorie goal for patients who are overweight or obese, and performed an intervention with further education for the patient.

Methods

The study population included veterans with either type 1 DM (T1DM) or T2DM with BMI > 25 on any DM control regimen. We performed a health record review of the eligible patients seen in the CAVHS Endocrine Clinic from June 1, 2016 to July 31, 2016 to determine the baseline percentage of visits that addressed obesity and provided weight loss advice to patients. We obtained a list of patients seen in the clinic during the study period from Strategic Management Service Services at CAVHS. We also obtained information that age, gender, medications, BMI, and last Endocrine clinic HCP assessment from the electronic health record. We reviewed the HCPs notes, including fellows and faculty who were involved in the patients’ treatment, to determine whether their notes documented a BMI > 25 and whether they discussed an intervention for overweight or obesity with the patient. The CAVHS Institutional Review Board reviewed and approved the initiative as a quality improvement study.

Intervention

Our clinic has a defined group of HCPs that we targeted for the intervention. After getting baseline information, during August 2017 we educated these HCPs on the tools available to calculate calorie goal for the patients. We advised the HCPs to use the Mifflin St Jyor equation for estimating energy expenditure and set a goal of initial weight loss between 5% and 7% of body weight. We gave specific instructions and advice to the providers (Table 1). HCPs also received educational material to distribute to patients that provided information on the healthy plate method, discussed how to count calories, and advised them on ADA goals with carbohydrate limitation. We encouraged HCPs to recommend that patients cut between 500 and 1,000 calories daily from their current diet. HCPs also received advice to seek help from clinical dieticians and the VA MOVE! Weight Management Program when appropriate.

Study of Effect of the Intervention

To study the effect of this intervention, we reviewed documentation by HCPs and assessed patient satisfaction. We obtained a list of patients and reviewed HCP notes on patients with BMI > 25 to assess whether providers addressed obesity in November and December 2017. We also evaluated whether HCPs offered a specific intervention to address the problem, such as providing education material to the patient or an estimate of daily calorie goal, or referring them to clinical dietician and/or the MOVE program. Patients received a 5-question survey that assessed their understanding and satisfaction at the end of the visit (Table 2).

Results

Of the 100 charts reviewed prior to intervention, HCPs discussed obesity management with only 6% of patients. After the intervention, we collected data again through chart review of the patients who were overweight or obese and seen for DM in the same clinic during a 2-month period. Of the 100 charts reviewed, we noticed that recognition and management of obesity improved to 60%.

To evaluate the impact of this intervention, patients received a questionnaire at the end of the visit. Nearly all (97%) patients mentioned that the provider discussed weight management during that visit. Most (83%) patients mentioned that weight management was discussed with them during prior visits, while 70% of patients felt their knowledge on working on weight loss had improved. Almost half (46%) were interested in further referral to a dietician or the MOVE program if they did not achieve desired results, but 78% were confident that they could implement the discussed weight management measures.

Discussion

Increased body weight is associated with worsening of DM and can result in poor glycemic control. Achieving weight loss in overweight or obese patients with DM can lead to clinical benefits; however, this is a challenge. In one study, a DM prevention program with lifestyle intervention leading to weight loss significantly reduced the rate of progression from impaired glucose tolerance to DM over a 3-year period and improved cardiovascular risk factors like elevated blood pressure and dyslipidemia.⁶ A randomized trial of an intensive lifestyle intervention to increase physical activity and decrease caloric intake vs standard DM education in people with T2DM showed a modest weight loss of 8.6% of initial weight at 1 year.⁷ This weight loss was associated with significant improvement in blood pressure, glycemic control, fasting blood glucose, high-density lipoprotein (HDL) cholesterol, and triglyceride levels and significant reductions in the use of DM, hypertension, and lipid-lowering medications.⁷ Obesity attributes to dyslipidemia with increased levels of cholesterol, low-density lipoprotein, very low-density lipoprotein, triglycerides, and decreased levels of HDL by about 5%.⁸ Obesity also is associated with hypertension, coronary heart disease, heart failure, and cardiovascular and all-cause mortality.⁹

Limitations

Limitations of this study include the small sample size and that multiple HCPs were involved. The nature of intervention might have differed with different HCPs or in a different setting than a VA clinic. In addition, we did not evaluate the effect on weight loss in specific patients as we only reviewed charts to check whether HCPs addressed weight loss. Nevertheless, our intervention was effective because it improved patient and provider awareness. It also gave us the opportunity to create framework for further collaborations and community building. The Endocrinology department at CAVHS is currently collaborating with the MOVE program, which is a part of the nutrition and food services. We hope to have an endocrinologist involved to provide guidance on medication management for obesity.

Conclusion

At CAVHS a simple intervention was instituted to evaluate whether HCPs were discussing weight loss in patients with DM, providing them with information to assess patients’ daily calorie goal, and prompting them for intervention to achieve weight loss. The intervention led to better management of patients with DM and obesity and greater engagement in weight loss from patients.

This project was a team effort. The clinic nurse documented patient’s BMI on the check in slip. HCPs discussed the problem and specific intervention. The clinical dieticians provided focused education for patients. The clerks collected the patient responses to questionnaire. This project also improved communication within the Endocrine Clinic team. Documentation of HCPs pertaining to addressing obesity improved by 54%. Improved patient satisfaction and insight was evident on patient responses to the questionnaire.

We believe that HCP apathy is a major contributor to the problem of obesity. Small steps like these go a long way for further management of obesity. Most VA hospitals have MOVE programs that provide dietary advice and encourage behavioral changes. However, getting patients to commit to these programs is a challenge. Primary care and endocrine clinics are important services that may help with patient awareness.

This project helped us better recognize patients with obesity and provide them with initial counseling and dietary advice. We received help from clinical dieticians and gave patients the option to join MOVE in situations where initial advice did not yield results and for more consistent follow up.

We tried to improve the care for patients with DM who were overweight or obese at CAVHS by prompting HCPs to focus on obesity as a problem and perform interventions to address this problem. The activities carried out and the data collected were used for internal quality improvement and for encouraging further interventions in the care of these patients.

Obesity is associated with a significant increase in mortality. It increases the risk of type 2 diabetes mellitus (T2DM), hypertension, hyperlipidemia, and coronary artery disease.¹ T2DM is strongly associated with obesity in all ethnic groups.

Medical nutrition therapy and weight loss are very important for DM management.² This includes providing education about diet modification, increased physical activity, daily calorie intake evaluation, and consistent carbohydrate intake. For patients with T2DM, health care providers (HCPs) should emphasize lowering caloric intake and inducing weight loss for those who are overweight (body mass index [BMI] between 25 and 29.9) and obese (BMI ≥ 30). This can improve glycemic control by decreasing insulin resistance. Initial recommendations for weight loss and physical activity are to lose between 5% and 10% of initial body weight and to accumulate at least 30 minutes of moderate physical activity over the course of most days of the week.^3,4

Several formulas are available to estimate baseline caloric intake for weight maintenance. For weight loss of 1 to 2 pounds per week, lowering 500 to 1,000 calories from daily weight maintenance calories serves the goal. The American Diabetes Association (ADA) also suggests that HCPs recommend diet, physical activity, and behavioral therapy designed to achieve > 5% weight loss to overweight and obese patients with T2DM.⁵

Recognizing the clinical benefits of achieving weight loss in overweight or obese patients with T2DM, we aimed to increase the number of visits in the Endocrine Clinic at Central Arkansas Veterans Healthcare System (CAVHS) in Little Rock that addressed obesity, documented calorie goal for patients who are overweight or obese, and performed an intervention with further education for the patient.

Methods

The study population included veterans with either type 1 DM (T1DM) or T2DM with BMI > 25 on any DM control regimen. We performed a health record review of the eligible patients seen in the CAVHS Endocrine Clinic from June 1, 2016 to July 31, 2016 to determine the baseline percentage of visits that addressed obesity and provided weight loss advice to patients. We obtained a list of patients seen in the clinic during the study period from Strategic Management Service Services at CAVHS. We also obtained information that age, gender, medications, BMI, and last Endocrine clinic HCP assessment from the electronic health record. We reviewed the HCPs notes, including fellows and faculty who were involved in the patients’ treatment, to determine whether their notes documented a BMI > 25 and whether they discussed an intervention for overweight or obesity with the patient. The CAVHS Institutional Review Board reviewed and approved the initiative as a quality improvement study.

Intervention

Our clinic has a defined group of HCPs that we targeted for the intervention. After getting baseline information, during August 2017 we educated these HCPs on the tools available to calculate calorie goal for the patients. We advised the HCPs to use the Mifflin St Jyor equation for estimating energy expenditure and set a goal of initial weight loss between 5% and 7% of body weight. We gave specific instructions and advice to the providers (Table 1). HCPs also received educational material to distribute to patients that provided information on the healthy plate method, discussed how to count calories, and advised them on ADA goals with carbohydrate limitation. We encouraged HCPs to recommend that patients cut between 500 and 1,000 calories daily from their current diet. HCPs also received advice to seek help from clinical dieticians and the VA MOVE! Weight Management Program when appropriate.

Study of Effect of the Intervention

To study the effect of this intervention, we reviewed documentation by HCPs and assessed patient satisfaction. We obtained a list of patients and reviewed HCP notes on patients with BMI > 25 to assess whether providers addressed obesity in November and December 2017. We also evaluated whether HCPs offered a specific intervention to address the problem, such as providing education material to the patient or an estimate of daily calorie goal, or referring them to clinical dietician and/or the MOVE program. Patients received a 5-question survey that assessed their understanding and satisfaction at the end of the visit (Table 2).

Results

Of the 100 charts reviewed prior to intervention, HCPs discussed obesity management with only 6% of patients. After the intervention, we collected data again through chart review of the patients who were overweight or obese and seen for DM in the same clinic during a 2-month period. Of the 100 charts reviewed, we noticed that recognition and management of obesity improved to 60%.

To evaluate the impact of this intervention, patients received a questionnaire at the end of the visit. Nearly all (97%) patients mentioned that the provider discussed weight management during that visit. Most (83%) patients mentioned that weight management was discussed with them during prior visits, while 70% of patients felt their knowledge on working on weight loss had improved. Almost half (46%) were interested in further referral to a dietician or the MOVE program if they did not achieve desired results, but 78% were confident that they could implement the discussed weight management measures.

Discussion

Increased body weight is associated with worsening of DM and can result in poor glycemic control. Achieving weight loss in overweight or obese patients with DM can lead to clinical benefits; however, this is a challenge. In one study, a DM prevention program with lifestyle intervention leading to weight loss significantly reduced the rate of progression from impaired glucose tolerance to DM over a 3-year period and improved cardiovascular risk factors like elevated blood pressure and dyslipidemia.⁶ A randomized trial of an intensive lifestyle intervention to increase physical activity and decrease caloric intake vs standard DM education in people with T2DM showed a modest weight loss of 8.6% of initial weight at 1 year.⁷ This weight loss was associated with significant improvement in blood pressure, glycemic control, fasting blood glucose, high-density lipoprotein (HDL) cholesterol, and triglyceride levels and significant reductions in the use of DM, hypertension, and lipid-lowering medications.⁷ Obesity attributes to dyslipidemia with increased levels of cholesterol, low-density lipoprotein, very low-density lipoprotein, triglycerides, and decreased levels of HDL by about 5%.⁸ Obesity also is associated with hypertension, coronary heart disease, heart failure, and cardiovascular and all-cause mortality.⁹

Limitations

Limitations of this study include the small sample size and that multiple HCPs were involved. The nature of intervention might have differed with different HCPs or in a different setting than a VA clinic. In addition, we did not evaluate the effect on weight loss in specific patients as we only reviewed charts to check whether HCPs addressed weight loss. Nevertheless, our intervention was effective because it improved patient and provider awareness. It also gave us the opportunity to create framework for further collaborations and community building. The Endocrinology department at CAVHS is currently collaborating with the MOVE program, which is a part of the nutrition and food services. We hope to have an endocrinologist involved to provide guidance on medication management for obesity.

Conclusion

At CAVHS a simple intervention was instituted to evaluate whether HCPs were discussing weight loss in patients with DM, providing them with information to assess patients’ daily calorie goal, and prompting them for intervention to achieve weight loss. The intervention led to better management of patients with DM and obesity and greater engagement in weight loss from patients.

This project was a team effort. The clinic nurse documented patient’s BMI on the check in slip. HCPs discussed the problem and specific intervention. The clinical dieticians provided focused education for patients. The clerks collected the patient responses to questionnaire. This project also improved communication within the Endocrine Clinic team. Documentation of HCPs pertaining to addressing obesity improved by 54%. Improved patient satisfaction and insight was evident on patient responses to the questionnaire.

We believe that HCP apathy is a major contributor to the problem of obesity. Small steps like these go a long way for further management of obesity. Most VA hospitals have MOVE programs that provide dietary advice and encourage behavioral changes. However, getting patients to commit to these programs is a challenge. Primary care and endocrine clinics are important services that may help with patient awareness.

This project helped us better recognize patients with obesity and provide them with initial counseling and dietary advice. We received help from clinical dieticians and gave patients the option to join MOVE in situations where initial advice did not yield results and for more consistent follow up.

We tried to improve the care for patients with DM who were overweight or obese at CAVHS by prompting HCPs to focus on obesity as a problem and perform interventions to address this problem. The activities carried out and the data collected were used for internal quality improvement and for encouraging further interventions in the care of these patients.

Obesity is associated with a significant increase in mortality. It increases the risk of type 2 diabetes mellitus (T2DM), hypertension, hyperlipidemia, and coronary artery disease.¹ T2DM is strongly associated with obesity in all ethnic groups.

Medical nutrition therapy and weight loss are very important for DM management.² This includes providing education about diet modification, increased physical activity, daily calorie intake evaluation, and consistent carbohydrate intake. For patients with T2DM, health care providers (HCPs) should emphasize lowering caloric intake and inducing weight loss for those who are overweight (body mass index [BMI] between 25 and 29.9) and obese (BMI ≥ 30). This can improve glycemic control by decreasing insulin resistance. Initial recommendations for weight loss and physical activity are to lose between 5% and 10% of initial body weight and to accumulate at least 30 minutes of moderate physical activity over the course of most days of the week.^3,4

Several formulas are available to estimate baseline caloric intake for weight maintenance. For weight loss of 1 to 2 pounds per week, lowering 500 to 1,000 calories from daily weight maintenance calories serves the goal. The American Diabetes Association (ADA) also suggests that HCPs recommend diet, physical activity, and behavioral therapy designed to achieve > 5% weight loss to overweight and obese patients with T2DM.⁵

Recognizing the clinical benefits of achieving weight loss in overweight or obese patients with T2DM, we aimed to increase the number of visits in the Endocrine Clinic at Central Arkansas Veterans Healthcare System (CAVHS) in Little Rock that addressed obesity, documented calorie goal for patients who are overweight or obese, and performed an intervention with further education for the patient.

Methods

The study population included veterans with either type 1 DM (T1DM) or T2DM with BMI > 25 on any DM control regimen. We performed a health record review of the eligible patients seen in the CAVHS Endocrine Clinic from June 1, 2016 to July 31, 2016 to determine the baseline percentage of visits that addressed obesity and provided weight loss advice to patients. We obtained a list of patients seen in the clinic during the study period from Strategic Management Service Services at CAVHS. We also obtained information that age, gender, medications, BMI, and last Endocrine clinic HCP assessment from the electronic health record. We reviewed the HCPs notes, including fellows and faculty who were involved in the patients’ treatment, to determine whether their notes documented a BMI > 25 and whether they discussed an intervention for overweight or obesity with the patient. The CAVHS Institutional Review Board reviewed and approved the initiative as a quality improvement study.

Intervention

Our clinic has a defined group of HCPs that we targeted for the intervention. After getting baseline information, during August 2017 we educated these HCPs on the tools available to calculate calorie goal for the patients. We advised the HCPs to use the Mifflin St Jyor equation for estimating energy expenditure and set a goal of initial weight loss between 5% and 7% of body weight. We gave specific instructions and advice to the providers (Table 1). HCPs also received educational material to distribute to patients that provided information on the healthy plate method, discussed how to count calories, and advised them on ADA goals with carbohydrate limitation. We encouraged HCPs to recommend that patients cut between 500 and 1,000 calories daily from their current diet. HCPs also received advice to seek help from clinical dieticians and the VA MOVE! Weight Management Program when appropriate.

Study of Effect of the Intervention

To study the effect of this intervention, we reviewed documentation by HCPs and assessed patient satisfaction. We obtained a list of patients and reviewed HCP notes on patients with BMI > 25 to assess whether providers addressed obesity in November and December 2017. We also evaluated whether HCPs offered a specific intervention to address the problem, such as providing education material to the patient or an estimate of daily calorie goal, or referring them to clinical dietician and/or the MOVE program. Patients received a 5-question survey that assessed their understanding and satisfaction at the end of the visit (Table 2).

Results

Of the 100 charts reviewed prior to intervention, HCPs discussed obesity management with only 6% of patients. After the intervention, we collected data again through chart review of the patients who were overweight or obese and seen for DM in the same clinic during a 2-month period. Of the 100 charts reviewed, we noticed that recognition and management of obesity improved to 60%.

To evaluate the impact of this intervention, patients received a questionnaire at the end of the visit. Nearly all (97%) patients mentioned that the provider discussed weight management during that visit. Most (83%) patients mentioned that weight management was discussed with them during prior visits, while 70% of patients felt their knowledge on working on weight loss had improved. Almost half (46%) were interested in further referral to a dietician or the MOVE program if they did not achieve desired results, but 78% were confident that they could implement the discussed weight management measures.

Discussion

Increased body weight is associated with worsening of DM and can result in poor glycemic control. Achieving weight loss in overweight or obese patients with DM can lead to clinical benefits; however, this is a challenge. In one study, a DM prevention program with lifestyle intervention leading to weight loss significantly reduced the rate of progression from impaired glucose tolerance to DM over a 3-year period and improved cardiovascular risk factors like elevated blood pressure and dyslipidemia.⁶ A randomized trial of an intensive lifestyle intervention to increase physical activity and decrease caloric intake vs standard DM education in people with T2DM showed a modest weight loss of 8.6% of initial weight at 1 year.⁷ This weight loss was associated with significant improvement in blood pressure, glycemic control, fasting blood glucose, high-density lipoprotein (HDL) cholesterol, and triglyceride levels and significant reductions in the use of DM, hypertension, and lipid-lowering medications.⁷ Obesity attributes to dyslipidemia with increased levels of cholesterol, low-density lipoprotein, very low-density lipoprotein, triglycerides, and decreased levels of HDL by about 5%.⁸ Obesity also is associated with hypertension, coronary heart disease, heart failure, and cardiovascular and all-cause mortality.⁹

Limitations

Limitations of this study include the small sample size and that multiple HCPs were involved. The nature of intervention might have differed with different HCPs or in a different setting than a VA clinic. In addition, we did not evaluate the effect on weight loss in specific patients as we only reviewed charts to check whether HCPs addressed weight loss. Nevertheless, our intervention was effective because it improved patient and provider awareness. It also gave us the opportunity to create framework for further collaborations and community building. The Endocrinology department at CAVHS is currently collaborating with the MOVE program, which is a part of the nutrition and food services. We hope to have an endocrinologist involved to provide guidance on medication management for obesity.

Conclusion

At CAVHS a simple intervention was instituted to evaluate whether HCPs were discussing weight loss in patients with DM, providing them with information to assess patients’ daily calorie goal, and prompting them for intervention to achieve weight loss. The intervention led to better management of patients with DM and obesity and greater engagement in weight loss from patients.

This project was a team effort. The clinic nurse documented patient’s BMI on the check in slip. HCPs discussed the problem and specific intervention. The clinical dieticians provided focused education for patients. The clerks collected the patient responses to questionnaire. This project also improved communication within the Endocrine Clinic team. Documentation of HCPs pertaining to addressing obesity improved by 54%. Improved patient satisfaction and insight was evident on patient responses to the questionnaire.

We believe that HCP apathy is a major contributor to the problem of obesity. Small steps like these go a long way for further management of obesity. Most VA hospitals have MOVE programs that provide dietary advice and encourage behavioral changes. However, getting patients to commit to these programs is a challenge. Primary care and endocrine clinics are important services that may help with patient awareness.

This project helped us better recognize patients with obesity and provide them with initial counseling and dietary advice. We received help from clinical dieticians and gave patients the option to join MOVE in situations where initial advice did not yield results and for more consistent follow up.

We tried to improve the care for patients with DM who were overweight or obese at CAVHS by prompting HCPs to focus on obesity as a problem and perform interventions to address this problem. The activities carried out and the data collected were used for internal quality improvement and for encouraging further interventions in the care of these patients.

The decision to perform stress testing in the evaluation of chest pain is often based on the pretest likelihood of coronary artery disease (CAD).^1-7 Cardiac risk scores, which incorporate smoking status, blood pressure, diabetes mellitus, and cholesterol levels, also may provide further risk stratification.^8-11 Assuming that the prevalence of CAD increases with age, young adults could be deemed low risk, not warranting cardiac screening.¹²

Professional society guidelines from the American College of Cardiology/American Heart Association and American College of Physicians^4,5 recommend stress testing as the initial diagnostic test for CAD in symptomatic patients; additionally, the guidelines also suggest that screening stress tests may confer primary prevention benefit in intermediate-risk asymptomatic patients.^9,13 Exercise treadmill testing is considered the initial modality of choice, given its technical ease and lower cost, compared with stress echocardiography.¹⁴

Previously published reports have shown the limited use of stress testing to screen young asymptomatic adults.^15-17 Because this patient demographic typically has a low pretest likelihood of CAD, positive stress tests are often false-positive results.^7,18 The consequence of false-positive testing may be unnecessary additional cardiac testing, potentially leading to more patient harm than benefit.^18,19 For active-duty service members, false-positive testing also has the potential to affect worldwide deployability and/or sea duty status while further risk stratification is performed; as a result, mission readiness may be impacted.Although the number of clinic visits for chest pain has declined, there has been a discordant increase in the rates of stress testing in the US.^20-22 Additionally, the rate of stress testing among young adults, specifically in the 25- to 34-year age group, has increased in recent years. Given the rising use of stress tests in the young patient population, the clinical use of stress testing needs to be reassessed.

Although much of the literature has already demonstrated the low value of stress testing in young asymptomatic adults, no data currently exist regarding its outpatient use in evaluating young symptomatic patients. The military represents a predominantly young cross-section of the general population suitable for exploring this topic. Using a cohort of active-duty service members, we aimed to determine the use of outpatient stress testing in evaluating young patients with atypical chest pain.

Methods

The US Department of Defense (DoD) Military Health System Database Repository (MDR) and Comprehensive Ambulatory Professional Encounter Record (CAPER) were the data sources for this study. The MDR contains continually updated, longitudinal electronic medical records (EMRs) for nearly 1.4 million active-duty service members and is composed of administrative, medical, pharmacy, and clinical data. The Naval Medical Center Portsmouth (NMCP) Institutional Review Board approved this study.

Study Cohort

We performed chart reviews of service members aged 18 to 35 years who received cardiac stress testing at NMCP, an academic tertiary care center, within 30 days after an office visit for atypical chest pain between October 1, 2010, and September 30, 2015. Atypical chest pain was defined as any outpatient claim with ICD-9 code, 786.5x, in the primary diagnosis field (Table 1).⁴  

Demographics and cardiac risk factors (ie, hypertension, hyperlipidemia, diabetes mellitus, and smoking status) were assessed prior to index chest pain evaluations and defined via ICD-9 codes within outpatient records.

Cardiac Testing Outcomes

Patients were initially categorized by the results of baseline electrocardiograms (ECG) and index stress tests (ie, exercise treadmill or stress echocardiography, exercise or Lexiscan myocardial perfusion imaging, dobutamine stress echocardiography). Positive tests were defined as those having electrical or structural ischemic changes. Chronotropic changes were infrequent and nonpathologic and were not counted. Patient endpoints were either additional cardiac testing or negative index stress test without additional testing.

Statistical Analysis

The agreement between both baseline ECG and index stress test as well as index stress test and additional cardiac testing were analyzed using McNemar test and matched-pair odds ratios (ORs) with corresponding 95% CIs. Analyses were stratified by demographics and cardiac risk factors to assess for potential confounding. Analyses were performed using SAS version 9.4 (Cary, NC).

Results

A total of 1,036 patients were evaluated for atypical chest pain and had index stress testing between October 1, 2010 and September 30, 2015. The study cohort was 69% male with a mean (SD) age of 27.3 (4.7) years. More than 60% of the cohort was older than aged > 25 years. 

In the initial testing cohort, exercise treadmill test (59.3%) and exercise echocardiogram (37.1%) were the most common stress testing modalities. The mean (SD) metabolic equivalents (METS) achieved among individuals who performed exercise stress testing was 13.9 (2.8). There were 65 patients who had a positive baseline ECG/negative index stress test, 958 patients had a negative ECG/negative index test, and 8 patients had a negative ECG/positive index test. 

There were 102 patients (10%) who performed additional cardiac testing. Among this subgroup, 13 patients (1.3%) had additional testing for further evaluation of a positive index stress test (Table 4) and 89 patients (8.6%) had testing for continuing atypical chest pain despite a negative index stress test. 

Coronary computed tomography angiography (CCTA) demonstrated nonobstructive CAD in 3 patients (0.3%) within the study cohort. There was no obstructive CAD identified in our cohort. Two patients had negative left heart catheterizations (LHC). One of these patients had a negative LHC and a negative Lexiscan after a CCTA showed CAD; all 3 of these additional tests were performed for evaluation of continued chest pain despite negative index stress testing. The positive predictive value of cardiac stress testing for nonobstructive CAD in this low-risk population was 15.4% (2 of 13). Stratification by demographics, CAD risk factors, and cardiac test results revealed no presence of confounding factors during analyses.

Discussion

In this retrospective, observational study of 1,036 young patients with atypical chest pain who had stress testing, there was relatively strong agreement between baseline ECG and index stress test results. Individuals also were 8 times more likely to have positive baseline ECGs and negative stress testing than having the opposite finding. Additional cardiac testing similarly demonstrated congruency with index stress testing and showed the propensity for false-positive stress tests. Further testing with CCTA demonstrated minimal nonobstructive CAD in < 1% of the study cohort and 2 LHC were negative. Despite the low prevalence of CAD and apparent low diagnostic use of stress testing in our young cohort, symptomatic service members still require stress testing to determine deployment suitability.

The low yield of outpatient stress testing in our young population is rooted in Bayes’ theorem, which highlights the importance of pretest likelihood in the diagnosis of CAD.^7,23 Because our cohort had a low prevalence and low pretest likelihood of CAD, positive index stress tests were often false-positive results and consequently did not increase the posttest likelihood of CAD, resulting in low positive predictive value. Additional cardiac testing had limited clinical value in our cohort. The 3 cases of nonobstructive CAD were unlikely to be pathologic given the minimal degree of observed stenosis and the 2 LHC did not require revascularization. These results are similar to those shown by Christman and colleagues and Mudrick and colleagues, which highlighted the low yield of additional cardiac studies and low rate of revascularization among symptomatic patients without known cardiac disease, respectively.^18,19

This is the first study, to our knowledge, to quantitatively demonstrate the low use of outpatient stress testing for young adults with atypical chest pain. Previous studies that assessed stress testing for young patients with chest pain in acute settings such as emergency departments and chest pain observation units, similarly demonstrated minimal yield of routine diagnostic testing.^23,24 This further highlights the premise that outpatient and even emergent-setting stress testing in low cardiac risk individuals may be of limited value and not always necessary.

Limitations

There were several study limitations. As a single-center, cross-sectional review, we may not be able to extrapolate our findings to the general population. However, given the low prevalence of CAD in young adults, stress testing would likely have limited value regardless of the sample distribution; so it may be possible to extend our findings beyond our cohort. Also, neither baseline ECG nor index stress test (irrespective of modality) could be given a diagnostic value in predicting ischemia alone; doing so would require comparison with the gold standard—heart catheterization. Although referral bias has been associated with diagnostic performance of stress testing, we did not adjust for this phenomenon.²⁵ Given the higher average metabolic equivalents achieved in our cohort, this potential bias likely did not affect diagnostic performance.

Conclusion

There was low diagnostic use of outpatient stress testing and additional cardiac testing for CAD among young patients with atypical chest pain. The limited value of cardiac stress testing is likely a function of the low CAD prevalence within this population, suggesting that younger patients may not necessarily require stress testing for chest pain evaluations as long as pretest likelihood is low. Despite our results, we maintain that the decision to perform stress testing should still be guided by clinical judgment, but perhaps our findings may alleviate physicians’ concerns over the urgency of when to refer low-risk patients for testing. Although we are cautious in inferring our findings to the general population, the similarity it shares with those from other published reports may suggest its applicability beyond our study cohort.

The decision to perform stress testing in the evaluation of chest pain is often based on the pretest likelihood of coronary artery disease (CAD).^1-7 Cardiac risk scores, which incorporate smoking status, blood pressure, diabetes mellitus, and cholesterol levels, also may provide further risk stratification.^8-11 Assuming that the prevalence of CAD increases with age, young adults could be deemed low risk, not warranting cardiac screening.¹²

Professional society guidelines from the American College of Cardiology/American Heart Association and American College of Physicians^4,5 recommend stress testing as the initial diagnostic test for CAD in symptomatic patients; additionally, the guidelines also suggest that screening stress tests may confer primary prevention benefit in intermediate-risk asymptomatic patients.^9,13 Exercise treadmill testing is considered the initial modality of choice, given its technical ease and lower cost, compared with stress echocardiography.¹⁴

Previously published reports have shown the limited use of stress testing to screen young asymptomatic adults.^15-17 Because this patient demographic typically has a low pretest likelihood of CAD, positive stress tests are often false-positive results.^7,18 The consequence of false-positive testing may be unnecessary additional cardiac testing, potentially leading to more patient harm than benefit.^18,19 For active-duty service members, false-positive testing also has the potential to affect worldwide deployability and/or sea duty status while further risk stratification is performed; as a result, mission readiness may be impacted.Although the number of clinic visits for chest pain has declined, there has been a discordant increase in the rates of stress testing in the US.^20-22 Additionally, the rate of stress testing among young adults, specifically in the 25- to 34-year age group, has increased in recent years. Given the rising use of stress tests in the young patient population, the clinical use of stress testing needs to be reassessed.

Although much of the literature has already demonstrated the low value of stress testing in young asymptomatic adults, no data currently exist regarding its outpatient use in evaluating young symptomatic patients. The military represents a predominantly young cross-section of the general population suitable for exploring this topic. Using a cohort of active-duty service members, we aimed to determine the use of outpatient stress testing in evaluating young patients with atypical chest pain.

Methods

The US Department of Defense (DoD) Military Health System Database Repository (MDR) and Comprehensive Ambulatory Professional Encounter Record (CAPER) were the data sources for this study. The MDR contains continually updated, longitudinal electronic medical records (EMRs) for nearly 1.4 million active-duty service members and is composed of administrative, medical, pharmacy, and clinical data. The Naval Medical Center Portsmouth (NMCP) Institutional Review Board approved this study.

Study Cohort

We performed chart reviews of service members aged 18 to 35 years who received cardiac stress testing at NMCP, an academic tertiary care center, within 30 days after an office visit for atypical chest pain between October 1, 2010, and September 30, 2015. Atypical chest pain was defined as any outpatient claim with ICD-9 code, 786.5x, in the primary diagnosis field (Table 1).⁴  

Demographics and cardiac risk factors (ie, hypertension, hyperlipidemia, diabetes mellitus, and smoking status) were assessed prior to index chest pain evaluations and defined via ICD-9 codes within outpatient records.

Cardiac Testing Outcomes

Patients were initially categorized by the results of baseline electrocardiograms (ECG) and index stress tests (ie, exercise treadmill or stress echocardiography, exercise or Lexiscan myocardial perfusion imaging, dobutamine stress echocardiography). Positive tests were defined as those having electrical or structural ischemic changes. Chronotropic changes were infrequent and nonpathologic and were not counted. Patient endpoints were either additional cardiac testing or negative index stress test without additional testing.

Statistical Analysis

The agreement between both baseline ECG and index stress test as well as index stress test and additional cardiac testing were analyzed using McNemar test and matched-pair odds ratios (ORs) with corresponding 95% CIs. Analyses were stratified by demographics and cardiac risk factors to assess for potential confounding. Analyses were performed using SAS version 9.4 (Cary, NC).

Results

A total of 1,036 patients were evaluated for atypical chest pain and had index stress testing between October 1, 2010 and September 30, 2015. The study cohort was 69% male with a mean (SD) age of 27.3 (4.7) years. More than 60% of the cohort was older than aged > 25 years. 

In the initial testing cohort, exercise treadmill test (59.3%) and exercise echocardiogram (37.1%) were the most common stress testing modalities. The mean (SD) metabolic equivalents (METS) achieved among individuals who performed exercise stress testing was 13.9 (2.8). There were 65 patients who had a positive baseline ECG/negative index stress test, 958 patients had a negative ECG/negative index test, and 8 patients had a negative ECG/positive index test. 

There were 102 patients (10%) who performed additional cardiac testing. Among this subgroup, 13 patients (1.3%) had additional testing for further evaluation of a positive index stress test (Table 4) and 89 patients (8.6%) had testing for continuing atypical chest pain despite a negative index stress test. 

Coronary computed tomography angiography (CCTA) demonstrated nonobstructive CAD in 3 patients (0.3%) within the study cohort. There was no obstructive CAD identified in our cohort. Two patients had negative left heart catheterizations (LHC). One of these patients had a negative LHC and a negative Lexiscan after a CCTA showed CAD; all 3 of these additional tests were performed for evaluation of continued chest pain despite negative index stress testing. The positive predictive value of cardiac stress testing for nonobstructive CAD in this low-risk population was 15.4% (2 of 13). Stratification by demographics, CAD risk factors, and cardiac test results revealed no presence of confounding factors during analyses.

Discussion

In this retrospective, observational study of 1,036 young patients with atypical chest pain who had stress testing, there was relatively strong agreement between baseline ECG and index stress test results. Individuals also were 8 times more likely to have positive baseline ECGs and negative stress testing than having the opposite finding. Additional cardiac testing similarly demonstrated congruency with index stress testing and showed the propensity for false-positive stress tests. Further testing with CCTA demonstrated minimal nonobstructive CAD in < 1% of the study cohort and 2 LHC were negative. Despite the low prevalence of CAD and apparent low diagnostic use of stress testing in our young cohort, symptomatic service members still require stress testing to determine deployment suitability.

The low yield of outpatient stress testing in our young population is rooted in Bayes’ theorem, which highlights the importance of pretest likelihood in the diagnosis of CAD.^7,23 Because our cohort had a low prevalence and low pretest likelihood of CAD, positive index stress tests were often false-positive results and consequently did not increase the posttest likelihood of CAD, resulting in low positive predictive value. Additional cardiac testing had limited clinical value in our cohort. The 3 cases of nonobstructive CAD were unlikely to be pathologic given the minimal degree of observed stenosis and the 2 LHC did not require revascularization. These results are similar to those shown by Christman and colleagues and Mudrick and colleagues, which highlighted the low yield of additional cardiac studies and low rate of revascularization among symptomatic patients without known cardiac disease, respectively.^18,19

This is the first study, to our knowledge, to quantitatively demonstrate the low use of outpatient stress testing for young adults with atypical chest pain. Previous studies that assessed stress testing for young patients with chest pain in acute settings such as emergency departments and chest pain observation units, similarly demonstrated minimal yield of routine diagnostic testing.^23,24 This further highlights the premise that outpatient and even emergent-setting stress testing in low cardiac risk individuals may be of limited value and not always necessary.

Limitations

There were several study limitations. As a single-center, cross-sectional review, we may not be able to extrapolate our findings to the general population. However, given the low prevalence of CAD in young adults, stress testing would likely have limited value regardless of the sample distribution; so it may be possible to extend our findings beyond our cohort. Also, neither baseline ECG nor index stress test (irrespective of modality) could be given a diagnostic value in predicting ischemia alone; doing so would require comparison with the gold standard—heart catheterization. Although referral bias has been associated with diagnostic performance of stress testing, we did not adjust for this phenomenon.²⁵ Given the higher average metabolic equivalents achieved in our cohort, this potential bias likely did not affect diagnostic performance.

Conclusion

There was low diagnostic use of outpatient stress testing and additional cardiac testing for CAD among young patients with atypical chest pain. The limited value of cardiac stress testing is likely a function of the low CAD prevalence within this population, suggesting that younger patients may not necessarily require stress testing for chest pain evaluations as long as pretest likelihood is low. Despite our results, we maintain that the decision to perform stress testing should still be guided by clinical judgment, but perhaps our findings may alleviate physicians’ concerns over the urgency of when to refer low-risk patients for testing. Although we are cautious in inferring our findings to the general population, the similarity it shares with those from other published reports may suggest its applicability beyond our study cohort.

The decision to perform stress testing in the evaluation of chest pain is often based on the pretest likelihood of coronary artery disease (CAD).^1-7 Cardiac risk scores, which incorporate smoking status, blood pressure, diabetes mellitus, and cholesterol levels, also may provide further risk stratification.^8-11 Assuming that the prevalence of CAD increases with age, young adults could be deemed low risk, not warranting cardiac screening.¹²

Professional society guidelines from the American College of Cardiology/American Heart Association and American College of Physicians^4,5 recommend stress testing as the initial diagnostic test for CAD in symptomatic patients; additionally, the guidelines also suggest that screening stress tests may confer primary prevention benefit in intermediate-risk asymptomatic patients.^9,13 Exercise treadmill testing is considered the initial modality of choice, given its technical ease and lower cost, compared with stress echocardiography.¹⁴

Previously published reports have shown the limited use of stress testing to screen young asymptomatic adults.^15-17 Because this patient demographic typically has a low pretest likelihood of CAD, positive stress tests are often false-positive results.^7,18 The consequence of false-positive testing may be unnecessary additional cardiac testing, potentially leading to more patient harm than benefit.^18,19 For active-duty service members, false-positive testing also has the potential to affect worldwide deployability and/or sea duty status while further risk stratification is performed; as a result, mission readiness may be impacted.Although the number of clinic visits for chest pain has declined, there has been a discordant increase in the rates of stress testing in the US.^20-22 Additionally, the rate of stress testing among young adults, specifically in the 25- to 34-year age group, has increased in recent years. Given the rising use of stress tests in the young patient population, the clinical use of stress testing needs to be reassessed.

Although much of the literature has already demonstrated the low value of stress testing in young asymptomatic adults, no data currently exist regarding its outpatient use in evaluating young symptomatic patients. The military represents a predominantly young cross-section of the general population suitable for exploring this topic. Using a cohort of active-duty service members, we aimed to determine the use of outpatient stress testing in evaluating young patients with atypical chest pain.

Methods

The US Department of Defense (DoD) Military Health System Database Repository (MDR) and Comprehensive Ambulatory Professional Encounter Record (CAPER) were the data sources for this study. The MDR contains continually updated, longitudinal electronic medical records (EMRs) for nearly 1.4 million active-duty service members and is composed of administrative, medical, pharmacy, and clinical data. The Naval Medical Center Portsmouth (NMCP) Institutional Review Board approved this study.

Study Cohort

We performed chart reviews of service members aged 18 to 35 years who received cardiac stress testing at NMCP, an academic tertiary care center, within 30 days after an office visit for atypical chest pain between October 1, 2010, and September 30, 2015. Atypical chest pain was defined as any outpatient claim with ICD-9 code, 786.5x, in the primary diagnosis field (Table 1).⁴  

Demographics and cardiac risk factors (ie, hypertension, hyperlipidemia, diabetes mellitus, and smoking status) were assessed prior to index chest pain evaluations and defined via ICD-9 codes within outpatient records.

Cardiac Testing Outcomes

Patients were initially categorized by the results of baseline electrocardiograms (ECG) and index stress tests (ie, exercise treadmill or stress echocardiography, exercise or Lexiscan myocardial perfusion imaging, dobutamine stress echocardiography). Positive tests were defined as those having electrical or structural ischemic changes. Chronotropic changes were infrequent and nonpathologic and were not counted. Patient endpoints were either additional cardiac testing or negative index stress test without additional testing.

Statistical Analysis

The agreement between both baseline ECG and index stress test as well as index stress test and additional cardiac testing were analyzed using McNemar test and matched-pair odds ratios (ORs) with corresponding 95% CIs. Analyses were stratified by demographics and cardiac risk factors to assess for potential confounding. Analyses were performed using SAS version 9.4 (Cary, NC).

Results

A total of 1,036 patients were evaluated for atypical chest pain and had index stress testing between October 1, 2010 and September 30, 2015. The study cohort was 69% male with a mean (SD) age of 27.3 (4.7) years. More than 60% of the cohort was older than aged > 25 years. 

In the initial testing cohort, exercise treadmill test (59.3%) and exercise echocardiogram (37.1%) were the most common stress testing modalities. The mean (SD) metabolic equivalents (METS) achieved among individuals who performed exercise stress testing was 13.9 (2.8). There were 65 patients who had a positive baseline ECG/negative index stress test, 958 patients had a negative ECG/negative index test, and 8 patients had a negative ECG/positive index test. 

There were 102 patients (10%) who performed additional cardiac testing. Among this subgroup, 13 patients (1.3%) had additional testing for further evaluation of a positive index stress test (Table 4) and 89 patients (8.6%) had testing for continuing atypical chest pain despite a negative index stress test. 

Coronary computed tomography angiography (CCTA) demonstrated nonobstructive CAD in 3 patients (0.3%) within the study cohort. There was no obstructive CAD identified in our cohort. Two patients had negative left heart catheterizations (LHC). One of these patients had a negative LHC and a negative Lexiscan after a CCTA showed CAD; all 3 of these additional tests were performed for evaluation of continued chest pain despite negative index stress testing. The positive predictive value of cardiac stress testing for nonobstructive CAD in this low-risk population was 15.4% (2 of 13). Stratification by demographics, CAD risk factors, and cardiac test results revealed no presence of confounding factors during analyses.

Discussion

In this retrospective, observational study of 1,036 young patients with atypical chest pain who had stress testing, there was relatively strong agreement between baseline ECG and index stress test results. Individuals also were 8 times more likely to have positive baseline ECGs and negative stress testing than having the opposite finding. Additional cardiac testing similarly demonstrated congruency with index stress testing and showed the propensity for false-positive stress tests. Further testing with CCTA demonstrated minimal nonobstructive CAD in < 1% of the study cohort and 2 LHC were negative. Despite the low prevalence of CAD and apparent low diagnostic use of stress testing in our young cohort, symptomatic service members still require stress testing to determine deployment suitability.

The low yield of outpatient stress testing in our young population is rooted in Bayes’ theorem, which highlights the importance of pretest likelihood in the diagnosis of CAD.^7,23 Because our cohort had a low prevalence and low pretest likelihood of CAD, positive index stress tests were often false-positive results and consequently did not increase the posttest likelihood of CAD, resulting in low positive predictive value. Additional cardiac testing had limited clinical value in our cohort. The 3 cases of nonobstructive CAD were unlikely to be pathologic given the minimal degree of observed stenosis and the 2 LHC did not require revascularization. These results are similar to those shown by Christman and colleagues and Mudrick and colleagues, which highlighted the low yield of additional cardiac studies and low rate of revascularization among symptomatic patients without known cardiac disease, respectively.^18,19

This is the first study, to our knowledge, to quantitatively demonstrate the low use of outpatient stress testing for young adults with atypical chest pain. Previous studies that assessed stress testing for young patients with chest pain in acute settings such as emergency departments and chest pain observation units, similarly demonstrated minimal yield of routine diagnostic testing.^23,24 This further highlights the premise that outpatient and even emergent-setting stress testing in low cardiac risk individuals may be of limited value and not always necessary.

Limitations

There were several study limitations. As a single-center, cross-sectional review, we may not be able to extrapolate our findings to the general population. However, given the low prevalence of CAD in young adults, stress testing would likely have limited value regardless of the sample distribution; so it may be possible to extend our findings beyond our cohort. Also, neither baseline ECG nor index stress test (irrespective of modality) could be given a diagnostic value in predicting ischemia alone; doing so would require comparison with the gold standard—heart catheterization. Although referral bias has been associated with diagnostic performance of stress testing, we did not adjust for this phenomenon.²⁵ Given the higher average metabolic equivalents achieved in our cohort, this potential bias likely did not affect diagnostic performance.

Conclusion

There was low diagnostic use of outpatient stress testing and additional cardiac testing for CAD among young patients with atypical chest pain. The limited value of cardiac stress testing is likely a function of the low CAD prevalence within this population, suggesting that younger patients may not necessarily require stress testing for chest pain evaluations as long as pretest likelihood is low. Despite our results, we maintain that the decision to perform stress testing should still be guided by clinical judgment, but perhaps our findings may alleviate physicians’ concerns over the urgency of when to refer low-risk patients for testing. Although we are cautious in inferring our findings to the general population, the similarity it shares with those from other published reports may suggest its applicability beyond our study cohort.

Osteoporosis is a disease characterized by the loss of bone density.¹ Bone is normally porous and is in a state of flux due to changes in regeneration caused by osteoclast or osteoblast activity. However, age and other factors can accelerate loss in bone density and lead to decreased bone strength and an increased risk of fracture. In men, bone mineral density (BMD) can begin to decline as early as age 30 to 40 years. By age 80 years, 25% of total bone mass may be lost.²

Of the 44 million Americans with low BMD or osteoporosis, 20% are men.¹ This group accounts for up to 40% of all osteoporotic fractures. About 1 in 4 men aged ≥ 50 years may experience a lifetime fracture. Fractures may lead to chronic pain, disability, increased dependence, and potentially death. These complications cause expenditures upward of $4.1 billion annually in North America alone.^3,4 About 80,000 US men will experience a hip fracture each year, one-third of whom will die within that year. This constitutes a mortality rate 2 to 3 times higher than that of women. Osteoporosis often goes undiagnosed and untreated due to a lack of symptoms until a fracture occurs, underlining the potential benefit of preemptive screening.

In 2007, Shekell and colleagues outlined how the US Department of Veterans Affairs (VA) screened men for osteoporosis.⁵ At the time, 95% of the VA population was male, though it has since dropped to 91%.⁶ Shekell and colleagues estimated that about 200,0000 to 400,0000 male veterans had osteoporosis.⁵ Osteoporotic risk factors deemed specific to veterans were excessive alcohol use, spinal cord injury and lack of weight-bearing exercise, prolonged corticosteroid use, and androgen deprivation therapy in prostate cancer. Different screening techniques were assessed, and the VA recommended the Osteoporosis Self-Assessment Tool (OST).⁵ Many organizations have developed clinical guidance, including who should be screened; however, screening for men remains a controversial area due to a lack of any strong recommendations (Table 1).

Endocrine Society screening guidelines for men are the most specific: testing BMD in men aged ≥ 70 years, or if aged 50 to 69 years with an additional risk factor (eg, low body weight, smoking, chronic obstructive pulmonary disease, chronic steroid use).¹ The Fracture Risk Assessment tool (FRAX) score is often cited as a common screening tool. It is a free online questionnaire that provides a 10-year probability risk of hip or major osteoporotic fracture.¹¹ However, this tool is limited by age, weight, and the assumption that all questions are answered accurately. Some of the information required includes the presence of a number of risk factors, such as alcohol use, glucocorticoids, and medical history of rheumatoid arthritis, among others (Table 2). The OST score, on the other hand, is a calculation that does not take into account other risk factors (Figure 1). This tool categorizes the patient into low, moderate, or high risk for osteoporosis.⁸

In a study of 4,000 men aged ≥ 70 years, Diem and colleagues found that OST performed better than FRAX in identifying men who were osteoporotic as well as reducing the proportion of men referred for dual-energy X-ray absorptiometry (DEXA) scan vs universal screening.¹² The mean study participant was aged 76 years, overweight, and had a history of smoking; the majority were white. An OST score of < 2 captured 64% of the total population, 82% of whom had a T-score of < 2.5, which is a diagnostic for osteoporosis. A FRAX score of 9.3% captured 42% of the total population, but only 59% of patients with a T-score of < 2.5.

A 2017 VA Office of Rural Health study examined the utility of OST to screen referred patients aged > 50 years to receive DEXA scans in patient aligned care team (PACT) clinics at 3 different VA locations.¹³ The study excluded patients who had been screened previously or treated for osteoporosis, were receiving hospice care; 1 site excluded patients aged > 88 years. Two of the sites also reviewed the patient’s medications to screen for agents that may contribute to increased fracture risk. Veterans identified as high risk were referred for education and offered a DEXA scan and treatment. In total, 867 veterans were screened; 19% (168) were deemed high risk, and 6% (53) underwent DEXA scans. The study noted that only 15 patients had reportable DEXA scans and 10 were positive for bone disease.

As there has been documented success in the PACT setting in implementing standardized protocols for screening and treating veterans, it is reasonable to extend the concept into other VA services. The home-based primary care (HBPC) population is especially vulnerable due to the age of patients, limited weight-bearing exercise to improve bone strength, and limited access to DEXA scans due to difficulty traveling outside of the home. Despite these issues, a goal of the HBPC service is to provide continual care for veterans and improve their health so they may return to the community setting. As a result, patients are followed frequently, providing many opportunities for interventions. This study aims to determine the proportion of HBPC patients who are at high risk for osteoporosis and can receive a DEXA scan for evaluation.

Methods

This study was a retrospective chart analysis using descriptive statistics. It was reviewed and approved by the institutional review board at Captain James A. Lovell Federal Health Care Center (FHCC). Patients were included in the study if they were enrolled in the HBPC program at FHCC. Patients were excluded if they were receiving hospice or palliative care, had a limited life expectancy per the HBPC provider, or had a diagnosis of osteoporosis that was being managed by a VA endocrinologist, rheumatologist, or non-VA provider.

The study was conducted from February 1, 2018, through November 30, 2018. All chart reviews were done through the FHCC electronic health record. A minimum of 80 and maximum of 150 charts were reviewed as this was the typical patient volume in the HBPC program. Basic demographic information was collected and analyzed by calculating FRAX and OST scores. With the results, patients were classified as low or high risk of developing osteoporosis, and whether a DEXA scan should be recommended.

Results

After chart review, 83 patients were enrolled in the FHCC HBPC program during the study period. Out of these, 5 patients were excluded due to hospice or palliative care status, limited life expectancy, or had their osteoporosis managed by another non-HBPC provider. As a result, 78 patients were analyzed to determine their risk of osteoporosis (Figure 2). Most of the patients were white males with a median age of 82 years. A majority of the patients did not have any current or previous treatment with bisphosphonates, 77% had normal vitamin D levels, and only 13% (10) were current smokers; of the male patients only 21% (15) had a previous DEXA scan (Table 3).

The FRAX and OST scores for each male patient were calculated (Table 4). Half the patients were low risk for osteoporosis. Just 20% (14) of the patients were at high risk for osteoporosis, and only 6 of those had DEXA scans. However, if expanding the criteria to OST scores of < 2, then only 24% (10) received DEXA scans. When calculating FRAX scores, 30% (21) had ≥ 9.3% for major osteoporotic fracture risk, and only 19% (4) had received a DEXA scan.

Discussion

Based on the collected data, many of the male HBPC patients have not had an evaluation for osteoporosis despite being in a high-risk population and meeting some of the screening guidelines by various organizations.¹ Based on Diem and colleagues and the 2007 VA report, utilizing OST scores could help capture a subset of patients that would be referred for DEXA scans.^5,12 Of the 60% (42) of patients that met OST scores of < 2, 76% (32) of them could have been referred for DEXA scans for osteoporosis evaluation. However, at the time of publication of this article, 50% (16) of the patients have been discharged from the service without interventions. Of the remaining 16 patients, only 2 were referred for a DEXA scan, and 1 patient had confirmed osteoporosis. Currently, these results have been reviewed by the HBPC provider, and plans are in place for DEXA scan referrals for the remaining patients. In addition, for new patients admitted to the program and during annual reviews, the plan is to use OST scores to help screen for osteoporosis.

Limitations

The HBPC population is often in flux due to discharges as patients pass away, become eligible for long-term care, advance to hospice or palliative care status, or see an improvement in their condition to transition back into the community. Along with patients who are bed-bound, have poor prognosis, and barriers to access (eg, transportation issues), interventions for DEXA scan referrals are often not clinically indicated. During calculations of the FRAX score, documentation is often missing from a patient’s medical chart, making it difficult to answer all questions on the questionnaire. This does increase the utility of the OST score as the calculation is much easier and does not rely on other osteoporotic factors. Despite these restrictions for offering DEXA scans, the HBPC service has a high standard of excellence in preventing falls, a major contributor to fractures. Physical therapy services are readily available, nursing visits are frequent and as clinically indicated, vitamin D levels are maintained within normal limits via supplementation, and medication management is performed at least quarterly among other interventions.

Conclusions

The retrospective chart review of patients in the HBPC program suggests that there may be a lack of standardized screening for osteoporosis in the male patient population. As seen within the data, there is great potential for interventions as many of the patients would be candidates for screening based on the OST score. The tool is easy to use and readily accessible to all health care providers and staff. By increasing screening of eligible patients, it also increases the identification of those who would benefit from osteoporosis treatment. While the HBPC population has access limitations (eg, homebound, limited life expectancy), the implementation of a protocol and extension of concepts from this study can be extrapolated into other PACT clinics at VA facilities. Osteoporosis in the male population is often overlooked, but screening procedures can help reduce health care expenditures.

Osteoporosis is a disease characterized by the loss of bone density.¹ Bone is normally porous and is in a state of flux due to changes in regeneration caused by osteoclast or osteoblast activity. However, age and other factors can accelerate loss in bone density and lead to decreased bone strength and an increased risk of fracture. In men, bone mineral density (BMD) can begin to decline as early as age 30 to 40 years. By age 80 years, 25% of total bone mass may be lost.²

Of the 44 million Americans with low BMD or osteoporosis, 20% are men.¹ This group accounts for up to 40% of all osteoporotic fractures. About 1 in 4 men aged ≥ 50 years may experience a lifetime fracture. Fractures may lead to chronic pain, disability, increased dependence, and potentially death. These complications cause expenditures upward of $4.1 billion annually in North America alone.^3,4 About 80,000 US men will experience a hip fracture each year, one-third of whom will die within that year. This constitutes a mortality rate 2 to 3 times higher than that of women. Osteoporosis often goes undiagnosed and untreated due to a lack of symptoms until a fracture occurs, underlining the potential benefit of preemptive screening.

In 2007, Shekell and colleagues outlined how the US Department of Veterans Affairs (VA) screened men for osteoporosis.⁵ At the time, 95% of the VA population was male, though it has since dropped to 91%.⁶ Shekell and colleagues estimated that about 200,0000 to 400,0000 male veterans had osteoporosis.⁵ Osteoporotic risk factors deemed specific to veterans were excessive alcohol use, spinal cord injury and lack of weight-bearing exercise, prolonged corticosteroid use, and androgen deprivation therapy in prostate cancer. Different screening techniques were assessed, and the VA recommended the Osteoporosis Self-Assessment Tool (OST).⁵ Many organizations have developed clinical guidance, including who should be screened; however, screening for men remains a controversial area due to a lack of any strong recommendations (Table 1).

Endocrine Society screening guidelines for men are the most specific: testing BMD in men aged ≥ 70 years, or if aged 50 to 69 years with an additional risk factor (eg, low body weight, smoking, chronic obstructive pulmonary disease, chronic steroid use).¹ The Fracture Risk Assessment tool (FRAX) score is often cited as a common screening tool. It is a free online questionnaire that provides a 10-year probability risk of hip or major osteoporotic fracture.¹¹ However, this tool is limited by age, weight, and the assumption that all questions are answered accurately. Some of the information required includes the presence of a number of risk factors, such as alcohol use, glucocorticoids, and medical history of rheumatoid arthritis, among others (Table 2). The OST score, on the other hand, is a calculation that does not take into account other risk factors (Figure 1). This tool categorizes the patient into low, moderate, or high risk for osteoporosis.⁸

In a study of 4,000 men aged ≥ 70 years, Diem and colleagues found that OST performed better than FRAX in identifying men who were osteoporotic as well as reducing the proportion of men referred for dual-energy X-ray absorptiometry (DEXA) scan vs universal screening.¹² The mean study participant was aged 76 years, overweight, and had a history of smoking; the majority were white. An OST score of < 2 captured 64% of the total population, 82% of whom had a T-score of < 2.5, which is a diagnostic for osteoporosis. A FRAX score of 9.3% captured 42% of the total population, but only 59% of patients with a T-score of < 2.5.

A 2017 VA Office of Rural Health study examined the utility of OST to screen referred patients aged > 50 years to receive DEXA scans in patient aligned care team (PACT) clinics at 3 different VA locations.¹³ The study excluded patients who had been screened previously or treated for osteoporosis, were receiving hospice care; 1 site excluded patients aged > 88 years. Two of the sites also reviewed the patient’s medications to screen for agents that may contribute to increased fracture risk. Veterans identified as high risk were referred for education and offered a DEXA scan and treatment. In total, 867 veterans were screened; 19% (168) were deemed high risk, and 6% (53) underwent DEXA scans. The study noted that only 15 patients had reportable DEXA scans and 10 were positive for bone disease.

As there has been documented success in the PACT setting in implementing standardized protocols for screening and treating veterans, it is reasonable to extend the concept into other VA services. The home-based primary care (HBPC) population is especially vulnerable due to the age of patients, limited weight-bearing exercise to improve bone strength, and limited access to DEXA scans due to difficulty traveling outside of the home. Despite these issues, a goal of the HBPC service is to provide continual care for veterans and improve their health so they may return to the community setting. As a result, patients are followed frequently, providing many opportunities for interventions. This study aims to determine the proportion of HBPC patients who are at high risk for osteoporosis and can receive a DEXA scan for evaluation.

Methods

This study was a retrospective chart analysis using descriptive statistics. It was reviewed and approved by the institutional review board at Captain James A. Lovell Federal Health Care Center (FHCC). Patients were included in the study if they were enrolled in the HBPC program at FHCC. Patients were excluded if they were receiving hospice or palliative care, had a limited life expectancy per the HBPC provider, or had a diagnosis of osteoporosis that was being managed by a VA endocrinologist, rheumatologist, or non-VA provider.

The study was conducted from February 1, 2018, through November 30, 2018. All chart reviews were done through the FHCC electronic health record. A minimum of 80 and maximum of 150 charts were reviewed as this was the typical patient volume in the HBPC program. Basic demographic information was collected and analyzed by calculating FRAX and OST scores. With the results, patients were classified as low or high risk of developing osteoporosis, and whether a DEXA scan should be recommended.

Results

After chart review, 83 patients were enrolled in the FHCC HBPC program during the study period. Out of these, 5 patients were excluded due to hospice or palliative care status, limited life expectancy, or had their osteoporosis managed by another non-HBPC provider. As a result, 78 patients were analyzed to determine their risk of osteoporosis (Figure 2). Most of the patients were white males with a median age of 82 years. A majority of the patients did not have any current or previous treatment with bisphosphonates, 77% had normal vitamin D levels, and only 13% (10) were current smokers; of the male patients only 21% (15) had a previous DEXA scan (Table 3).

The FRAX and OST scores for each male patient were calculated (Table 4). Half the patients were low risk for osteoporosis. Just 20% (14) of the patients were at high risk for osteoporosis, and only 6 of those had DEXA scans. However, if expanding the criteria to OST scores of < 2, then only 24% (10) received DEXA scans. When calculating FRAX scores, 30% (21) had ≥ 9.3% for major osteoporotic fracture risk, and only 19% (4) had received a DEXA scan.

Discussion

Based on the collected data, many of the male HBPC patients have not had an evaluation for osteoporosis despite being in a high-risk population and meeting some of the screening guidelines by various organizations.¹ Based on Diem and colleagues and the 2007 VA report, utilizing OST scores could help capture a subset of patients that would be referred for DEXA scans.^5,12 Of the 60% (42) of patients that met OST scores of < 2, 76% (32) of them could have been referred for DEXA scans for osteoporosis evaluation. However, at the time of publication of this article, 50% (16) of the patients have been discharged from the service without interventions. Of the remaining 16 patients, only 2 were referred for a DEXA scan, and 1 patient had confirmed osteoporosis. Currently, these results have been reviewed by the HBPC provider, and plans are in place for DEXA scan referrals for the remaining patients. In addition, for new patients admitted to the program and during annual reviews, the plan is to use OST scores to help screen for osteoporosis.

Limitations

The HBPC population is often in flux due to discharges as patients pass away, become eligible for long-term care, advance to hospice or palliative care status, or see an improvement in their condition to transition back into the community. Along with patients who are bed-bound, have poor prognosis, and barriers to access (eg, transportation issues), interventions for DEXA scan referrals are often not clinically indicated. During calculations of the FRAX score, documentation is often missing from a patient’s medical chart, making it difficult to answer all questions on the questionnaire. This does increase the utility of the OST score as the calculation is much easier and does not rely on other osteoporotic factors. Despite these restrictions for offering DEXA scans, the HBPC service has a high standard of excellence in preventing falls, a major contributor to fractures. Physical therapy services are readily available, nursing visits are frequent and as clinically indicated, vitamin D levels are maintained within normal limits via supplementation, and medication management is performed at least quarterly among other interventions.

Conclusions

The retrospective chart review of patients in the HBPC program suggests that there may be a lack of standardized screening for osteoporosis in the male patient population. As seen within the data, there is great potential for interventions as many of the patients would be candidates for screening based on the OST score. The tool is easy to use and readily accessible to all health care providers and staff. By increasing screening of eligible patients, it also increases the identification of those who would benefit from osteoporosis treatment. While the HBPC population has access limitations (eg, homebound, limited life expectancy), the implementation of a protocol and extension of concepts from this study can be extrapolated into other PACT clinics at VA facilities. Osteoporosis in the male population is often overlooked, but screening procedures can help reduce health care expenditures.

Osteoporosis is a disease characterized by the loss of bone density.¹ Bone is normally porous and is in a state of flux due to changes in regeneration caused by osteoclast or osteoblast activity. However, age and other factors can accelerate loss in bone density and lead to decreased bone strength and an increased risk of fracture. In men, bone mineral density (BMD) can begin to decline as early as age 30 to 40 years. By age 80 years, 25% of total bone mass may be lost.²

Of the 44 million Americans with low BMD or osteoporosis, 20% are men.¹ This group accounts for up to 40% of all osteoporotic fractures. About 1 in 4 men aged ≥ 50 years may experience a lifetime fracture. Fractures may lead to chronic pain, disability, increased dependence, and potentially death. These complications cause expenditures upward of $4.1 billion annually in North America alone.^3,4 About 80,000 US men will experience a hip fracture each year, one-third of whom will die within that year. This constitutes a mortality rate 2 to 3 times higher than that of women. Osteoporosis often goes undiagnosed and untreated due to a lack of symptoms until a fracture occurs, underlining the potential benefit of preemptive screening.

In 2007, Shekell and colleagues outlined how the US Department of Veterans Affairs (VA) screened men for osteoporosis.⁵ At the time, 95% of the VA population was male, though it has since dropped to 91%.⁶ Shekell and colleagues estimated that about 200,0000 to 400,0000 male veterans had osteoporosis.⁵ Osteoporotic risk factors deemed specific to veterans were excessive alcohol use, spinal cord injury and lack of weight-bearing exercise, prolonged corticosteroid use, and androgen deprivation therapy in prostate cancer. Different screening techniques were assessed, and the VA recommended the Osteoporosis Self-Assessment Tool (OST).⁵ Many organizations have developed clinical guidance, including who should be screened; however, screening for men remains a controversial area due to a lack of any strong recommendations (Table 1).

Endocrine Society screening guidelines for men are the most specific: testing BMD in men aged ≥ 70 years, or if aged 50 to 69 years with an additional risk factor (eg, low body weight, smoking, chronic obstructive pulmonary disease, chronic steroid use).¹ The Fracture Risk Assessment tool (FRAX) score is often cited as a common screening tool. It is a free online questionnaire that provides a 10-year probability risk of hip or major osteoporotic fracture.¹¹ However, this tool is limited by age, weight, and the assumption that all questions are answered accurately. Some of the information required includes the presence of a number of risk factors, such as alcohol use, glucocorticoids, and medical history of rheumatoid arthritis, among others (Table 2). The OST score, on the other hand, is a calculation that does not take into account other risk factors (Figure 1). This tool categorizes the patient into low, moderate, or high risk for osteoporosis.⁸

In a study of 4,000 men aged ≥ 70 years, Diem and colleagues found that OST performed better than FRAX in identifying men who were osteoporotic as well as reducing the proportion of men referred for dual-energy X-ray absorptiometry (DEXA) scan vs universal screening.¹² The mean study participant was aged 76 years, overweight, and had a history of smoking; the majority were white. An OST score of < 2 captured 64% of the total population, 82% of whom had a T-score of < 2.5, which is a diagnostic for osteoporosis. A FRAX score of 9.3% captured 42% of the total population, but only 59% of patients with a T-score of < 2.5.

A 2017 VA Office of Rural Health study examined the utility of OST to screen referred patients aged > 50 years to receive DEXA scans in patient aligned care team (PACT) clinics at 3 different VA locations.¹³ The study excluded patients who had been screened previously or treated for osteoporosis, were receiving hospice care; 1 site excluded patients aged > 88 years. Two of the sites also reviewed the patient’s medications to screen for agents that may contribute to increased fracture risk. Veterans identified as high risk were referred for education and offered a DEXA scan and treatment. In total, 867 veterans were screened; 19% (168) were deemed high risk, and 6% (53) underwent DEXA scans. The study noted that only 15 patients had reportable DEXA scans and 10 were positive for bone disease.

As there has been documented success in the PACT setting in implementing standardized protocols for screening and treating veterans, it is reasonable to extend the concept into other VA services. The home-based primary care (HBPC) population is especially vulnerable due to the age of patients, limited weight-bearing exercise to improve bone strength, and limited access to DEXA scans due to difficulty traveling outside of the home. Despite these issues, a goal of the HBPC service is to provide continual care for veterans and improve their health so they may return to the community setting. As a result, patients are followed frequently, providing many opportunities for interventions. This study aims to determine the proportion of HBPC patients who are at high risk for osteoporosis and can receive a DEXA scan for evaluation.

Methods

This study was a retrospective chart analysis using descriptive statistics. It was reviewed and approved by the institutional review board at Captain James A. Lovell Federal Health Care Center (FHCC). Patients were included in the study if they were enrolled in the HBPC program at FHCC. Patients were excluded if they were receiving hospice or palliative care, had a limited life expectancy per the HBPC provider, or had a diagnosis of osteoporosis that was being managed by a VA endocrinologist, rheumatologist, or non-VA provider.

The study was conducted from February 1, 2018, through November 30, 2018. All chart reviews were done through the FHCC electronic health record. A minimum of 80 and maximum of 150 charts were reviewed as this was the typical patient volume in the HBPC program. Basic demographic information was collected and analyzed by calculating FRAX and OST scores. With the results, patients were classified as low or high risk of developing osteoporosis, and whether a DEXA scan should be recommended.

Results

After chart review, 83 patients were enrolled in the FHCC HBPC program during the study period. Out of these, 5 patients were excluded due to hospice or palliative care status, limited life expectancy, or had their osteoporosis managed by another non-HBPC provider. As a result, 78 patients were analyzed to determine their risk of osteoporosis (Figure 2). Most of the patients were white males with a median age of 82 years. A majority of the patients did not have any current or previous treatment with bisphosphonates, 77% had normal vitamin D levels, and only 13% (10) were current smokers; of the male patients only 21% (15) had a previous DEXA scan (Table 3).

The FRAX and OST scores for each male patient were calculated (Table 4). Half the patients were low risk for osteoporosis. Just 20% (14) of the patients were at high risk for osteoporosis, and only 6 of those had DEXA scans. However, if expanding the criteria to OST scores of < 2, then only 24% (10) received DEXA scans. When calculating FRAX scores, 30% (21) had ≥ 9.3% for major osteoporotic fracture risk, and only 19% (4) had received a DEXA scan.

Discussion

Based on the collected data, many of the male HBPC patients have not had an evaluation for osteoporosis despite being in a high-risk population and meeting some of the screening guidelines by various organizations.¹ Based on Diem and colleagues and the 2007 VA report, utilizing OST scores could help capture a subset of patients that would be referred for DEXA scans.^5,12 Of the 60% (42) of patients that met OST scores of < 2, 76% (32) of them could have been referred for DEXA scans for osteoporosis evaluation. However, at the time of publication of this article, 50% (16) of the patients have been discharged from the service without interventions. Of the remaining 16 patients, only 2 were referred for a DEXA scan, and 1 patient had confirmed osteoporosis. Currently, these results have been reviewed by the HBPC provider, and plans are in place for DEXA scan referrals for the remaining patients. In addition, for new patients admitted to the program and during annual reviews, the plan is to use OST scores to help screen for osteoporosis.

Limitations

The HBPC population is often in flux due to discharges as patients pass away, become eligible for long-term care, advance to hospice or palliative care status, or see an improvement in their condition to transition back into the community. Along with patients who are bed-bound, have poor prognosis, and barriers to access (eg, transportation issues), interventions for DEXA scan referrals are often not clinically indicated. During calculations of the FRAX score, documentation is often missing from a patient’s medical chart, making it difficult to answer all questions on the questionnaire. This does increase the utility of the OST score as the calculation is much easier and does not rely on other osteoporotic factors. Despite these restrictions for offering DEXA scans, the HBPC service has a high standard of excellence in preventing falls, a major contributor to fractures. Physical therapy services are readily available, nursing visits are frequent and as clinically indicated, vitamin D levels are maintained within normal limits via supplementation, and medication management is performed at least quarterly among other interventions.

Conclusions

The retrospective chart review of patients in the HBPC program suggests that there may be a lack of standardized screening for osteoporosis in the male patient population. As seen within the data, there is great potential for interventions as many of the patients would be candidates for screening based on the OST score. The tool is easy to use and readily accessible to all health care providers and staff. By increasing screening of eligible patients, it also increases the identification of those who would benefit from osteoporosis treatment. While the HBPC population has access limitations (eg, homebound, limited life expectancy), the implementation of a protocol and extension of concepts from this study can be extrapolated into other PACT clinics at VA facilities. Osteoporosis in the male population is often overlooked, but screening procedures can help reduce health care expenditures.

Digitalization of patient-specific information over the past 2 decades has dramatically altered health care delivery. Nonetheless, this technology has yet to live up to its promise of improving patient outcomes, in part due to data storage challenges as well as the emphasis on data entry to support administrative and financial goals of the institution.^1-4 Substantially less emphasis has been placed on the retrieval of information required for accurate diagnosis.

A new search engine, Voogle, is now available through Microsoft Internet Explorer (Redmond, WA) to all providers in the US Department of Veterans Affairs (VA) on any intranet-enabled computer behind the VA firewall. Voogle facilitates rapid query-based search and retrieval of patient-specific data in the VA Corporate Data Warehouse (CDW).

Case Example

A veteran presented requesting consideration for implantation of a new device for obstructive sleep apnea. Guidelines for implantation of the new device specify a narrow therapeutic window, so determination of his apnea-hypopnea index (AHI) was critical. The patient had received care at more than 20 VA facilities and knew the approximate year the test had been performed at a non-VA facility.

A health care provider (HCP) using Voogle from his VA computer indexed all Veterans Information Systems and Technology Architecture (VistA) notes for the desired date range. The indexing of > 200 notes was completed in seconds. The HCP opened the indexed records with Voogle and entered a query for “sleep apnea,” which displayed multiple instances of the term within the patient record notes. A VA HCP had previously entered the data from the outside sleep study into a note shortly after the study.

This information was found immediately by sorting the indexed notes by date. The total time required by Voogle to find and display the critical information from the sleep study entered at a different VA more than a dozen years earlier was about 1 minute. These data provided the information needed for decision making at the time of the current patient encounter, without which repeat (and unnecessary) testing would have been required.

Information Overload

Electronic health records (EHRs) such as VistA, upload, store, collate, and present data in near real-time across multiple locations. Although the availability of these data can potentially reduce the risk of error due to missing critical information, its sheer volume limits its utility for point-of-care decision making. Much patient-specific text data found in clinical notes are recorded for administrative, financial, and business purposes rather than to support patient care decision making.^1-3 The majority of data documents processes of care rather than HCP observations, assessment of current status, or plans for care. Much of this text is inserted into templates, consists of imported structured data elements, and may contain repeated copy-and-paste free text.

Data uploaded to the CDW are aggregated from multiple hospitals, each with its own “instance” of VistA. Often the CDW contains thousands of text notes for a single patient. This volume of text may conceal critical historical information needed for patient care mixed with a plethora of duplicated or extraneous text entered to satisfy administrative requirements. The effects of information overload and poor system usability have been studied extensively in other disciplines, but this science has largely not been incorporated into EHR design.^1,3,4

A position paper published recently by the American College of Physicians notes that physician cognitive work is adversely impacted by the incorporation of nonclinical information into the EHR for use by other administrative and financial functions.²

Information Chaos

Beasley and colleagues noted that information in an EHR needed for optimal care may be unavailable, inadequate, scattered, conflicting, lost, or inaccurate, a condition they term information chaos.⁵ Smith and colleagues reported that decision making in 1 of 7 primary care visits was impaired by missing critical information. Surveyed HCPs estimated that 44% of patients with missing information may receive compromised care as a result, including delayed or erroneous diagnosis and increased costs due to duplication of diagnostic testing.⁶

Even when technically available, the usability of patient-specific data needed for accurate diagnosis is compromised if the HCP cannot find the information. In most systems data storage paradigms mirror database design rather than provider cognitive models. Ultimately, the design of current EHR interaction paradigms squanders precious cognitive resources and time, particularly during patient encounters, leaving little available for the cognitive tasks necessary for accurate diagnosis and treatment decisions.^1,3,4,7

VA Corporate Data Warehouse

VistA was implemented as a decentralized system with 130 instances, each of which is a freestanding EHR. However, as all systems share common data structures, the data can be combined from multiple instances when needed. The VA established a CDW more than 15 years ago in order to collate information from multiple sites to support operations as well as to seek new insights. The CDW currently updates nightly from all 130 EHR instances and is the only location in which patient information from all treating sites is combined. Voogle can access the CDW through the Veterans Informatics and Computing Infrastructure (VINCI), which is a mirror of the CDW databases and was established as a secure research environment.

The CDW contains information on 25 million veterans, with about 15 terabytes of text data. Approximately 4 billion data points, including 1 million text notes, are accrued nightly. The Integrated Control Number (ICN), a unique patient identifier, is assigned to each CDW record and is cross-indexed in the master patient index. All CDW data are tied to the ICN, facilitating access to and attribution of all patient data from all VA sites. Voogle relies on this identifier to build indexed files, or domains (which are document collections), of requested specific patient information to support its search algorithm.

Structured Data

Most of the data accrued in an EHR are structured data (such as laboratory test results and vital signs) and stored in a defined database framework. Voogle uses iFind (Intersystems Inc, Cambridge, MA) to index, count, and then search for requested information within structured data fields.

Unstructured Text

In contrast to structured data, text notes are stored as documents that are retrievable by patient, author, date, clinic, as well as numerous other fields. Unstructured (free) text notes are more information rich than either structured data or templated notes since their narrative format more closely parallels providers’ cognitive processes.^1,7 The value of the narrative becomes even more critical in understanding complex clinical scenarios with multiple interacting disease processes. Narratives emphasize important details, reducing cognitive overload by reducing the salience of detail the author deems to be less critical. Narrative notes simultaneously assure availability through the use of unstandardized language, often including specialty and disease-specific abbreviations.¹ Information needed for decision making in the illustrative case in this report was present only in HCP-entered free-text notes, as the structured data from which the free text was derived were not available.

Search

The introduction of search engines can be considered one of the major technologic disruptors of the 21st century.⁸ However, this advance has not yet made significant inroads into health care, despite advances in other domains. As of 2019, EHR users are still required to be familiar with the system’s data and menu structure in order to find needed information (or enter orders, code visits, or any of a number of tasks). Anecdotally, one of the authors (David Eibling) observed that the most common question from his trainees is “How do you . . .?” referring not to the care of the patient but rather to interaction with the EHR.

What is needed is a simple query-based application that finds the data on request. In addition to Voogle, other advances are being made in this arena such as the EMERSE, medical record search engine (project-emerse.org). Voogle was released to VA providers in 2017 and is available through the Internet Explorer browser on VA computers with VA intranet access. The goal of Voogle is to reduce HCP cognitive load by reducing the time and effort needed to seek relevant information for the care of a specific patient.

Natural Language Processing

Linguistic analysis of text seeking to understand its meaning constitutes a rapidly expanding field, with current heavy emphasis on the role of artificial intelligence and machine learning.¹ Advances in processing both structured data and free-text notes in the health care domain is in its infancy, despite the investment of considerable resources. Undoubtedly, advances in this arena will dramatically change provider cognitive work in the next decades.

VistA is coded in MUMPS (Massachusetts General Hospital Utility Multi-Programming System, also known as M), which has been in use for more than 50 years. Voogle employs iKnow, a novel natural language processing (NLP) application that resides in Caché (Intersystems, Boston, MA), the vendor-supported MUMPS infrastructure VistA uses to perform text analysis. iKnow does not attempt to interpret the meaning of text as do other common NLP applications, but instead relies on the expert user to interpret the meaning of the analyzed text. iKnow initially divides sentences into relations (usually verbs) and concepts, and then generates an index of these entities. The efficiency of iKnow results in very rapid indexing—often several thousand notes (not an uncommon number) can be indexed in 20 to 30 seconds. iKnow responds to a user query by searching for specific terms or similar terms within the indexed text, and then displays these terms within the original source documents, similar to well-known commercial search engines. Structured data are indexed by the iFind program simultaneously with free-text indexing (Figure 1).

Security

Maintaining high levels of security of Health Insurance Portability and Accountability (HIPAA)-compliant information in an online application such as Voogle is critical to ensure trust of veterans and HCPs. All patient data accessed by Voogle reside within the secure firewall-protected VINCI environment. All moving information is protected with high-level encryption protocols (transport layer security [TLS]), and data at rest are also encrypted. As the application is online, no data are stored on the accessing device. Voogle uses a secure Microsoft Windows logon using VA Active Directory coupled with VistA authorization to regulate who can see the data and use the application. All access is audited, not only for “sensitive patients,” but also for specific data types. Users are reminded of this Voogle attribute on the home screen.

Accessing Voogle

Voogle is available on the VA intranet to all authorized users at https://voogle.vha.med.va.gov/voogle. To assure high-level security the application can only be accessed with the Internet Explorer browser using established user identification protocols to avoid unauthorized access or duplicative log-in tasks.

Indexing

Indexing is user-driven and is required prior to patient selection and term query. The user is prompted for a patient identifier and a date range. The CDW unique patient identifier is used for all internal processing. However, a social security number look-up table is incorporated to facilitate patient selection. The date field defaults to 3 years but can be extended to approximately the year 2000.

Queries

Entering the patient name in Lastname, Firstname (no space) format will yield a list of indexed patients. All access is audited in order to deter unauthorized queries. Data from a demonstration patient are displayed in Figures 2, 3, 4, 5,
and 6.

Structured Data Searches

Structured data categories that contain the queried term, as well as a term count, are displayed after the “Structured Data” toggle is selected (Figure 2). After the desired category (Figure 2: “Outpatient Rx”) is selected, Voogle accesses the data file and displays it as a grid (medication list, Figure 3). Filter and sort functions enable display of specific medications, drug classes, or date ranges (Figure 4).

Display of Terms Within Text Notes

Selecting a term from the drop-down list (Figure 5) opens a grid with the term highlighted in a snippet of text (Figure 6). Opening the document displays the context of the term, along with negation terms (ie, not, denies, no, etc) in red font if present. Voogle, unlike other NLP tools that attempt to interpret medical notes, relies on interpretation by the HCP user. Duplicate note fragments will be displayed in multiple notes, often across multiple screens, vividly demonstrating the pervasive use of the copy-and-paste text-entry strategy. Voogle satisfies 2 of the 4 recommendations of the recent report on copy-and-paste by Tsou and colleagues.⁹ The Voogle text display grid identifies copy-and-pasted text as well as establishes the provenance of the text (by sorting on the date column). Text can be copied from Voogle into an active Computerized Patient Record System (CPRS) note if needed for active patient care. Reindexing the following day and then repeating the search will demonstrate the newly copied-and-pasted text appended to the sequence.

Limitations

Voogle is unable to access all VA patient data currently. There are a dozen or so clinical domains that are indexed by Voogle that include prescriptions, problem lists, health factors, and others. More domains can be added with minimal effort and would then be available for fast search. The most critical deficiency is its inability to access, index, or query text reports stored as images within VistA Imaging. This includes nearly all reports from outside HCPs, emergency department visits or discharge summaries from unlinked hospitals, anesthesia reports, intensive care unit flow sheets, electrocardiograms, as well as numerous other text reports such as pulmonary function reports or sleep studies. Information that is transcribed by the provider into VistA as text (as in the case presented) is available within the CDW and can be found and displayed by Voogle search.

Voogle requires that the user initiates the indexing process prior to initiating the search process. Although Voogle defaults to 3 years prior to the current date, the user can specify a start date extending to close to the year 2000. The volume of data flowing into the CDW precludes automatic indexing of all patient data, as well as automatic updating of previously indexed data. We have explored the feasibility of queueing scheduled appointments for the following day, and although the strategy shows some promise, avoiding conflict with user-requested on-demand indexing remains challenging.

The current VA network architecture updates the CDW every night, resulting in up to a 24-hour delay in data availability. However, this delay should be reduced to several minutes after implementation of real-time data feeds accompanying the coming transition to a new EHR platform.

Conclusions

The recent introduction of the Joint Legacy Viewer (JLV) to the VA EHR desktop has enhanced the breadth of patient-specific information available to any VHA clinician, with recent enhancements providing access to some community care notes from outside HCPs. Voogle builds on this capability by enabling rapid search of text notes and structured data from multiple VA sites, over an extended time frame, and perhaps entered by hundreds of authors, as demonstrated in the case example. Formal usability and workload studies have not been performed; however, anecdotal reports indicate the application dramatically reduces the time required to search for critical information needed for care of complex patients who have been treated in multiple different VA hospitals and clinics.

The Voogle paradigm of leveraging patient information stored within a large enterprise-wide data warehouse through NLP techniques may be applicable to other systems as well, and warrants exploration. We believe that replacing traditional data search paradigms that require knowledge of data structure with a true query-based paradigm is a potential game changer for health information systems. Ultimately this strategy may help provide an antidote for the information chaos impacting HCP cognition. Moreover, reducing HCP cognitive load and time on task may lessen overall health care costs, reduce provider burn-out, and improve the quality of care received by patients.

Near real-time data feeds and adding additional clinical domains will potentially provide other benefits to patient care. For example, the authors plan to investigate whether sampling incoming data may assist with behind-the-scenes continuous monitoring of indicators of patient status to facilitate early warning of impending physiologic collapse.¹⁰ Other possible applications could include real-time scans for biosurveillance or other population screening requirements.

Acknowledgments
The authors express their sincere appreciation to Leslie DeYoung for documentation and Justin Wilson who constructed much of the graphical user interface for the Voogle application and design. Without their expertise, passion, and commitment the application would not be available as it is now.

Digitalization of patient-specific information over the past 2 decades has dramatically altered health care delivery. Nonetheless, this technology has yet to live up to its promise of improving patient outcomes, in part due to data storage challenges as well as the emphasis on data entry to support administrative and financial goals of the institution.^1-4 Substantially less emphasis has been placed on the retrieval of information required for accurate diagnosis.

A new search engine, Voogle, is now available through Microsoft Internet Explorer (Redmond, WA) to all providers in the US Department of Veterans Affairs (VA) on any intranet-enabled computer behind the VA firewall. Voogle facilitates rapid query-based search and retrieval of patient-specific data in the VA Corporate Data Warehouse (CDW).

Case Example

A veteran presented requesting consideration for implantation of a new device for obstructive sleep apnea. Guidelines for implantation of the new device specify a narrow therapeutic window, so determination of his apnea-hypopnea index (AHI) was critical. The patient had received care at more than 20 VA facilities and knew the approximate year the test had been performed at a non-VA facility.

A health care provider (HCP) using Voogle from his VA computer indexed all Veterans Information Systems and Technology Architecture (VistA) notes for the desired date range. The indexing of > 200 notes was completed in seconds. The HCP opened the indexed records with Voogle and entered a query for “sleep apnea,” which displayed multiple instances of the term within the patient record notes. A VA HCP had previously entered the data from the outside sleep study into a note shortly after the study.

This information was found immediately by sorting the indexed notes by date. The total time required by Voogle to find and display the critical information from the sleep study entered at a different VA more than a dozen years earlier was about 1 minute. These data provided the information needed for decision making at the time of the current patient encounter, without which repeat (and unnecessary) testing would have been required.

Information Overload

Electronic health records (EHRs) such as VistA, upload, store, collate, and present data in near real-time across multiple locations. Although the availability of these data can potentially reduce the risk of error due to missing critical information, its sheer volume limits its utility for point-of-care decision making. Much patient-specific text data found in clinical notes are recorded for administrative, financial, and business purposes rather than to support patient care decision making.^1-3 The majority of data documents processes of care rather than HCP observations, assessment of current status, or plans for care. Much of this text is inserted into templates, consists of imported structured data elements, and may contain repeated copy-and-paste free text.

Data uploaded to the CDW are aggregated from multiple hospitals, each with its own “instance” of VistA. Often the CDW contains thousands of text notes for a single patient. This volume of text may conceal critical historical information needed for patient care mixed with a plethora of duplicated or extraneous text entered to satisfy administrative requirements. The effects of information overload and poor system usability have been studied extensively in other disciplines, but this science has largely not been incorporated into EHR design.^1,3,4

A position paper published recently by the American College of Physicians notes that physician cognitive work is adversely impacted by the incorporation of nonclinical information into the EHR for use by other administrative and financial functions.²

Information Chaos

Beasley and colleagues noted that information in an EHR needed for optimal care may be unavailable, inadequate, scattered, conflicting, lost, or inaccurate, a condition they term information chaos.⁵ Smith and colleagues reported that decision making in 1 of 7 primary care visits was impaired by missing critical information. Surveyed HCPs estimated that 44% of patients with missing information may receive compromised care as a result, including delayed or erroneous diagnosis and increased costs due to duplication of diagnostic testing.⁶

Even when technically available, the usability of patient-specific data needed for accurate diagnosis is compromised if the HCP cannot find the information. In most systems data storage paradigms mirror database design rather than provider cognitive models. Ultimately, the design of current EHR interaction paradigms squanders precious cognitive resources and time, particularly during patient encounters, leaving little available for the cognitive tasks necessary for accurate diagnosis and treatment decisions.^1,3,4,7

VA Corporate Data Warehouse

VistA was implemented as a decentralized system with 130 instances, each of which is a freestanding EHR. However, as all systems share common data structures, the data can be combined from multiple instances when needed. The VA established a CDW more than 15 years ago in order to collate information from multiple sites to support operations as well as to seek new insights. The CDW currently updates nightly from all 130 EHR instances and is the only location in which patient information from all treating sites is combined. Voogle can access the CDW through the Veterans Informatics and Computing Infrastructure (VINCI), which is a mirror of the CDW databases and was established as a secure research environment.

The CDW contains information on 25 million veterans, with about 15 terabytes of text data. Approximately 4 billion data points, including 1 million text notes, are accrued nightly. The Integrated Control Number (ICN), a unique patient identifier, is assigned to each CDW record and is cross-indexed in the master patient index. All CDW data are tied to the ICN, facilitating access to and attribution of all patient data from all VA sites. Voogle relies on this identifier to build indexed files, or domains (which are document collections), of requested specific patient information to support its search algorithm.

Structured Data

Most of the data accrued in an EHR are structured data (such as laboratory test results and vital signs) and stored in a defined database framework. Voogle uses iFind (Intersystems Inc, Cambridge, MA) to index, count, and then search for requested information within structured data fields.

Unstructured Text

In contrast to structured data, text notes are stored as documents that are retrievable by patient, author, date, clinic, as well as numerous other fields. Unstructured (free) text notes are more information rich than either structured data or templated notes since their narrative format more closely parallels providers’ cognitive processes.^1,7 The value of the narrative becomes even more critical in understanding complex clinical scenarios with multiple interacting disease processes. Narratives emphasize important details, reducing cognitive overload by reducing the salience of detail the author deems to be less critical. Narrative notes simultaneously assure availability through the use of unstandardized language, often including specialty and disease-specific abbreviations.¹ Information needed for decision making in the illustrative case in this report was present only in HCP-entered free-text notes, as the structured data from which the free text was derived were not available.

Search

The introduction of search engines can be considered one of the major technologic disruptors of the 21st century.⁸ However, this advance has not yet made significant inroads into health care, despite advances in other domains. As of 2019, EHR users are still required to be familiar with the system’s data and menu structure in order to find needed information (or enter orders, code visits, or any of a number of tasks). Anecdotally, one of the authors (David Eibling) observed that the most common question from his trainees is “How do you . . .?” referring not to the care of the patient but rather to interaction with the EHR.

What is needed is a simple query-based application that finds the data on request. In addition to Voogle, other advances are being made in this arena such as the EMERSE, medical record search engine (project-emerse.org). Voogle was released to VA providers in 2017 and is available through the Internet Explorer browser on VA computers with VA intranet access. The goal of Voogle is to reduce HCP cognitive load by reducing the time and effort needed to seek relevant information for the care of a specific patient.

Natural Language Processing

Linguistic analysis of text seeking to understand its meaning constitutes a rapidly expanding field, with current heavy emphasis on the role of artificial intelligence and machine learning.¹ Advances in processing both structured data and free-text notes in the health care domain is in its infancy, despite the investment of considerable resources. Undoubtedly, advances in this arena will dramatically change provider cognitive work in the next decades.

VistA is coded in MUMPS (Massachusetts General Hospital Utility Multi-Programming System, also known as M), which has been in use for more than 50 years. Voogle employs iKnow, a novel natural language processing (NLP) application that resides in Caché (Intersystems, Boston, MA), the vendor-supported MUMPS infrastructure VistA uses to perform text analysis. iKnow does not attempt to interpret the meaning of text as do other common NLP applications, but instead relies on the expert user to interpret the meaning of the analyzed text. iKnow initially divides sentences into relations (usually verbs) and concepts, and then generates an index of these entities. The efficiency of iKnow results in very rapid indexing—often several thousand notes (not an uncommon number) can be indexed in 20 to 30 seconds. iKnow responds to a user query by searching for specific terms or similar terms within the indexed text, and then displays these terms within the original source documents, similar to well-known commercial search engines. Structured data are indexed by the iFind program simultaneously with free-text indexing (Figure 1).

Security

Maintaining high levels of security of Health Insurance Portability and Accountability (HIPAA)-compliant information in an online application such as Voogle is critical to ensure trust of veterans and HCPs. All patient data accessed by Voogle reside within the secure firewall-protected VINCI environment. All moving information is protected with high-level encryption protocols (transport layer security [TLS]), and data at rest are also encrypted. As the application is online, no data are stored on the accessing device. Voogle uses a secure Microsoft Windows logon using VA Active Directory coupled with VistA authorization to regulate who can see the data and use the application. All access is audited, not only for “sensitive patients,” but also for specific data types. Users are reminded of this Voogle attribute on the home screen.

Accessing Voogle

Voogle is available on the VA intranet to all authorized users at https://voogle.vha.med.va.gov/voogle. To assure high-level security the application can only be accessed with the Internet Explorer browser using established user identification protocols to avoid unauthorized access or duplicative log-in tasks.

Indexing

Indexing is user-driven and is required prior to patient selection and term query. The user is prompted for a patient identifier and a date range. The CDW unique patient identifier is used for all internal processing. However, a social security number look-up table is incorporated to facilitate patient selection. The date field defaults to 3 years but can be extended to approximately the year 2000.

Queries

Entering the patient name in Lastname, Firstname (no space) format will yield a list of indexed patients. All access is audited in order to deter unauthorized queries. Data from a demonstration patient are displayed in Figures 2, 3, 4, 5,
and 6.

Structured Data Searches

Structured data categories that contain the queried term, as well as a term count, are displayed after the “Structured Data” toggle is selected (Figure 2). After the desired category (Figure 2: “Outpatient Rx”) is selected, Voogle accesses the data file and displays it as a grid (medication list, Figure 3). Filter and sort functions enable display of specific medications, drug classes, or date ranges (Figure 4).

Display of Terms Within Text Notes

Selecting a term from the drop-down list (Figure 5) opens a grid with the term highlighted in a snippet of text (Figure 6). Opening the document displays the context of the term, along with negation terms (ie, not, denies, no, etc) in red font if present. Voogle, unlike other NLP tools that attempt to interpret medical notes, relies on interpretation by the HCP user. Duplicate note fragments will be displayed in multiple notes, often across multiple screens, vividly demonstrating the pervasive use of the copy-and-paste text-entry strategy. Voogle satisfies 2 of the 4 recommendations of the recent report on copy-and-paste by Tsou and colleagues.⁹ The Voogle text display grid identifies copy-and-pasted text as well as establishes the provenance of the text (by sorting on the date column). Text can be copied from Voogle into an active Computerized Patient Record System (CPRS) note if needed for active patient care. Reindexing the following day and then repeating the search will demonstrate the newly copied-and-pasted text appended to the sequence.

Limitations

Voogle is unable to access all VA patient data currently. There are a dozen or so clinical domains that are indexed by Voogle that include prescriptions, problem lists, health factors, and others. More domains can be added with minimal effort and would then be available for fast search. The most critical deficiency is its inability to access, index, or query text reports stored as images within VistA Imaging. This includes nearly all reports from outside HCPs, emergency department visits or discharge summaries from unlinked hospitals, anesthesia reports, intensive care unit flow sheets, electrocardiograms, as well as numerous other text reports such as pulmonary function reports or sleep studies. Information that is transcribed by the provider into VistA as text (as in the case presented) is available within the CDW and can be found and displayed by Voogle search.

Voogle requires that the user initiates the indexing process prior to initiating the search process. Although Voogle defaults to 3 years prior to the current date, the user can specify a start date extending to close to the year 2000. The volume of data flowing into the CDW precludes automatic indexing of all patient data, as well as automatic updating of previously indexed data. We have explored the feasibility of queueing scheduled appointments for the following day, and although the strategy shows some promise, avoiding conflict with user-requested on-demand indexing remains challenging.

The current VA network architecture updates the CDW every night, resulting in up to a 24-hour delay in data availability. However, this delay should be reduced to several minutes after implementation of real-time data feeds accompanying the coming transition to a new EHR platform.

Conclusions

The recent introduction of the Joint Legacy Viewer (JLV) to the VA EHR desktop has enhanced the breadth of patient-specific information available to any VHA clinician, with recent enhancements providing access to some community care notes from outside HCPs. Voogle builds on this capability by enabling rapid search of text notes and structured data from multiple VA sites, over an extended time frame, and perhaps entered by hundreds of authors, as demonstrated in the case example. Formal usability and workload studies have not been performed; however, anecdotal reports indicate the application dramatically reduces the time required to search for critical information needed for care of complex patients who have been treated in multiple different VA hospitals and clinics.

The Voogle paradigm of leveraging patient information stored within a large enterprise-wide data warehouse through NLP techniques may be applicable to other systems as well, and warrants exploration. We believe that replacing traditional data search paradigms that require knowledge of data structure with a true query-based paradigm is a potential game changer for health information systems. Ultimately this strategy may help provide an antidote for the information chaos impacting HCP cognition. Moreover, reducing HCP cognitive load and time on task may lessen overall health care costs, reduce provider burn-out, and improve the quality of care received by patients.

Near real-time data feeds and adding additional clinical domains will potentially provide other benefits to patient care. For example, the authors plan to investigate whether sampling incoming data may assist with behind-the-scenes continuous monitoring of indicators of patient status to facilitate early warning of impending physiologic collapse.¹⁰ Other possible applications could include real-time scans for biosurveillance or other population screening requirements.

Acknowledgments
The authors express their sincere appreciation to Leslie DeYoung for documentation and Justin Wilson who constructed much of the graphical user interface for the Voogle application and design. Without their expertise, passion, and commitment the application would not be available as it is now.

Digitalization of patient-specific information over the past 2 decades has dramatically altered health care delivery. Nonetheless, this technology has yet to live up to its promise of improving patient outcomes, in part due to data storage challenges as well as the emphasis on data entry to support administrative and financial goals of the institution.^1-4 Substantially less emphasis has been placed on the retrieval of information required for accurate diagnosis.

A new search engine, Voogle, is now available through Microsoft Internet Explorer (Redmond, WA) to all providers in the US Department of Veterans Affairs (VA) on any intranet-enabled computer behind the VA firewall. Voogle facilitates rapid query-based search and retrieval of patient-specific data in the VA Corporate Data Warehouse (CDW).

Case Example

A veteran presented requesting consideration for implantation of a new device for obstructive sleep apnea. Guidelines for implantation of the new device specify a narrow therapeutic window, so determination of his apnea-hypopnea index (AHI) was critical. The patient had received care at more than 20 VA facilities and knew the approximate year the test had been performed at a non-VA facility.

A health care provider (HCP) using Voogle from his VA computer indexed all Veterans Information Systems and Technology Architecture (VistA) notes for the desired date range. The indexing of > 200 notes was completed in seconds. The HCP opened the indexed records with Voogle and entered a query for “sleep apnea,” which displayed multiple instances of the term within the patient record notes. A VA HCP had previously entered the data from the outside sleep study into a note shortly after the study.

This information was found immediately by sorting the indexed notes by date. The total time required by Voogle to find and display the critical information from the sleep study entered at a different VA more than a dozen years earlier was about 1 minute. These data provided the information needed for decision making at the time of the current patient encounter, without which repeat (and unnecessary) testing would have been required.

Information Overload

Electronic health records (EHRs) such as VistA, upload, store, collate, and present data in near real-time across multiple locations. Although the availability of these data can potentially reduce the risk of error due to missing critical information, its sheer volume limits its utility for point-of-care decision making. Much patient-specific text data found in clinical notes are recorded for administrative, financial, and business purposes rather than to support patient care decision making.^1-3 The majority of data documents processes of care rather than HCP observations, assessment of current status, or plans for care. Much of this text is inserted into templates, consists of imported structured data elements, and may contain repeated copy-and-paste free text.

Data uploaded to the CDW are aggregated from multiple hospitals, each with its own “instance” of VistA. Often the CDW contains thousands of text notes for a single patient. This volume of text may conceal critical historical information needed for patient care mixed with a plethora of duplicated or extraneous text entered to satisfy administrative requirements. The effects of information overload and poor system usability have been studied extensively in other disciplines, but this science has largely not been incorporated into EHR design.^1,3,4

A position paper published recently by the American College of Physicians notes that physician cognitive work is adversely impacted by the incorporation of nonclinical information into the EHR for use by other administrative and financial functions.²

Information Chaos

Beasley and colleagues noted that information in an EHR needed for optimal care may be unavailable, inadequate, scattered, conflicting, lost, or inaccurate, a condition they term information chaos.⁵ Smith and colleagues reported that decision making in 1 of 7 primary care visits was impaired by missing critical information. Surveyed HCPs estimated that 44% of patients with missing information may receive compromised care as a result, including delayed or erroneous diagnosis and increased costs due to duplication of diagnostic testing.⁶

Even when technically available, the usability of patient-specific data needed for accurate diagnosis is compromised if the HCP cannot find the information. In most systems data storage paradigms mirror database design rather than provider cognitive models. Ultimately, the design of current EHR interaction paradigms squanders precious cognitive resources and time, particularly during patient encounters, leaving little available for the cognitive tasks necessary for accurate diagnosis and treatment decisions.^1,3,4,7

VA Corporate Data Warehouse

VistA was implemented as a decentralized system with 130 instances, each of which is a freestanding EHR. However, as all systems share common data structures, the data can be combined from multiple instances when needed. The VA established a CDW more than 15 years ago in order to collate information from multiple sites to support operations as well as to seek new insights. The CDW currently updates nightly from all 130 EHR instances and is the only location in which patient information from all treating sites is combined. Voogle can access the CDW through the Veterans Informatics and Computing Infrastructure (VINCI), which is a mirror of the CDW databases and was established as a secure research environment.

The CDW contains information on 25 million veterans, with about 15 terabytes of text data. Approximately 4 billion data points, including 1 million text notes, are accrued nightly. The Integrated Control Number (ICN), a unique patient identifier, is assigned to each CDW record and is cross-indexed in the master patient index. All CDW data are tied to the ICN, facilitating access to and attribution of all patient data from all VA sites. Voogle relies on this identifier to build indexed files, or domains (which are document collections), of requested specific patient information to support its search algorithm.

Structured Data

Most of the data accrued in an EHR are structured data (such as laboratory test results and vital signs) and stored in a defined database framework. Voogle uses iFind (Intersystems Inc, Cambridge, MA) to index, count, and then search for requested information within structured data fields.

Unstructured Text

In contrast to structured data, text notes are stored as documents that are retrievable by patient, author, date, clinic, as well as numerous other fields. Unstructured (free) text notes are more information rich than either structured data or templated notes since their narrative format more closely parallels providers’ cognitive processes.^1,7 The value of the narrative becomes even more critical in understanding complex clinical scenarios with multiple interacting disease processes. Narratives emphasize important details, reducing cognitive overload by reducing the salience of detail the author deems to be less critical. Narrative notes simultaneously assure availability through the use of unstandardized language, often including specialty and disease-specific abbreviations.¹ Information needed for decision making in the illustrative case in this report was present only in HCP-entered free-text notes, as the structured data from which the free text was derived were not available.

Search

The introduction of search engines can be considered one of the major technologic disruptors of the 21st century.⁸ However, this advance has not yet made significant inroads into health care, despite advances in other domains. As of 2019, EHR users are still required to be familiar with the system’s data and menu structure in order to find needed information (or enter orders, code visits, or any of a number of tasks). Anecdotally, one of the authors (David Eibling) observed that the most common question from his trainees is “How do you . . .?” referring not to the care of the patient but rather to interaction with the EHR.

What is needed is a simple query-based application that finds the data on request. In addition to Voogle, other advances are being made in this arena such as the EMERSE, medical record search engine (project-emerse.org). Voogle was released to VA providers in 2017 and is available through the Internet Explorer browser on VA computers with VA intranet access. The goal of Voogle is to reduce HCP cognitive load by reducing the time and effort needed to seek relevant information for the care of a specific patient.

Natural Language Processing

Linguistic analysis of text seeking to understand its meaning constitutes a rapidly expanding field, with current heavy emphasis on the role of artificial intelligence and machine learning.¹ Advances in processing both structured data and free-text notes in the health care domain is in its infancy, despite the investment of considerable resources. Undoubtedly, advances in this arena will dramatically change provider cognitive work in the next decades.

VistA is coded in MUMPS (Massachusetts General Hospital Utility Multi-Programming System, also known as M), which has been in use for more than 50 years. Voogle employs iKnow, a novel natural language processing (NLP) application that resides in Caché (Intersystems, Boston, MA), the vendor-supported MUMPS infrastructure VistA uses to perform text analysis. iKnow does not attempt to interpret the meaning of text as do other common NLP applications, but instead relies on the expert user to interpret the meaning of the analyzed text. iKnow initially divides sentences into relations (usually verbs) and concepts, and then generates an index of these entities. The efficiency of iKnow results in very rapid indexing—often several thousand notes (not an uncommon number) can be indexed in 20 to 30 seconds. iKnow responds to a user query by searching for specific terms or similar terms within the indexed text, and then displays these terms within the original source documents, similar to well-known commercial search engines. Structured data are indexed by the iFind program simultaneously with free-text indexing (Figure 1).

Security

Maintaining high levels of security of Health Insurance Portability and Accountability (HIPAA)-compliant information in an online application such as Voogle is critical to ensure trust of veterans and HCPs. All patient data accessed by Voogle reside within the secure firewall-protected VINCI environment. All moving information is protected with high-level encryption protocols (transport layer security [TLS]), and data at rest are also encrypted. As the application is online, no data are stored on the accessing device. Voogle uses a secure Microsoft Windows logon using VA Active Directory coupled with VistA authorization to regulate who can see the data and use the application. All access is audited, not only for “sensitive patients,” but also for specific data types. Users are reminded of this Voogle attribute on the home screen.

Accessing Voogle

Voogle is available on the VA intranet to all authorized users at https://voogle.vha.med.va.gov/voogle. To assure high-level security the application can only be accessed with the Internet Explorer browser using established user identification protocols to avoid unauthorized access or duplicative log-in tasks.

Indexing

Indexing is user-driven and is required prior to patient selection and term query. The user is prompted for a patient identifier and a date range. The CDW unique patient identifier is used for all internal processing. However, a social security number look-up table is incorporated to facilitate patient selection. The date field defaults to 3 years but can be extended to approximately the year 2000.

Queries

Entering the patient name in Lastname, Firstname (no space) format will yield a list of indexed patients. All access is audited in order to deter unauthorized queries. Data from a demonstration patient are displayed in Figures 2, 3, 4, 5,
and 6.

Structured Data Searches

Structured data categories that contain the queried term, as well as a term count, are displayed after the “Structured Data” toggle is selected (Figure 2). After the desired category (Figure 2: “Outpatient Rx”) is selected, Voogle accesses the data file and displays it as a grid (medication list, Figure 3). Filter and sort functions enable display of specific medications, drug classes, or date ranges (Figure 4).

Display of Terms Within Text Notes

Selecting a term from the drop-down list (Figure 5) opens a grid with the term highlighted in a snippet of text (Figure 6). Opening the document displays the context of the term, along with negation terms (ie, not, denies, no, etc) in red font if present. Voogle, unlike other NLP tools that attempt to interpret medical notes, relies on interpretation by the HCP user. Duplicate note fragments will be displayed in multiple notes, often across multiple screens, vividly demonstrating the pervasive use of the copy-and-paste text-entry strategy. Voogle satisfies 2 of the 4 recommendations of the recent report on copy-and-paste by Tsou and colleagues.⁹ The Voogle text display grid identifies copy-and-pasted text as well as establishes the provenance of the text (by sorting on the date column). Text can be copied from Voogle into an active Computerized Patient Record System (CPRS) note if needed for active patient care. Reindexing the following day and then repeating the search will demonstrate the newly copied-and-pasted text appended to the sequence.

Limitations

Voogle is unable to access all VA patient data currently. There are a dozen or so clinical domains that are indexed by Voogle that include prescriptions, problem lists, health factors, and others. More domains can be added with minimal effort and would then be available for fast search. The most critical deficiency is its inability to access, index, or query text reports stored as images within VistA Imaging. This includes nearly all reports from outside HCPs, emergency department visits or discharge summaries from unlinked hospitals, anesthesia reports, intensive care unit flow sheets, electrocardiograms, as well as numerous other text reports such as pulmonary function reports or sleep studies. Information that is transcribed by the provider into VistA as text (as in the case presented) is available within the CDW and can be found and displayed by Voogle search.

Voogle requires that the user initiates the indexing process prior to initiating the search process. Although Voogle defaults to 3 years prior to the current date, the user can specify a start date extending to close to the year 2000. The volume of data flowing into the CDW precludes automatic indexing of all patient data, as well as automatic updating of previously indexed data. We have explored the feasibility of queueing scheduled appointments for the following day, and although the strategy shows some promise, avoiding conflict with user-requested on-demand indexing remains challenging.

The current VA network architecture updates the CDW every night, resulting in up to a 24-hour delay in data availability. However, this delay should be reduced to several minutes after implementation of real-time data feeds accompanying the coming transition to a new EHR platform.

Conclusions

The recent introduction of the Joint Legacy Viewer (JLV) to the VA EHR desktop has enhanced the breadth of patient-specific information available to any VHA clinician, with recent enhancements providing access to some community care notes from outside HCPs. Voogle builds on this capability by enabling rapid search of text notes and structured data from multiple VA sites, over an extended time frame, and perhaps entered by hundreds of authors, as demonstrated in the case example. Formal usability and workload studies have not been performed; however, anecdotal reports indicate the application dramatically reduces the time required to search for critical information needed for care of complex patients who have been treated in multiple different VA hospitals and clinics.

The Voogle paradigm of leveraging patient information stored within a large enterprise-wide data warehouse through NLP techniques may be applicable to other systems as well, and warrants exploration. We believe that replacing traditional data search paradigms that require knowledge of data structure with a true query-based paradigm is a potential game changer for health information systems. Ultimately this strategy may help provide an antidote for the information chaos impacting HCP cognition. Moreover, reducing HCP cognitive load and time on task may lessen overall health care costs, reduce provider burn-out, and improve the quality of care received by patients.

Near real-time data feeds and adding additional clinical domains will potentially provide other benefits to patient care. For example, the authors plan to investigate whether sampling incoming data may assist with behind-the-scenes continuous monitoring of indicators of patient status to facilitate early warning of impending physiologic collapse.¹⁰ Other possible applications could include real-time scans for biosurveillance or other population screening requirements.

Acknowledgments
The authors express their sincere appreciation to Leslie DeYoung for documentation and Justin Wilson who constructed much of the graphical user interface for the Voogle application and design. Without their expertise, passion, and commitment the application would not be available as it is now.

As the VA moves toward increased utilization of non-VA care, it is crucial to understand and address the challenges of transitional care faced by dual-use veterans to provide high-quality care that improves healthcare outcomes.^7,11,12 The VA implemented a shift in policy from the Veterans Access, Choice, and Accountability Act of 2014 (Public Law 113-146; “Choice Act”) to the VA Maintaining Internal Systems and Strengthening Integrated Outside Networks (MISSION) Act beginning June 6, 2019.^13,14 Under the MISSION Act, veterans have more ways to access healthcare within the VA’s network and through approved non-VA medical providers in the community known as “community care providers.”¹⁵ This shift expanded the existing VA Choice Act of 2014, where the program allowed those veterans who are unable to schedule an appointment within 30 days of their preferred date or the clinically appropriate date, or on the basis of their place of residence, to elect to receive care from eligible non-VA healthcare entities or providers.^14,15 These efforts to better serve veterans by increasing non-VA care might present added care coordination challenges for patients and their providers when they seek care in the VA.

High-quality transitional care prevents poor outcomes such as hospital readmissions.^16-18 When communication and coordination across healthcare delivery systems are lacking, patients and their families often find themselves at risk for adverse events.^19,20 Past research shows that patients have fewer adverse events when they receive comprehensive postdischarge care, including instructions on medications and self-care, symptom recognition and management, and reminders to attend follow-up appointments.^17,21,22 Although researchers have identified the components of effective transitional care,²³ barriers persist. The communication and collaboration needed to provide coordinated care across healthcare delivery systems are difficult due to the lack of standardized approaches between systems.²⁴ Consequently, follow-up care may be delayed or missed altogether. To our knowledge, there is no published research identifying transitional care challenges for clinicians, staff, and veterans in transitioning from non-VA hospitals to a VA primary care setting.

Study Design

We conducted a qualitative assessment within the VA Eastern Colorado Health Care System, an urban tertiary medical center, as well as urban and rural non-VA hospitals used by veterans. Semi-structured interview guides informed by the practical robust implementation and sustainability (PRISM) model, the Lean approach, and the Ideal Transitions of Care Bridge were used.^25-27 We explored the PRISM domains such as recipient’s characteristics, the interaction with the external environment, and the implementation and sustainability infrastructure to inform the design and implementation of the intervention.²⁵ The Lean approach included methods to optimize processes by maximizing efficiency and minimizing waste.²⁶ The Ideal Transitions of Care Bridge was used to identify the domains in transitions of care such as discharge planning, communication of information, and care coordination.²⁷

Setting and Participants

We identified the top 10 non-VA hospitals serving the most urban and rural veterans in 2015 using VA administrative data. Purposive sampling was used to ensure that urban and rural non-VA hospitals and different roles within these hospitals were represented. VA clinicians and staff were selected from the Denver VA Medical Center, a tertiary hospital within the Eastern Colorado Health Care System and one VA Community-Based Outpatient Clinic (CBOC) that primarily serves rural veterans. The Denver VA Medical Center has three clinics staffed by Patient Aligned Care Teams (PACTs), a model built on the concept of Patient-Centered Medical Home.²⁸ Hospital leadership were initially approached for permission to recruit their staff and to be involved as key informants, and all agreed. To ensure representativeness, diversity of roles was recruited, including PACT primary care physicians, nurses, and other staff members such as medical assistants and administrators. Veterans were approached for sampling if they were discharged from a non-VA hospital during June–September 2015 and used the VA for primary care. This was to ensure that they remembered the process they went through postdischarge at the time of the interview.

Data Collection and Analysis

The evaluation team members (RA, EL, and MM) conducted the interviews from November 2015 to July 2016. Clinicians, staff, and veterans were asked semi-structured questions about their experiences and their role in transitioning VA patients across systems (see Appendix for interview guides). Veterans were asked to describe their experience and satisfaction with the current postdischarge transition process. We stopped the interviews when we reached data saturation.²⁹

Interviews were audio-recorded, transcribed verbatim, and validated (transcribed interviews were double-checked against recording) to ensure data quality and accuracy. Coding was guided by a conventional content analysis technique^{30, 31} using a deductive and inductive coding approach.³¹ The deductive coding approach was drawn from the Ideal Transitions of Care Bridge and PRISM domains. ^32,33 Two evaluation team members (RA and EL) defined the initial code book by independently coding the first three interviews, worked to clarify the meanings of emergent codes, and came to a consensus when disagreements occurred. Next, a priori codes were added by team members to include the PRISM domains. These PRISM domains included the implementation and sustainability infrastructure, the external environment, the characteristics of intervention recipients, and the organizational and patient perspectives of an intervention.

Additional emergent codes were added to the code book and agreed upon by team members (RA, EL, and MM). Consistent with previously used methods, consensus building was achieved by identifying and resolving differences by discussing with team members (RA, EL, MM, CB, and RB).²⁹ Codes were examined and organized into themes by team members.^29,34-36 This process was continued until no new themes were identified. Results were reviewed by all evaluation team members to assess thoroughness and comprehensiveness.^34,35 In addition, team members triangulated the findings with VA and non-VA participants to ensure validity and reduce researcher bias.^29,37

We conducted a total of 70 interviews with 23 VA and 29 non-VA hospital clinicians and staff and 18 veterans (Table 1). Overall, we found that there was no standardized process for transitioning veterans across healthcare delivery systems. Participants reported that transitions were often inefficient when non-VA hospitals could not (1) identify patients as veterans and notify VA primary care of discharge; (2) transfer non-VA hospital medical records to VA primary care; (3) obtain follow-up care appointments with VA primary care; and (4) write VA formulary medications for veterans to fill at VA pharmacies. In addition, participants discussed about facilitators and suggestions to overcome these inefficiencies and improve transitional care (Table2). We mapped the identified barriers as well as the suggestions for improvement to the PRISM and the Ideal Transitions of Care Bridge domains (Table 3).

Unable to Identify Patients as Veterans and Notify VA Primary Care of Discharge

VA and non-VA participants reported difficulty in communicating about veterans’ hospitalizations and discharge follow-up needs across systems. Non-VA clinicians referenced difficulty in identifying patients as veterans to communicate with VA, except in instances where the VA is a payor, while VA providers described feeling largely uninformed of the veterans non-VA hospitalization. For non-VA clinicians, the lack of a systematic method for veteran identification often left them to inadvertently identify veteran status by asking about their primary care clinicians and insurance and even through an offhanded comment made by the veteran. If a veteran was identified, non-VA clinicians described being uncertain about the best way to notify VA primary care of the patient’s impending discharge. Veterans described instances of the non-VA hospital knowing their veteran status upon admission, but accounts varied on whether the non-VA hospital notified the VA primary care of their hospitalization (Table 2, Theme 1).

Unable to Transfer Non-VA Hospital Medical Records to VA Primary Care

VA clinicians discussed about the challenges associated with obtaining the veteran’s medical record from the non-VA hospitals, and when it was received, it was often incomplete information and significantly delayed. They described relying on the veteran’s description of the care received, which was not complete or accurate information needed to make clinical judgment or coordinate follow-up care. Non-VA clinicians mentioned about trying several methods for transferring the medical record to VA primary care, including discharge summary via electronic system and sometimes solely relying on patients to deliver discharge paperwork to their primary care clinicians. In instances where non-VA hospitals sent discharge paperwork to VA, there was no way for non-VA hospitals to verify whether the faxed electronic medical record was received by the VA hospital. Most of the veterans discussed receiving written postdischarge instructions to take to their VA primary care clinicians; however, they were unsure whether the VA primary care received their medical record or any other information from the non-VA hospital (Table 2, Theme 2).

Unable to Obtain Follow-Up Care Appointments with VA Primary Care

All participants described how difficult it was to obtain a follow-up appointment for veterans with VA primary care. This often resulted in delayed follow-up care. VA clinicians also shared that a non-VA hospitalization can be the impetus for a veteran to seek care at the VA for the very first time. Once eligibility is determined, the veteran is assigned a VA primary care clinician. This process may take up to six weeks, and in the meantime, the veteran is scheduled in VA urgent care for immediate postdischarge care. This lag in primary care assignment creates delayed and fragmented care (Table 2, Theme 3).

Non-VA clinicians, administrators, and staff also discussed the difficulties in scheduling follow-up care with VA primary care. Although discharge paperwork instructed patients to see their VA clinicians, there was no process in place for non-VA clinicians to confirm whether the follow-up care was received due to lack of bilateral communication. In addition, veterans discussed the inefficiencies in scheduling follow-up appointments with VA clinicians where attempts to follow-up with primary care clinicians took eight weeks or more. Several veterans described walking into the clinic without an appointment asking to be seen postdischarge or utilizing the VA emergency department for follow-up care after discharge from a non-VA hospital. Veterans admitted utilizing the VA emergency department for nonemergent reasons such as filling their prescriptions because they are unable to see a VA PCP in a timely manner (Table 2, Theme 3).

Unable to Write VA Formulary Medications for Veterans to Fill at VA Pharmacies

All participants described the difficulties in obtaining medications at VA pharmacies when prescribed by the non-VA hospital clinicians. VA clinicians often had to reassess, and rewrite prescriptions written by clinicians, causing delays. Moreover, rural VA clinicians described lack of VA pharmacies in their locations, where veterans had to mail order medications, causing further delays in needed medications. Non-VA clinicians echoed these frustrations. They noted that veterans were confused about their VA pharmacy benefits as well as the need for the non-VA clinicians to follow VA formulary guidelines. Veterans expressed that it was especially challenging to physically go to the VA pharmacy to pick up medications after discharge due to lack of transportation, limited VA pharmacy hours, and long wait times. Several veterans discussed paying for their prescriptions out of pocket even though they had VA pharmacy benefits because it was more convenient to use the non-VA pharmacy. In other instances, veterans discussed going to a VA emergency department and waiting for hours to have their non-VA clinician prescription rewritten by a VA clinician (Table 2, Theme 4).

Facilitators of the Current Transition Process

Several participants provided examples of when transitional care communication between systems occurred seamlessly. VA staff and veterans noted that the VA increased the availability of urgent care appointments, which allowed for timelier postacute care follow-up appointments. Non-VA hospital clinicians also noted the availability of additional appointment slots but stated that they did not learn about these additional appointments directly from the VA. Instead, they learned of these through medical residents caring for patients at both VA and non-VA hospitals. One VA CBOC designated two nurses to care for walk-in veterans for their postdischarge follow-up needs. Some VA participants also noted that the VA Call Center Nurses occasionally called veterans upon discharge to schedule a follow-up appointment and facilitated timely care.

Participants from a VA CBOC discussed being part of a Community Transitions Consortium aimed at identifying high-utilizing patients (veteran and nonveteran) and improving communication across systems. The consortium members discussed each facility’s transition-of-care process, described having access to local non-VA hospital medical records and a backline phone number at the non-VA hospitals to coordinate transitional care. This allowed the VA clinicians to learn about non-VA hospital processes and veteran needs.

Suggestions for Improving the Transitional Care Process

VA and non-VA clinicians suggested hiring a VA liaison, preferably with a clinical background to facilitate care coordination across healthcare systems. They recommended that this person work closely with VA primary care, strengthen the relationship with non-VA hospitals, and help veterans learn more about the transition-of-care processes. Topics discussed for veteran education included how to (1) access their primary care team; (2) alert VA of non-VA hospitalization and the billing process; (3) recognize symptoms and manage care; and (4) obtain follow-up care appointments. Furthermore, they suggested that the liaison would help facilitate the transfer of medical records between VA and non-VA hospitals. Other suggestions included allowing veterans to fill prescriptions at non-VA pharmacies and dedicating a phone line for non-VA clinicians to speak to VA clinicians and staff.

Veterans agreed that improvements to the current process should include an efficient system for obtaining medications and the ability to schedule timely follow-up appointments. Furthermore, veterans wanted education about the VA transition-of-care process following a non-VA hospitalization, including payment and VA notification processes (Table 2, Theme 5).

Participants described the current transitional care process as inefficient with specific barriers that have negative consequences on patient care and clinician and staff work processes. They described difficulties in obtaining medications prescribed by non-VA clinicians from VA pharmacies, delays in follow-up appointments at the VA, and lack of bilateral communication between systems and medical record transfer. Participants also provided concrete suggestions to improving the current process, including a care coordinator with clinical background. These findings are important in the context of VA increasing veteran access to care in the community.

Despite an increasing emphasis on veteran access to non-VA care as a result of the VA strategic goals and several new programs,^7,12,13 there has not been a close examination of the current transition-of-care process from non-VA hospitals to VA primary care. Several studies have shown that the period following a hospitalization is especially vulnerable and associated with adverse events such as readmission, high cost, and death.^12,31,32 Our findings agree with previous research that identified medical record transfer across systems as one of the most challenging issues contributing to deficits in communication between care teams.³³ In addition, our study brought into focus the significant challenges faced by veterans in obtaining medications post non-VA hospital discharge. Addressing these key barriers in transitional care will improve the quality, safety, and value of healthcare in the current transition process.^38,39

Based on our findings, our participants’ concern in transitional care can be addressed in various ways. First, as veterans are increasingly receiving care in the community, identifying their veteran status early on in the non-VA hospital setting could help in improved, real time communication with the VA. This could be done by updating patient intake forms to ask patients whether they are veterans or not. Second, VA policy-level changes should work to provide veterans access to non-VA pharmacy benefits equivalent to the access patients are receiving for hospital, specialty, and outpatient care. Third, patient and provider satisfaction for dual-use veterans should be examined closely. Although participants expressed frustration with the overall transitions of care from non-VA hospitals to VA primary care setting, influence of this on the Quadruple Aim-improving patient outcomes, experience, and reducing clinician and staff burnout should be examined closely.⁴⁰ Fourth, evidence-based interventions such as nurse-led transitional care programs that have proven helpful in reducing adverse outcomes in both VA and non-VA settings will be useful to implement.^41-45 Such programs could be located in the VA, and a care coordinator role could help facilitate transitional care needs for veterans by working with multiple non-VA hospitals.

The limitations of this study are that the perspectives shared by these participants may not represent all VA and non-VA hospitals as well as veterans’ experiences with transition of care. In addition, the study was conducted in one state and the findings may not be applicable to other healthcare systems. However, our study highlighted the consistent challenges of receiving care across VA and other hospital systems. Two strengths of this study are that it was conducted by multidisciplinary research team members with expertise in qualitative research, clinical care, and implementation science and that we obtained convergent information from VA, non-VA, and veteran participants.

Our current transition-of-care process has several shortcomings. There was a clear agreement on barriers, facilitators, and suggestions for improving the current transitions-of-care process among VA and non-VA hospital participants, as well as from veterans who experienced transitions across different delivery systems. Transitioning veterans to VA primary care following a non-VA hospitalization is a crucial first step for improving care for veterans and reducing adverse outcomes such as avoidable hospital readmissions and death.

These results describe the inefficiencies experienced by patients, clinicians, and staff and their suggestions to alleviate these barriers for optimal continuum of care. To avoid frustration and inefficiencies, the increased emphasis of providing non-VA care for veterans should consider the challenges experienced in transitional care and the opportunities for increased coordination of care.

As the VA moves toward increased utilization of non-VA care, it is crucial to understand and address the challenges of transitional care faced by dual-use veterans to provide high-quality care that improves healthcare outcomes.^7,11,12 The VA implemented a shift in policy from the Veterans Access, Choice, and Accountability Act of 2014 (Public Law 113-146; “Choice Act”) to the VA Maintaining Internal Systems and Strengthening Integrated Outside Networks (MISSION) Act beginning June 6, 2019.^13,14 Under the MISSION Act, veterans have more ways to access healthcare within the VA’s network and through approved non-VA medical providers in the community known as “community care providers.”¹⁵ This shift expanded the existing VA Choice Act of 2014, where the program allowed those veterans who are unable to schedule an appointment within 30 days of their preferred date or the clinically appropriate date, or on the basis of their place of residence, to elect to receive care from eligible non-VA healthcare entities or providers.^14,15 These efforts to better serve veterans by increasing non-VA care might present added care coordination challenges for patients and their providers when they seek care in the VA.

High-quality transitional care prevents poor outcomes such as hospital readmissions.^16-18 When communication and coordination across healthcare delivery systems are lacking, patients and their families often find themselves at risk for adverse events.^19,20 Past research shows that patients have fewer adverse events when they receive comprehensive postdischarge care, including instructions on medications and self-care, symptom recognition and management, and reminders to attend follow-up appointments.^17,21,22 Although researchers have identified the components of effective transitional care,²³ barriers persist. The communication and collaboration needed to provide coordinated care across healthcare delivery systems are difficult due to the lack of standardized approaches between systems.²⁴ Consequently, follow-up care may be delayed or missed altogether. To our knowledge, there is no published research identifying transitional care challenges for clinicians, staff, and veterans in transitioning from non-VA hospitals to a VA primary care setting.

Study Design

We conducted a qualitative assessment within the VA Eastern Colorado Health Care System, an urban tertiary medical center, as well as urban and rural non-VA hospitals used by veterans. Semi-structured interview guides informed by the practical robust implementation and sustainability (PRISM) model, the Lean approach, and the Ideal Transitions of Care Bridge were used.^25-27 We explored the PRISM domains such as recipient’s characteristics, the interaction with the external environment, and the implementation and sustainability infrastructure to inform the design and implementation of the intervention.²⁵ The Lean approach included methods to optimize processes by maximizing efficiency and minimizing waste.²⁶ The Ideal Transitions of Care Bridge was used to identify the domains in transitions of care such as discharge planning, communication of information, and care coordination.²⁷

Setting and Participants

We identified the top 10 non-VA hospitals serving the most urban and rural veterans in 2015 using VA administrative data. Purposive sampling was used to ensure that urban and rural non-VA hospitals and different roles within these hospitals were represented. VA clinicians and staff were selected from the Denver VA Medical Center, a tertiary hospital within the Eastern Colorado Health Care System and one VA Community-Based Outpatient Clinic (CBOC) that primarily serves rural veterans. The Denver VA Medical Center has three clinics staffed by Patient Aligned Care Teams (PACTs), a model built on the concept of Patient-Centered Medical Home.²⁸ Hospital leadership were initially approached for permission to recruit their staff and to be involved as key informants, and all agreed. To ensure representativeness, diversity of roles was recruited, including PACT primary care physicians, nurses, and other staff members such as medical assistants and administrators. Veterans were approached for sampling if they were discharged from a non-VA hospital during June–September 2015 and used the VA for primary care. This was to ensure that they remembered the process they went through postdischarge at the time of the interview.

Data Collection and Analysis

The evaluation team members (RA, EL, and MM) conducted the interviews from November 2015 to July 2016. Clinicians, staff, and veterans were asked semi-structured questions about their experiences and their role in transitioning VA patients across systems (see Appendix for interview guides). Veterans were asked to describe their experience and satisfaction with the current postdischarge transition process. We stopped the interviews when we reached data saturation.²⁹

Interviews were audio-recorded, transcribed verbatim, and validated (transcribed interviews were double-checked against recording) to ensure data quality and accuracy. Coding was guided by a conventional content analysis technique^{30, 31} using a deductive and inductive coding approach.³¹ The deductive coding approach was drawn from the Ideal Transitions of Care Bridge and PRISM domains. ^32,33 Two evaluation team members (RA and EL) defined the initial code book by independently coding the first three interviews, worked to clarify the meanings of emergent codes, and came to a consensus when disagreements occurred. Next, a priori codes were added by team members to include the PRISM domains. These PRISM domains included the implementation and sustainability infrastructure, the external environment, the characteristics of intervention recipients, and the organizational and patient perspectives of an intervention.

Additional emergent codes were added to the code book and agreed upon by team members (RA, EL, and MM). Consistent with previously used methods, consensus building was achieved by identifying and resolving differences by discussing with team members (RA, EL, MM, CB, and RB).²⁹ Codes were examined and organized into themes by team members.^29,34-36 This process was continued until no new themes were identified. Results were reviewed by all evaluation team members to assess thoroughness and comprehensiveness.^34,35 In addition, team members triangulated the findings with VA and non-VA participants to ensure validity and reduce researcher bias.^29,37

We conducted a total of 70 interviews with 23 VA and 29 non-VA hospital clinicians and staff and 18 veterans (Table 1). Overall, we found that there was no standardized process for transitioning veterans across healthcare delivery systems. Participants reported that transitions were often inefficient when non-VA hospitals could not (1) identify patients as veterans and notify VA primary care of discharge; (2) transfer non-VA hospital medical records to VA primary care; (3) obtain follow-up care appointments with VA primary care; and (4) write VA formulary medications for veterans to fill at VA pharmacies. In addition, participants discussed about facilitators and suggestions to overcome these inefficiencies and improve transitional care (Table2). We mapped the identified barriers as well as the suggestions for improvement to the PRISM and the Ideal Transitions of Care Bridge domains (Table 3).

Unable to Identify Patients as Veterans and Notify VA Primary Care of Discharge

VA and non-VA participants reported difficulty in communicating about veterans’ hospitalizations and discharge follow-up needs across systems. Non-VA clinicians referenced difficulty in identifying patients as veterans to communicate with VA, except in instances where the VA is a payor, while VA providers described feeling largely uninformed of the veterans non-VA hospitalization. For non-VA clinicians, the lack of a systematic method for veteran identification often left them to inadvertently identify veteran status by asking about their primary care clinicians and insurance and even through an offhanded comment made by the veteran. If a veteran was identified, non-VA clinicians described being uncertain about the best way to notify VA primary care of the patient’s impending discharge. Veterans described instances of the non-VA hospital knowing their veteran status upon admission, but accounts varied on whether the non-VA hospital notified the VA primary care of their hospitalization (Table 2, Theme 1).

Unable to Transfer Non-VA Hospital Medical Records to VA Primary Care

VA clinicians discussed about the challenges associated with obtaining the veteran’s medical record from the non-VA hospitals, and when it was received, it was often incomplete information and significantly delayed. They described relying on the veteran’s description of the care received, which was not complete or accurate information needed to make clinical judgment or coordinate follow-up care. Non-VA clinicians mentioned about trying several methods for transferring the medical record to VA primary care, including discharge summary via electronic system and sometimes solely relying on patients to deliver discharge paperwork to their primary care clinicians. In instances where non-VA hospitals sent discharge paperwork to VA, there was no way for non-VA hospitals to verify whether the faxed electronic medical record was received by the VA hospital. Most of the veterans discussed receiving written postdischarge instructions to take to their VA primary care clinicians; however, they were unsure whether the VA primary care received their medical record or any other information from the non-VA hospital (Table 2, Theme 2).

Unable to Obtain Follow-Up Care Appointments with VA Primary Care

All participants described how difficult it was to obtain a follow-up appointment for veterans with VA primary care. This often resulted in delayed follow-up care. VA clinicians also shared that a non-VA hospitalization can be the impetus for a veteran to seek care at the VA for the very first time. Once eligibility is determined, the veteran is assigned a VA primary care clinician. This process may take up to six weeks, and in the meantime, the veteran is scheduled in VA urgent care for immediate postdischarge care. This lag in primary care assignment creates delayed and fragmented care (Table 2, Theme 3).

Non-VA clinicians, administrators, and staff also discussed the difficulties in scheduling follow-up care with VA primary care. Although discharge paperwork instructed patients to see their VA clinicians, there was no process in place for non-VA clinicians to confirm whether the follow-up care was received due to lack of bilateral communication. In addition, veterans discussed the inefficiencies in scheduling follow-up appointments with VA clinicians where attempts to follow-up with primary care clinicians took eight weeks or more. Several veterans described walking into the clinic without an appointment asking to be seen postdischarge or utilizing the VA emergency department for follow-up care after discharge from a non-VA hospital. Veterans admitted utilizing the VA emergency department for nonemergent reasons such as filling their prescriptions because they are unable to see a VA PCP in a timely manner (Table 2, Theme 3).

Unable to Write VA Formulary Medications for Veterans to Fill at VA Pharmacies

All participants described the difficulties in obtaining medications at VA pharmacies when prescribed by the non-VA hospital clinicians. VA clinicians often had to reassess, and rewrite prescriptions written by clinicians, causing delays. Moreover, rural VA clinicians described lack of VA pharmacies in their locations, where veterans had to mail order medications, causing further delays in needed medications. Non-VA clinicians echoed these frustrations. They noted that veterans were confused about their VA pharmacy benefits as well as the need for the non-VA clinicians to follow VA formulary guidelines. Veterans expressed that it was especially challenging to physically go to the VA pharmacy to pick up medications after discharge due to lack of transportation, limited VA pharmacy hours, and long wait times. Several veterans discussed paying for their prescriptions out of pocket even though they had VA pharmacy benefits because it was more convenient to use the non-VA pharmacy. In other instances, veterans discussed going to a VA emergency department and waiting for hours to have their non-VA clinician prescription rewritten by a VA clinician (Table 2, Theme 4).

Facilitators of the Current Transition Process

Several participants provided examples of when transitional care communication between systems occurred seamlessly. VA staff and veterans noted that the VA increased the availability of urgent care appointments, which allowed for timelier postacute care follow-up appointments. Non-VA hospital clinicians also noted the availability of additional appointment slots but stated that they did not learn about these additional appointments directly from the VA. Instead, they learned of these through medical residents caring for patients at both VA and non-VA hospitals. One VA CBOC designated two nurses to care for walk-in veterans for their postdischarge follow-up needs. Some VA participants also noted that the VA Call Center Nurses occasionally called veterans upon discharge to schedule a follow-up appointment and facilitated timely care.

Participants from a VA CBOC discussed being part of a Community Transitions Consortium aimed at identifying high-utilizing patients (veteran and nonveteran) and improving communication across systems. The consortium members discussed each facility’s transition-of-care process, described having access to local non-VA hospital medical records and a backline phone number at the non-VA hospitals to coordinate transitional care. This allowed the VA clinicians to learn about non-VA hospital processes and veteran needs.

Suggestions for Improving the Transitional Care Process

VA and non-VA clinicians suggested hiring a VA liaison, preferably with a clinical background to facilitate care coordination across healthcare systems. They recommended that this person work closely with VA primary care, strengthen the relationship with non-VA hospitals, and help veterans learn more about the transition-of-care processes. Topics discussed for veteran education included how to (1) access their primary care team; (2) alert VA of non-VA hospitalization and the billing process; (3) recognize symptoms and manage care; and (4) obtain follow-up care appointments. Furthermore, they suggested that the liaison would help facilitate the transfer of medical records between VA and non-VA hospitals. Other suggestions included allowing veterans to fill prescriptions at non-VA pharmacies and dedicating a phone line for non-VA clinicians to speak to VA clinicians and staff.

Veterans agreed that improvements to the current process should include an efficient system for obtaining medications and the ability to schedule timely follow-up appointments. Furthermore, veterans wanted education about the VA transition-of-care process following a non-VA hospitalization, including payment and VA notification processes (Table 2, Theme 5).

Participants described the current transitional care process as inefficient with specific barriers that have negative consequences on patient care and clinician and staff work processes. They described difficulties in obtaining medications prescribed by non-VA clinicians from VA pharmacies, delays in follow-up appointments at the VA, and lack of bilateral communication between systems and medical record transfer. Participants also provided concrete suggestions to improving the current process, including a care coordinator with clinical background. These findings are important in the context of VA increasing veteran access to care in the community.

Despite an increasing emphasis on veteran access to non-VA care as a result of the VA strategic goals and several new programs,^7,12,13 there has not been a close examination of the current transition-of-care process from non-VA hospitals to VA primary care. Several studies have shown that the period following a hospitalization is especially vulnerable and associated with adverse events such as readmission, high cost, and death.^12,31,32 Our findings agree with previous research that identified medical record transfer across systems as one of the most challenging issues contributing to deficits in communication between care teams.³³ In addition, our study brought into focus the significant challenges faced by veterans in obtaining medications post non-VA hospital discharge. Addressing these key barriers in transitional care will improve the quality, safety, and value of healthcare in the current transition process.^38,39

Based on our findings, our participants’ concern in transitional care can be addressed in various ways. First, as veterans are increasingly receiving care in the community, identifying their veteran status early on in the non-VA hospital setting could help in improved, real time communication with the VA. This could be done by updating patient intake forms to ask patients whether they are veterans or not. Second, VA policy-level changes should work to provide veterans access to non-VA pharmacy benefits equivalent to the access patients are receiving for hospital, specialty, and outpatient care. Third, patient and provider satisfaction for dual-use veterans should be examined closely. Although participants expressed frustration with the overall transitions of care from non-VA hospitals to VA primary care setting, influence of this on the Quadruple Aim-improving patient outcomes, experience, and reducing clinician and staff burnout should be examined closely.⁴⁰ Fourth, evidence-based interventions such as nurse-led transitional care programs that have proven helpful in reducing adverse outcomes in both VA and non-VA settings will be useful to implement.^41-45 Such programs could be located in the VA, and a care coordinator role could help facilitate transitional care needs for veterans by working with multiple non-VA hospitals.

The limitations of this study are that the perspectives shared by these participants may not represent all VA and non-VA hospitals as well as veterans’ experiences with transition of care. In addition, the study was conducted in one state and the findings may not be applicable to other healthcare systems. However, our study highlighted the consistent challenges of receiving care across VA and other hospital systems. Two strengths of this study are that it was conducted by multidisciplinary research team members with expertise in qualitative research, clinical care, and implementation science and that we obtained convergent information from VA, non-VA, and veteran participants.

Our current transition-of-care process has several shortcomings. There was a clear agreement on barriers, facilitators, and suggestions for improving the current transitions-of-care process among VA and non-VA hospital participants, as well as from veterans who experienced transitions across different delivery systems. Transitioning veterans to VA primary care following a non-VA hospitalization is a crucial first step for improving care for veterans and reducing adverse outcomes such as avoidable hospital readmissions and death.

These results describe the inefficiencies experienced by patients, clinicians, and staff and their suggestions to alleviate these barriers for optimal continuum of care. To avoid frustration and inefficiencies, the increased emphasis of providing non-VA care for veterans should consider the challenges experienced in transitional care and the opportunities for increased coordination of care.

As the VA moves toward increased utilization of non-VA care, it is crucial to understand and address the challenges of transitional care faced by dual-use veterans to provide high-quality care that improves healthcare outcomes.^7,11,12 The VA implemented a shift in policy from the Veterans Access, Choice, and Accountability Act of 2014 (Public Law 113-146; “Choice Act”) to the VA Maintaining Internal Systems and Strengthening Integrated Outside Networks (MISSION) Act beginning June 6, 2019.^13,14 Under the MISSION Act, veterans have more ways to access healthcare within the VA’s network and through approved non-VA medical providers in the community known as “community care providers.”¹⁵ This shift expanded the existing VA Choice Act of 2014, where the program allowed those veterans who are unable to schedule an appointment within 30 days of their preferred date or the clinically appropriate date, or on the basis of their place of residence, to elect to receive care from eligible non-VA healthcare entities or providers.^14,15 These efforts to better serve veterans by increasing non-VA care might present added care coordination challenges for patients and their providers when they seek care in the VA.

High-quality transitional care prevents poor outcomes such as hospital readmissions.^16-18 When communication and coordination across healthcare delivery systems are lacking, patients and their families often find themselves at risk for adverse events.^19,20 Past research shows that patients have fewer adverse events when they receive comprehensive postdischarge care, including instructions on medications and self-care, symptom recognition and management, and reminders to attend follow-up appointments.^17,21,22 Although researchers have identified the components of effective transitional care,²³ barriers persist. The communication and collaboration needed to provide coordinated care across healthcare delivery systems are difficult due to the lack of standardized approaches between systems.²⁴ Consequently, follow-up care may be delayed or missed altogether. To our knowledge, there is no published research identifying transitional care challenges for clinicians, staff, and veterans in transitioning from non-VA hospitals to a VA primary care setting.

Study Design

We conducted a qualitative assessment within the VA Eastern Colorado Health Care System, an urban tertiary medical center, as well as urban and rural non-VA hospitals used by veterans. Semi-structured interview guides informed by the practical robust implementation and sustainability (PRISM) model, the Lean approach, and the Ideal Transitions of Care Bridge were used.^25-27 We explored the PRISM domains such as recipient’s characteristics, the interaction with the external environment, and the implementation and sustainability infrastructure to inform the design and implementation of the intervention.²⁵ The Lean approach included methods to optimize processes by maximizing efficiency and minimizing waste.²⁶ The Ideal Transitions of Care Bridge was used to identify the domains in transitions of care such as discharge planning, communication of information, and care coordination.²⁷

Setting and Participants

We identified the top 10 non-VA hospitals serving the most urban and rural veterans in 2015 using VA administrative data. Purposive sampling was used to ensure that urban and rural non-VA hospitals and different roles within these hospitals were represented. VA clinicians and staff were selected from the Denver VA Medical Center, a tertiary hospital within the Eastern Colorado Health Care System and one VA Community-Based Outpatient Clinic (CBOC) that primarily serves rural veterans. The Denver VA Medical Center has three clinics staffed by Patient Aligned Care Teams (PACTs), a model built on the concept of Patient-Centered Medical Home.²⁸ Hospital leadership were initially approached for permission to recruit their staff and to be involved as key informants, and all agreed. To ensure representativeness, diversity of roles was recruited, including PACT primary care physicians, nurses, and other staff members such as medical assistants and administrators. Veterans were approached for sampling if they were discharged from a non-VA hospital during June–September 2015 and used the VA for primary care. This was to ensure that they remembered the process they went through postdischarge at the time of the interview.

Data Collection and Analysis

The evaluation team members (RA, EL, and MM) conducted the interviews from November 2015 to July 2016. Clinicians, staff, and veterans were asked semi-structured questions about their experiences and their role in transitioning VA patients across systems (see Appendix for interview guides). Veterans were asked to describe their experience and satisfaction with the current postdischarge transition process. We stopped the interviews when we reached data saturation.²⁹

Interviews were audio-recorded, transcribed verbatim, and validated (transcribed interviews were double-checked against recording) to ensure data quality and accuracy. Coding was guided by a conventional content analysis technique^{30, 31} using a deductive and inductive coding approach.³¹ The deductive coding approach was drawn from the Ideal Transitions of Care Bridge and PRISM domains. ^32,33 Two evaluation team members (RA and EL) defined the initial code book by independently coding the first three interviews, worked to clarify the meanings of emergent codes, and came to a consensus when disagreements occurred. Next, a priori codes were added by team members to include the PRISM domains. These PRISM domains included the implementation and sustainability infrastructure, the external environment, the characteristics of intervention recipients, and the organizational and patient perspectives of an intervention.

Additional emergent codes were added to the code book and agreed upon by team members (RA, EL, and MM). Consistent with previously used methods, consensus building was achieved by identifying and resolving differences by discussing with team members (RA, EL, MM, CB, and RB).²⁹ Codes were examined and organized into themes by team members.^29,34-36 This process was continued until no new themes were identified. Results were reviewed by all evaluation team members to assess thoroughness and comprehensiveness.^34,35 In addition, team members triangulated the findings with VA and non-VA participants to ensure validity and reduce researcher bias.^29,37

We conducted a total of 70 interviews with 23 VA and 29 non-VA hospital clinicians and staff and 18 veterans (Table 1). Overall, we found that there was no standardized process for transitioning veterans across healthcare delivery systems. Participants reported that transitions were often inefficient when non-VA hospitals could not (1) identify patients as veterans and notify VA primary care of discharge; (2) transfer non-VA hospital medical records to VA primary care; (3) obtain follow-up care appointments with VA primary care; and (4) write VA formulary medications for veterans to fill at VA pharmacies. In addition, participants discussed about facilitators and suggestions to overcome these inefficiencies and improve transitional care (Table2). We mapped the identified barriers as well as the suggestions for improvement to the PRISM and the Ideal Transitions of Care Bridge domains (Table 3).

Unable to Identify Patients as Veterans and Notify VA Primary Care of Discharge

VA and non-VA participants reported difficulty in communicating about veterans’ hospitalizations and discharge follow-up needs across systems. Non-VA clinicians referenced difficulty in identifying patients as veterans to communicate with VA, except in instances where the VA is a payor, while VA providers described feeling largely uninformed of the veterans non-VA hospitalization. For non-VA clinicians, the lack of a systematic method for veteran identification often left them to inadvertently identify veteran status by asking about their primary care clinicians and insurance and even through an offhanded comment made by the veteran. If a veteran was identified, non-VA clinicians described being uncertain about the best way to notify VA primary care of the patient’s impending discharge. Veterans described instances of the non-VA hospital knowing their veteran status upon admission, but accounts varied on whether the non-VA hospital notified the VA primary care of their hospitalization (Table 2, Theme 1).

Unable to Transfer Non-VA Hospital Medical Records to VA Primary Care

VA clinicians discussed about the challenges associated with obtaining the veteran’s medical record from the non-VA hospitals, and when it was received, it was often incomplete information and significantly delayed. They described relying on the veteran’s description of the care received, which was not complete or accurate information needed to make clinical judgment or coordinate follow-up care. Non-VA clinicians mentioned about trying several methods for transferring the medical record to VA primary care, including discharge summary via electronic system and sometimes solely relying on patients to deliver discharge paperwork to their primary care clinicians. In instances where non-VA hospitals sent discharge paperwork to VA, there was no way for non-VA hospitals to verify whether the faxed electronic medical record was received by the VA hospital. Most of the veterans discussed receiving written postdischarge instructions to take to their VA primary care clinicians; however, they were unsure whether the VA primary care received their medical record or any other information from the non-VA hospital (Table 2, Theme 2).

Unable to Obtain Follow-Up Care Appointments with VA Primary Care

All participants described how difficult it was to obtain a follow-up appointment for veterans with VA primary care. This often resulted in delayed follow-up care. VA clinicians also shared that a non-VA hospitalization can be the impetus for a veteran to seek care at the VA for the very first time. Once eligibility is determined, the veteran is assigned a VA primary care clinician. This process may take up to six weeks, and in the meantime, the veteran is scheduled in VA urgent care for immediate postdischarge care. This lag in primary care assignment creates delayed and fragmented care (Table 2, Theme 3).

Non-VA clinicians, administrators, and staff also discussed the difficulties in scheduling follow-up care with VA primary care. Although discharge paperwork instructed patients to see their VA clinicians, there was no process in place for non-VA clinicians to confirm whether the follow-up care was received due to lack of bilateral communication. In addition, veterans discussed the inefficiencies in scheduling follow-up appointments with VA clinicians where attempts to follow-up with primary care clinicians took eight weeks or more. Several veterans described walking into the clinic without an appointment asking to be seen postdischarge or utilizing the VA emergency department for follow-up care after discharge from a non-VA hospital. Veterans admitted utilizing the VA emergency department for nonemergent reasons such as filling their prescriptions because they are unable to see a VA PCP in a timely manner (Table 2, Theme 3).

Unable to Write VA Formulary Medications for Veterans to Fill at VA Pharmacies

All participants described the difficulties in obtaining medications at VA pharmacies when prescribed by the non-VA hospital clinicians. VA clinicians often had to reassess, and rewrite prescriptions written by clinicians, causing delays. Moreover, rural VA clinicians described lack of VA pharmacies in their locations, where veterans had to mail order medications, causing further delays in needed medications. Non-VA clinicians echoed these frustrations. They noted that veterans were confused about their VA pharmacy benefits as well as the need for the non-VA clinicians to follow VA formulary guidelines. Veterans expressed that it was especially challenging to physically go to the VA pharmacy to pick up medications after discharge due to lack of transportation, limited VA pharmacy hours, and long wait times. Several veterans discussed paying for their prescriptions out of pocket even though they had VA pharmacy benefits because it was more convenient to use the non-VA pharmacy. In other instances, veterans discussed going to a VA emergency department and waiting for hours to have their non-VA clinician prescription rewritten by a VA clinician (Table 2, Theme 4).

Facilitators of the Current Transition Process

Several participants provided examples of when transitional care communication between systems occurred seamlessly. VA staff and veterans noted that the VA increased the availability of urgent care appointments, which allowed for timelier postacute care follow-up appointments. Non-VA hospital clinicians also noted the availability of additional appointment slots but stated that they did not learn about these additional appointments directly from the VA. Instead, they learned of these through medical residents caring for patients at both VA and non-VA hospitals. One VA CBOC designated two nurses to care for walk-in veterans for their postdischarge follow-up needs. Some VA participants also noted that the VA Call Center Nurses occasionally called veterans upon discharge to schedule a follow-up appointment and facilitated timely care.

Participants from a VA CBOC discussed being part of a Community Transitions Consortium aimed at identifying high-utilizing patients (veteran and nonveteran) and improving communication across systems. The consortium members discussed each facility’s transition-of-care process, described having access to local non-VA hospital medical records and a backline phone number at the non-VA hospitals to coordinate transitional care. This allowed the VA clinicians to learn about non-VA hospital processes and veteran needs.

Suggestions for Improving the Transitional Care Process

VA and non-VA clinicians suggested hiring a VA liaison, preferably with a clinical background to facilitate care coordination across healthcare systems. They recommended that this person work closely with VA primary care, strengthen the relationship with non-VA hospitals, and help veterans learn more about the transition-of-care processes. Topics discussed for veteran education included how to (1) access their primary care team; (2) alert VA of non-VA hospitalization and the billing process; (3) recognize symptoms and manage care; and (4) obtain follow-up care appointments. Furthermore, they suggested that the liaison would help facilitate the transfer of medical records between VA and non-VA hospitals. Other suggestions included allowing veterans to fill prescriptions at non-VA pharmacies and dedicating a phone line for non-VA clinicians to speak to VA clinicians and staff.

Veterans agreed that improvements to the current process should include an efficient system for obtaining medications and the ability to schedule timely follow-up appointments. Furthermore, veterans wanted education about the VA transition-of-care process following a non-VA hospitalization, including payment and VA notification processes (Table 2, Theme 5).

Participants described the current transitional care process as inefficient with specific barriers that have negative consequences on patient care and clinician and staff work processes. They described difficulties in obtaining medications prescribed by non-VA clinicians from VA pharmacies, delays in follow-up appointments at the VA, and lack of bilateral communication between systems and medical record transfer. Participants also provided concrete suggestions to improving the current process, including a care coordinator with clinical background. These findings are important in the context of VA increasing veteran access to care in the community.

Despite an increasing emphasis on veteran access to non-VA care as a result of the VA strategic goals and several new programs,^7,12,13 there has not been a close examination of the current transition-of-care process from non-VA hospitals to VA primary care. Several studies have shown that the period following a hospitalization is especially vulnerable and associated with adverse events such as readmission, high cost, and death.^12,31,32 Our findings agree with previous research that identified medical record transfer across systems as one of the most challenging issues contributing to deficits in communication between care teams.³³ In addition, our study brought into focus the significant challenges faced by veterans in obtaining medications post non-VA hospital discharge. Addressing these key barriers in transitional care will improve the quality, safety, and value of healthcare in the current transition process.^38,39

Based on our findings, our participants’ concern in transitional care can be addressed in various ways. First, as veterans are increasingly receiving care in the community, identifying their veteran status early on in the non-VA hospital setting could help in improved, real time communication with the VA. This could be done by updating patient intake forms to ask patients whether they are veterans or not. Second, VA policy-level changes should work to provide veterans access to non-VA pharmacy benefits equivalent to the access patients are receiving for hospital, specialty, and outpatient care. Third, patient and provider satisfaction for dual-use veterans should be examined closely. Although participants expressed frustration with the overall transitions of care from non-VA hospitals to VA primary care setting, influence of this on the Quadruple Aim-improving patient outcomes, experience, and reducing clinician and staff burnout should be examined closely.⁴⁰ Fourth, evidence-based interventions such as nurse-led transitional care programs that have proven helpful in reducing adverse outcomes in both VA and non-VA settings will be useful to implement.^41-45 Such programs could be located in the VA, and a care coordinator role could help facilitate transitional care needs for veterans by working with multiple non-VA hospitals.

The limitations of this study are that the perspectives shared by these participants may not represent all VA and non-VA hospitals as well as veterans’ experiences with transition of care. In addition, the study was conducted in one state and the findings may not be applicable to other healthcare systems. However, our study highlighted the consistent challenges of receiving care across VA and other hospital systems. Two strengths of this study are that it was conducted by multidisciplinary research team members with expertise in qualitative research, clinical care, and implementation science and that we obtained convergent information from VA, non-VA, and veteran participants.

Our current transition-of-care process has several shortcomings. There was a clear agreement on barriers, facilitators, and suggestions for improving the current transitions-of-care process among VA and non-VA hospital participants, as well as from veterans who experienced transitions across different delivery systems. Transitioning veterans to VA primary care following a non-VA hospitalization is a crucial first step for improving care for veterans and reducing adverse outcomes such as avoidable hospital readmissions and death.

These results describe the inefficiencies experienced by patients, clinicians, and staff and their suggestions to alleviate these barriers for optimal continuum of care. To avoid frustration and inefficiencies, the increased emphasis of providing non-VA care for veterans should consider the challenges experienced in transitional care and the opportunities for increased coordination of care.

Nearly 2.7 million veterans who rely on the Veterans Health Administration (VHA) for their health care live in rural communities.¹ Of these, more than half are aged ≥ 65 years. Rural veterans have greater rates of service-related disability and chronic medical conditions than do their urban counterparts.^1,2 Yet because of their rural location, they face unique challenges, including long travel times and distances to health care services, lack of public transportation options, and limited availability of specialized medical and social support services.

Compounding these geographic barriers is a more general lack of workforce infrastructure and a dearth of clinical health care providers (HCPs) skilled in geriatric medicine. The demand for geriatricians is projected to outpace supply and result in a national shortage of nearly 27 000 geriatricians by 2025.³ Moreover, the overwhelming majority (90%) of HCPs identifying as geriatric specialists reside in urban areas.⁴ This creates tremendous pressure on the health care system to provide remote care for older veterans contending with complex conditions, and ultimately these veterans may not receive the specialized care they need.

Telehealth modalities bridge these gaps by bringing health care to veterans in rural communities. They may also hold promise for strengthening community care in rural areas through workforce development and dissemination of educational resources. The VHA has been recognized as a leader in the field of telehealth since it began offering telehealth services to veterans in 1977^5-8 and served more than 677 000 Veterans via telehealth in fiscal year (FY) 2015.⁹ The VHA currently employs multiple modes of telehealth to increase veterans’ access to health care, including: (1) synchronous technology like clinical video telehealth (CVT), which provides live encounters between HCPs and patients using videoconferencing software; and (2) asynchronous technology, such as store-and-forward communication that offers remote transmission and clinical interpretation of veteran health data. The VHA has also strengthened its broad telehealth infrastructure by staffing VHA clinical sites with telehealth clinical technicians and providing telehealth hardware throughout.

The Department of Veterans Affairs (VA) Office of Geriatrics and Extended Care (GEC) and Office of Rural Health (ORH) established the Geriatric Research Education and Clinical Centers (GRECC) Connect project in 2014 to leverage the existing telehealth technologies at the VA to meet the health care needs of older veterans. GRECC Connect builds on the VHA network of geriatrics expertise in GRECCs by providing telehealth-based consultative support for rural primary care provider (PCP) teams, older veterans, and their families. This program profile describes this project’s mission, structure, and activities.

Program Overview

GRECC Connect leverages the clinical expertise and administrative infrastructure of participating GRECCs in order to reach clinicians and veterans in primarily rural communities.¹⁰ GRECCs are VA centers of excellence focused on aging and comprise a large network of interdisciplinary geriatrics expertise. All GRECCs have strong affiliations with local universities and are located in urban VA medical centers (VAMCs). GRECC Connect is based on a hub-and-spoke model in which urban GRECC hub sites are connected to community-based outpatient clinic (CBOC) and VAMC spokes that primarily serve veterans in other communities. CBOCs are stand-alone clinics that are geographically separate from a related VA medical center and provide outpatient primary care, mental health care services, and some specialty care services such as cardiology or neurology. They range in size from small, mainly telehealth clinics with 1 technician to large clinics with several specialty providers. Each GRECC hub site partners with an average of 6 CBOCs (range 3-16), each of which is an average distance of 92.8 miles from the related VA medical center (range 20-406 miles).

GRECC Connect was established under the umbrella of the VA Geriatric Scholars Program, which since 2008 integrates geriatrics into rural primary care practices through tailored education for continuing professional development.¹¹ Through intensive courses in geriatrics and quality improvement methods and through participation in local quality improvement projects benefiting older veterans, the Geriatric Scholars Program trains rural PCPs so that they can more effectively and independently diagnose and manage common geriatric syndromes.¹² The network of clinician scholars developed by the Geriatric Scholars Program, all rural frontline clinicians at VA clinics, has given the GRECC Connect project a well-prepared, geriatrics-trained workforce to act as project champions at rural CBOCs and VAMCs. The GRECC Connect project’s goals are to enhance access to geriatric specialty care among older veterans with complex medical problems, geriatric syndromes, and increased risk for institutionalization, and to provide geriatrics-focused educational support to rural HCP teams.

Geriatric Provider Consultations

The first overarching goal of the GRECC Connect project is to improve access to geriatrics specialty care by facilitating linkages between GRECC hub sites and the CBOCs and VAMCs that primarily serve veterans in rural communities. GRECC hub sites offer consultative support from geriatrics specialty team members (eg, geriatricians, nurse practitioners, pharmacists, gero- or neuropsychologists, registered nurses [RNs], and social workers) to rural PCP in their catchment area. This support is offered through a variety of telehealth modalities readily available in the VA (Table 1). These include CVT, in which a veteran located at a rural CBOC is seen using videoconferencing software by a geriatrics specialty provider who is located at a GRECC hub site. At some GRECC hub sites, CVT has also been used to conduct group visits between a GRECC provider at the hub site and several veterans who participate from a rural CBOC. Electronic consultations, or e-consults, involve a rural provider entering a clinical question in the VA Computerized Patient Record System. The question is then triaged, and a geriatrics provider at a GRECC responds, based on review of that veteran’s chart. At some GRECC hub sites, the e-consults are more extensive and may include telephone contact with the veteran or their caregiver.

Consultations between GRECC-based teams and rural PCPs may cover any aspect of geriatrics care, ranging from broad concerns to subspecialty areas of geriatric medicine. For instance, general geriatrics consultation may address polypharmacy, during either care transitions or ongoing care. Consultation may also reflect the specific focus area of a particular GRECC, such as cognitive assessment (eg, Pittsburgh GRECC), management of osteoporosis to address falls (eg, Durham GRECC, Miami GRECC), and continence care (eg, Birmingham/Atlanta GRECC).¹³ Most consultations are initiated by a remote HCP who is seeking geriatrics expertise from the GRECC team.

Some GRECC hub sites, however, employ case finding strategies, or detailed chart reviews, in order to identify older veterans who may benefit from geriatrics consultation. For veterans identified through those mechanisms, the GRECC clinicians suggest that the rural HCP either request or allow an e-consult or evaluation via CVT for those veterans. The geriatric consultations may help identify additional care needs for older veterans and lead to recommendations, orders, or remote provision of a variety of other actions, including VA or non-VA services (eg, home-based primary care, home nursing service, respite service, social support services such as Meals on Wheels); neuropsychological testing; physical or occupational therapy; audiology or optometry referral; falls and fracture risk assessment and interventions to reduce falls (eg, home safety evaluation, physical therapy); osteoporosis risk assessments (eg, densitometry, recommendations for pharmacologic therapy) to reduce the risk of injury or nontraumatic fractures from falls; palliative care for incontinence and hospice; and counseling on geriatric issues such as dementia caregiving, advanced directives, and driving cessation.

More recently, the Miami GRECC has begun evaluating rural veterans at risk for hypoglycemia, providing patient education and counseling about hypoglycemia, and making recommendations to the veterans’ primary care teams.¹⁴ Consultations may also lead to the appropriate use or discontinuation of medications, related to polypharmacy. GRECC-based teams, for example, have helped rural HCPs modify medication doses, start appropriate medications for dementia and depression, and identify and stop potentially inappropriate medications (eg, those that increase fall risk or that have significant anticholinergic properties).¹⁵

GRECC Connect Geriatric Case Conference Series

The second overarching goal of the GRECC Connect project is to provide geriatrics-focused educational support to equip PCPs to better serve their aging veteran patients. This is achieved through twice-monthly, case-based conferences supported by the VA Employee Education System (EES) and delivered through a webinar interface. Case conferences are targeted to members of the health care team who may provide care for rural older adults, including physicians, nurse practitioners, physician assistants, RNs, psychologists, social workers, physical and occupational therapists, and pharmacists. The format of these sessions includes a clinical case presentation, a didactic portion to enhance knowledge of participants, and an open question/answer period. The conferences focus on discussions of challenging clinical cases, addressing common problems (eg, driving concerns), and the assessment/management of geriatric syndromes (eg, cognitive decline, falls, polypharmacy). These conferences aim to improve the knowledge and skills of rural clinical teams in taking care of older veterans and to disseminate best practices in geriatric medicine, using case discussions to highlight practical applications of practices to clinical care. Recent GRECC Connect geriatric case conferences are listed in Table 2 and are recorded and archived to ensure that busy clinicians may access these trainings at the time of their choosing. These materials are catalogued and archived on the EES server.

Early Experience

GRECC Connect tracks on an annual basis the number of unique veterans served, number of participating GRECC hub sites and CBOCs, mileage from veteran homes to teleconsultation sites, and number of clinicians and staff engaged in GRECC Connect education programs.¹⁶ Since its inception in 2014, the GRECC Connect project has provided direct clinical support to more than 4000 unique veterans (eFigure), of whom half were seen for a cognition-related issue. Consultations were made on behalf of 1,622 veterans in FY 2018, of whom 60% were from rural or highly rural communities and 56.8% were served by CVT visits. The number of GRECC hub sites has increased from 4 in FY 2014 to 12 (of 20 total GRECCs) in FY 2018. The locations of current GRECC hub sites can be found on the Geriatric Scholars website: www.gerischolars.org. Through this expansion, GRECC Connect provides geriatric consultative and educational support to > 70 rural VA clinics in 10 of the 18 Veterans Integrated Service Networks (VISNs).

To assess the reduction in commute times from teleconsultation, we calculated the difference between the mileage from veteran homes to teleconsultation sites (ie, rural clinics) and the mileage from veteran homes to VAMCs where geriatric teams are located. We estimate that the 1622 veterans served in FY 2018 saved a total of 179 121 miles in travel through GRECC Connect. Veterans traveled 106 fewer miles and on average saved $58 in out-of-pocket savings (based on US General Services Administration 2018 standard mileage reimbursement rate of $0.545 per mile). However, many of the veterans have reported anecdotally that the reduction in mileage traveled was less important than the elimination of stress involved in urban navigating, driving, and parking.

More difficult to measure, GRECC Connect seeks to enhance veteran safety by reducing driving distances for older veterans whose driving abilities may be influenced by many age-related health conditions (eg, visual changes, cognitive impairment). For these and other reasons, surveyed veterans overwhelmingly reported that they would be likely to recommend teleconsultation services to other veterans, and that they preferred telemedicine consultation over traveling long distances for in-person clinical consultations.¹⁶

Since its inception in 2014, GRECC Connect has provided case-based education to a total of 2335 unique clinicians and staff. Participants have included physicians, nurse practitioners, RNs, social workers, and pharmacists. This distribution reflects the interdisciplinary nature of geriatric care. A plurality of participants (39%) were RNs. Surveyed participants in the GRECC Connect geriatrics case conference series report high overall satisfaction with the learning activity, acquisition of new knowledge and skills, and intention to apply new knowledge and skills to improve job performance.¹⁰ In addition, participants agreed that the online training platform was effective for learning and that they would recommend the education series to other HCPs.^10,16

Discussion

During its rapid 4-year scale up, GRECC Connect has established a national network and enhanced relationships between GRECC-based clinical teams and rural provider teams. In doing so, the program has begun to improve rural veterans’ access to geriatric specialty care. By providing continuing education to members of the interprofessional health care team, GRECC Connect develops rural providers’ clinical competency and promotes geriatrics skills and expertise. These activities are synergistic: Clinical support enables rural HCPs to become better at managing their own patients, while formal educational activities highlight the availability of specialized consultation available through GRECC Connect. Through ongoing creation of handbooks, workflows, and data analytic strategies, GRECC Connect aims to disseminate this model to additional GRECCs as well as other GEC programs to promote “anywhere to anywhere” VA health care.¹⁷

Barriers and Facilitators

GRECC Connect has had notable implementation challenges while new consultation relationships have been forged in order to provide geriatric expertise to rural areas where it is not otherwise available. Many GRECCs had already established connections with rural CBOCs. Among GRECCs that had previously established consultative relationships with rural clinics, the use of telehealth modalities to provide geriatric clinical resources has been a natural extension of these partnerships. GRECCs that lacked these connections, however, often had to obtain buy-in from multiple stakeholders, including rural HCPs and teams, administrative leads, and local telehealth coordinators, and they required VISN- and facility-level leadership to encourage and sustain rural team participation.

Depending on the distance of the GRECC hub-site to the CBOC, efforts to establish and sustain partnerships may require multiple contacts over time (eg, via face-to-face meetings, one-on-one outreach) and large-scale advertising of consultative services. Continuous engagement with CBOC-based teams also involves development of case finding strategies (eg, hospital discharge information, diagnoses, clinical criteria) to better identify veterans who may benefit from GRECC Connect consultation. Owing to the heterogeneity of technological resources, space, scheduling capacity, and staffing at CBOCs, GRECC sites continue to have variable engagement with their CBOC partners.

The inclusion of GRECC Connect within the Geriatric Scholars Program helps ensure that clinician scholars can serve as project champions at their respective rural sites. Rural HCPs with full-time clinical duties initially had difficulty carving out time to participate in GRECC Connect’s case-based conferences. However, the webinar platform has improved and sustained provider participation, and enduring recordings of the presentations allow clinicians to participate in the conferences at their convenience. Finally, the project experienced delays in taking certain administrative steps and hiring staff needed to support the establishment of telehealth modalities—even within a single health care system like the VA, each medical center and regional system has unique policies that complicate how telehealth modalities can be set up.

Conclusion and Future Directions

The GRECC Connect project aims to establish and support meaningful partnerships between urban geriatric specialists and rural HCPs to facilitate veterans’ increased access to geriatric specialty care. VA ORH has recognized it as a Rural Promising Practice, and GRECC Connect is currently being disseminated through an enterprise-wide initiative. Early evidence demonstrates that over 4 years, the expansion of GRECC Connect has helped meet critical aims of improving provider confidence and skills in geriatric management, and of increasing direct service provision. We have also used nationwide education platforms (eg, VA EES) to deliver geriatrics-focused education to health care teams.

Older rural veterans and their caregivers may benefit from this program through decreased travel-associated burden and report high satisfaction with these programs. Through a recently established collaboration with the GEC Data Analysis Center, we will use national data to refine our ability to identify at-risk, older rural veterans and to better evaluate their service needs and the GRECC Connect clinical impact. Because the VA is rapidly expanding use of telehealth and other virtual and digital methods to increase access to care, continued investments in telehealth are central to the VA 5-year strategic plan.¹⁸ In this spirit, GRECC Connect will continue to expand its program offerings and to leverage telehealth technologies to meet the needs of older veterans.

Acknowledgments

The authors wish to acknowledge Lisa Tenover, MD, PhD, (Palo Alto GRECC) for her contributions to this manuscript; the VA Rural Health Resource Center–Western Region; and GRECC Connect team members for their tireless work to ensure this project’s success. The GRECC Teams include Atlanta/Birmingham (Julia [Annette] Tedford, RN; Marquitta Cox, LMSW; Lisa Welch, LMSW; Mark Phillips; Lanie Walters, PharmD; Kroshona Tabb, PhD; Robert Langford, and Jason [Thomas] Sanders, HT, TCT); Bronx/NY Harbor (Ab Brody, RN; PhD, GNP-BC; Nick Koufacos, LMSW; and Shatice Jones); Canandaigua (Gary Kochersberger, MD; Suzanne Gillespie, MD; Gary Warner, PhD; Christie Hylwa, RPh CCP; Sharon Fell, LMSW; and Dorian Savino, MPA); Durham (Mamata Yanamadala, MBBS; Christy Knight, LCSW, MSW; and Julie Vognsen); Eastern Colorado (Larry Bourg, MD; Skotti Church, MD; Morgan Elmore, DO; Stephanie Hartz, LCSW; Carolyn Horney, MD; Steven Huart, AuD; Kathryn Nearing, PhD; Elizabeth O’Brien, PharmD; Laurence Robbins, MD; Robert Schwartz, MD; Karen Shea, MD; and Joleen Sussman, PhD); Little Rock (Prasad Padala, MD; and Tanya Taylor, RN); Madison (Ryan Bartkus, MD; Timothy Howell, MD; Lindsay Clark, PhD; Lauren Welch, PharmD, BCGP; Ellen Wanninger, MSW, CAPSW; Stacie Monson, RN, BSN; and Teresa Swader, MSW, LCSW); Miami (Carlos Gomez Orozo); New England (Malissa Kraft, PsyD); Palo Alto (Terri Huh, PhD, ABPP; Philip Choe, DO; Dawna Dougherty, LCSW; Ashley Scales, MPH); Pittsburgh (Stacey Shaffer, MD; Carol Dolbee, CRNP; Nancy Kovell, LCSW; Paul Bulgarelli, DO; Lauren Jost, PsyD; and Marcia Homer, RN-BC); and San Antonio (Becky Powers, MD; Che Kelly, RN, BSN; Cynthia Stewart, LCSW; Rebecca Rottman-Sagebiel, PharmD, BCPS, CGP; Melody Moris; Daniel MacCarthy; and Chen-pin Wang, PhD).

Nearly 2.7 million veterans who rely on the Veterans Health Administration (VHA) for their health care live in rural communities.¹ Of these, more than half are aged ≥ 65 years. Rural veterans have greater rates of service-related disability and chronic medical conditions than do their urban counterparts.^1,2 Yet because of their rural location, they face unique challenges, including long travel times and distances to health care services, lack of public transportation options, and limited availability of specialized medical and social support services.

Compounding these geographic barriers is a more general lack of workforce infrastructure and a dearth of clinical health care providers (HCPs) skilled in geriatric medicine. The demand for geriatricians is projected to outpace supply and result in a national shortage of nearly 27 000 geriatricians by 2025.³ Moreover, the overwhelming majority (90%) of HCPs identifying as geriatric specialists reside in urban areas.⁴ This creates tremendous pressure on the health care system to provide remote care for older veterans contending with complex conditions, and ultimately these veterans may not receive the specialized care they need.

Telehealth modalities bridge these gaps by bringing health care to veterans in rural communities. They may also hold promise for strengthening community care in rural areas through workforce development and dissemination of educational resources. The VHA has been recognized as a leader in the field of telehealth since it began offering telehealth services to veterans in 1977^5-8 and served more than 677 000 Veterans via telehealth in fiscal year (FY) 2015.⁹ The VHA currently employs multiple modes of telehealth to increase veterans’ access to health care, including: (1) synchronous technology like clinical video telehealth (CVT), which provides live encounters between HCPs and patients using videoconferencing software; and (2) asynchronous technology, such as store-and-forward communication that offers remote transmission and clinical interpretation of veteran health data. The VHA has also strengthened its broad telehealth infrastructure by staffing VHA clinical sites with telehealth clinical technicians and providing telehealth hardware throughout.

The Department of Veterans Affairs (VA) Office of Geriatrics and Extended Care (GEC) and Office of Rural Health (ORH) established the Geriatric Research Education and Clinical Centers (GRECC) Connect project in 2014 to leverage the existing telehealth technologies at the VA to meet the health care needs of older veterans. GRECC Connect builds on the VHA network of geriatrics expertise in GRECCs by providing telehealth-based consultative support for rural primary care provider (PCP) teams, older veterans, and their families. This program profile describes this project’s mission, structure, and activities.

Program Overview

GRECC Connect leverages the clinical expertise and administrative infrastructure of participating GRECCs in order to reach clinicians and veterans in primarily rural communities.¹⁰ GRECCs are VA centers of excellence focused on aging and comprise a large network of interdisciplinary geriatrics expertise. All GRECCs have strong affiliations with local universities and are located in urban VA medical centers (VAMCs). GRECC Connect is based on a hub-and-spoke model in which urban GRECC hub sites are connected to community-based outpatient clinic (CBOC) and VAMC spokes that primarily serve veterans in other communities. CBOCs are stand-alone clinics that are geographically separate from a related VA medical center and provide outpatient primary care, mental health care services, and some specialty care services such as cardiology or neurology. They range in size from small, mainly telehealth clinics with 1 technician to large clinics with several specialty providers. Each GRECC hub site partners with an average of 6 CBOCs (range 3-16), each of which is an average distance of 92.8 miles from the related VA medical center (range 20-406 miles).

GRECC Connect was established under the umbrella of the VA Geriatric Scholars Program, which since 2008 integrates geriatrics into rural primary care practices through tailored education for continuing professional development.¹¹ Through intensive courses in geriatrics and quality improvement methods and through participation in local quality improvement projects benefiting older veterans, the Geriatric Scholars Program trains rural PCPs so that they can more effectively and independently diagnose and manage common geriatric syndromes.¹² The network of clinician scholars developed by the Geriatric Scholars Program, all rural frontline clinicians at VA clinics, has given the GRECC Connect project a well-prepared, geriatrics-trained workforce to act as project champions at rural CBOCs and VAMCs. The GRECC Connect project’s goals are to enhance access to geriatric specialty care among older veterans with complex medical problems, geriatric syndromes, and increased risk for institutionalization, and to provide geriatrics-focused educational support to rural HCP teams.

Geriatric Provider Consultations

The first overarching goal of the GRECC Connect project is to improve access to geriatrics specialty care by facilitating linkages between GRECC hub sites and the CBOCs and VAMCs that primarily serve veterans in rural communities. GRECC hub sites offer consultative support from geriatrics specialty team members (eg, geriatricians, nurse practitioners, pharmacists, gero- or neuropsychologists, registered nurses [RNs], and social workers) to rural PCP in their catchment area. This support is offered through a variety of telehealth modalities readily available in the VA (Table 1). These include CVT, in which a veteran located at a rural CBOC is seen using videoconferencing software by a geriatrics specialty provider who is located at a GRECC hub site. At some GRECC hub sites, CVT has also been used to conduct group visits between a GRECC provider at the hub site and several veterans who participate from a rural CBOC. Electronic consultations, or e-consults, involve a rural provider entering a clinical question in the VA Computerized Patient Record System. The question is then triaged, and a geriatrics provider at a GRECC responds, based on review of that veteran’s chart. At some GRECC hub sites, the e-consults are more extensive and may include telephone contact with the veteran or their caregiver.

Consultations between GRECC-based teams and rural PCPs may cover any aspect of geriatrics care, ranging from broad concerns to subspecialty areas of geriatric medicine. For instance, general geriatrics consultation may address polypharmacy, during either care transitions or ongoing care. Consultation may also reflect the specific focus area of a particular GRECC, such as cognitive assessment (eg, Pittsburgh GRECC), management of osteoporosis to address falls (eg, Durham GRECC, Miami GRECC), and continence care (eg, Birmingham/Atlanta GRECC).¹³ Most consultations are initiated by a remote HCP who is seeking geriatrics expertise from the GRECC team.

Some GRECC hub sites, however, employ case finding strategies, or detailed chart reviews, in order to identify older veterans who may benefit from geriatrics consultation. For veterans identified through those mechanisms, the GRECC clinicians suggest that the rural HCP either request or allow an e-consult or evaluation via CVT for those veterans. The geriatric consultations may help identify additional care needs for older veterans and lead to recommendations, orders, or remote provision of a variety of other actions, including VA or non-VA services (eg, home-based primary care, home nursing service, respite service, social support services such as Meals on Wheels); neuropsychological testing; physical or occupational therapy; audiology or optometry referral; falls and fracture risk assessment and interventions to reduce falls (eg, home safety evaluation, physical therapy); osteoporosis risk assessments (eg, densitometry, recommendations for pharmacologic therapy) to reduce the risk of injury or nontraumatic fractures from falls; palliative care for incontinence and hospice; and counseling on geriatric issues such as dementia caregiving, advanced directives, and driving cessation.

More recently, the Miami GRECC has begun evaluating rural veterans at risk for hypoglycemia, providing patient education and counseling about hypoglycemia, and making recommendations to the veterans’ primary care teams.¹⁴ Consultations may also lead to the appropriate use or discontinuation of medications, related to polypharmacy. GRECC-based teams, for example, have helped rural HCPs modify medication doses, start appropriate medications for dementia and depression, and identify and stop potentially inappropriate medications (eg, those that increase fall risk or that have significant anticholinergic properties).¹⁵

GRECC Connect Geriatric Case Conference Series

The second overarching goal of the GRECC Connect project is to provide geriatrics-focused educational support to equip PCPs to better serve their aging veteran patients. This is achieved through twice-monthly, case-based conferences supported by the VA Employee Education System (EES) and delivered through a webinar interface. Case conferences are targeted to members of the health care team who may provide care for rural older adults, including physicians, nurse practitioners, physician assistants, RNs, psychologists, social workers, physical and occupational therapists, and pharmacists. The format of these sessions includes a clinical case presentation, a didactic portion to enhance knowledge of participants, and an open question/answer period. The conferences focus on discussions of challenging clinical cases, addressing common problems (eg, driving concerns), and the assessment/management of geriatric syndromes (eg, cognitive decline, falls, polypharmacy). These conferences aim to improve the knowledge and skills of rural clinical teams in taking care of older veterans and to disseminate best practices in geriatric medicine, using case discussions to highlight practical applications of practices to clinical care. Recent GRECC Connect geriatric case conferences are listed in Table 2 and are recorded and archived to ensure that busy clinicians may access these trainings at the time of their choosing. These materials are catalogued and archived on the EES server.

Early Experience

GRECC Connect tracks on an annual basis the number of unique veterans served, number of participating GRECC hub sites and CBOCs, mileage from veteran homes to teleconsultation sites, and number of clinicians and staff engaged in GRECC Connect education programs.¹⁶ Since its inception in 2014, the GRECC Connect project has provided direct clinical support to more than 4000 unique veterans (eFigure), of whom half were seen for a cognition-related issue. Consultations were made on behalf of 1,622 veterans in FY 2018, of whom 60% were from rural or highly rural communities and 56.8% were served by CVT visits. The number of GRECC hub sites has increased from 4 in FY 2014 to 12 (of 20 total GRECCs) in FY 2018. The locations of current GRECC hub sites can be found on the Geriatric Scholars website: www.gerischolars.org. Through this expansion, GRECC Connect provides geriatric consultative and educational support to > 70 rural VA clinics in 10 of the 18 Veterans Integrated Service Networks (VISNs).

To assess the reduction in commute times from teleconsultation, we calculated the difference between the mileage from veteran homes to teleconsultation sites (ie, rural clinics) and the mileage from veteran homes to VAMCs where geriatric teams are located. We estimate that the 1622 veterans served in FY 2018 saved a total of 179 121 miles in travel through GRECC Connect. Veterans traveled 106 fewer miles and on average saved $58 in out-of-pocket savings (based on US General Services Administration 2018 standard mileage reimbursement rate of $0.545 per mile). However, many of the veterans have reported anecdotally that the reduction in mileage traveled was less important than the elimination of stress involved in urban navigating, driving, and parking.

More difficult to measure, GRECC Connect seeks to enhance veteran safety by reducing driving distances for older veterans whose driving abilities may be influenced by many age-related health conditions (eg, visual changes, cognitive impairment). For these and other reasons, surveyed veterans overwhelmingly reported that they would be likely to recommend teleconsultation services to other veterans, and that they preferred telemedicine consultation over traveling long distances for in-person clinical consultations.¹⁶

Since its inception in 2014, GRECC Connect has provided case-based education to a total of 2335 unique clinicians and staff. Participants have included physicians, nurse practitioners, RNs, social workers, and pharmacists. This distribution reflects the interdisciplinary nature of geriatric care. A plurality of participants (39%) were RNs. Surveyed participants in the GRECC Connect geriatrics case conference series report high overall satisfaction with the learning activity, acquisition of new knowledge and skills, and intention to apply new knowledge and skills to improve job performance.¹⁰ In addition, participants agreed that the online training platform was effective for learning and that they would recommend the education series to other HCPs.^10,16

Discussion

During its rapid 4-year scale up, GRECC Connect has established a national network and enhanced relationships between GRECC-based clinical teams and rural provider teams. In doing so, the program has begun to improve rural veterans’ access to geriatric specialty care. By providing continuing education to members of the interprofessional health care team, GRECC Connect develops rural providers’ clinical competency and promotes geriatrics skills and expertise. These activities are synergistic: Clinical support enables rural HCPs to become better at managing their own patients, while formal educational activities highlight the availability of specialized consultation available through GRECC Connect. Through ongoing creation of handbooks, workflows, and data analytic strategies, GRECC Connect aims to disseminate this model to additional GRECCs as well as other GEC programs to promote “anywhere to anywhere” VA health care.¹⁷

Barriers and Facilitators

GRECC Connect has had notable implementation challenges while new consultation relationships have been forged in order to provide geriatric expertise to rural areas where it is not otherwise available. Many GRECCs had already established connections with rural CBOCs. Among GRECCs that had previously established consultative relationships with rural clinics, the use of telehealth modalities to provide geriatric clinical resources has been a natural extension of these partnerships. GRECCs that lacked these connections, however, often had to obtain buy-in from multiple stakeholders, including rural HCPs and teams, administrative leads, and local telehealth coordinators, and they required VISN- and facility-level leadership to encourage and sustain rural team participation.

Depending on the distance of the GRECC hub-site to the CBOC, efforts to establish and sustain partnerships may require multiple contacts over time (eg, via face-to-face meetings, one-on-one outreach) and large-scale advertising of consultative services. Continuous engagement with CBOC-based teams also involves development of case finding strategies (eg, hospital discharge information, diagnoses, clinical criteria) to better identify veterans who may benefit from GRECC Connect consultation. Owing to the heterogeneity of technological resources, space, scheduling capacity, and staffing at CBOCs, GRECC sites continue to have variable engagement with their CBOC partners.

The inclusion of GRECC Connect within the Geriatric Scholars Program helps ensure that clinician scholars can serve as project champions at their respective rural sites. Rural HCPs with full-time clinical duties initially had difficulty carving out time to participate in GRECC Connect’s case-based conferences. However, the webinar platform has improved and sustained provider participation, and enduring recordings of the presentations allow clinicians to participate in the conferences at their convenience. Finally, the project experienced delays in taking certain administrative steps and hiring staff needed to support the establishment of telehealth modalities—even within a single health care system like the VA, each medical center and regional system has unique policies that complicate how telehealth modalities can be set up.

Conclusion and Future Directions

The GRECC Connect project aims to establish and support meaningful partnerships between urban geriatric specialists and rural HCPs to facilitate veterans’ increased access to geriatric specialty care. VA ORH has recognized it as a Rural Promising Practice, and GRECC Connect is currently being disseminated through an enterprise-wide initiative. Early evidence demonstrates that over 4 years, the expansion of GRECC Connect has helped meet critical aims of improving provider confidence and skills in geriatric management, and of increasing direct service provision. We have also used nationwide education platforms (eg, VA EES) to deliver geriatrics-focused education to health care teams.

Older rural veterans and their caregivers may benefit from this program through decreased travel-associated burden and report high satisfaction with these programs. Through a recently established collaboration with the GEC Data Analysis Center, we will use national data to refine our ability to identify at-risk, older rural veterans and to better evaluate their service needs and the GRECC Connect clinical impact. Because the VA is rapidly expanding use of telehealth and other virtual and digital methods to increase access to care, continued investments in telehealth are central to the VA 5-year strategic plan.¹⁸ In this spirit, GRECC Connect will continue to expand its program offerings and to leverage telehealth technologies to meet the needs of older veterans.

Acknowledgments

The authors wish to acknowledge Lisa Tenover, MD, PhD, (Palo Alto GRECC) for her contributions to this manuscript; the VA Rural Health Resource Center–Western Region; and GRECC Connect team members for their tireless work to ensure this project’s success. The GRECC Teams include Atlanta/Birmingham (Julia [Annette] Tedford, RN; Marquitta Cox, LMSW; Lisa Welch, LMSW; Mark Phillips; Lanie Walters, PharmD; Kroshona Tabb, PhD; Robert Langford, and Jason [Thomas] Sanders, HT, TCT); Bronx/NY Harbor (Ab Brody, RN; PhD, GNP-BC; Nick Koufacos, LMSW; and Shatice Jones); Canandaigua (Gary Kochersberger, MD; Suzanne Gillespie, MD; Gary Warner, PhD; Christie Hylwa, RPh CCP; Sharon Fell, LMSW; and Dorian Savino, MPA); Durham (Mamata Yanamadala, MBBS; Christy Knight, LCSW, MSW; and Julie Vognsen); Eastern Colorado (Larry Bourg, MD; Skotti Church, MD; Morgan Elmore, DO; Stephanie Hartz, LCSW; Carolyn Horney, MD; Steven Huart, AuD; Kathryn Nearing, PhD; Elizabeth O’Brien, PharmD; Laurence Robbins, MD; Robert Schwartz, MD; Karen Shea, MD; and Joleen Sussman, PhD); Little Rock (Prasad Padala, MD; and Tanya Taylor, RN); Madison (Ryan Bartkus, MD; Timothy Howell, MD; Lindsay Clark, PhD; Lauren Welch, PharmD, BCGP; Ellen Wanninger, MSW, CAPSW; Stacie Monson, RN, BSN; and Teresa Swader, MSW, LCSW); Miami (Carlos Gomez Orozo); New England (Malissa Kraft, PsyD); Palo Alto (Terri Huh, PhD, ABPP; Philip Choe, DO; Dawna Dougherty, LCSW; Ashley Scales, MPH); Pittsburgh (Stacey Shaffer, MD; Carol Dolbee, CRNP; Nancy Kovell, LCSW; Paul Bulgarelli, DO; Lauren Jost, PsyD; and Marcia Homer, RN-BC); and San Antonio (Becky Powers, MD; Che Kelly, RN, BSN; Cynthia Stewart, LCSW; Rebecca Rottman-Sagebiel, PharmD, BCPS, CGP; Melody Moris; Daniel MacCarthy; and Chen-pin Wang, PhD).

Nearly 2.7 million veterans who rely on the Veterans Health Administration (VHA) for their health care live in rural communities.¹ Of these, more than half are aged ≥ 65 years. Rural veterans have greater rates of service-related disability and chronic medical conditions than do their urban counterparts.^1,2 Yet because of their rural location, they face unique challenges, including long travel times and distances to health care services, lack of public transportation options, and limited availability of specialized medical and social support services.

Compounding these geographic barriers is a more general lack of workforce infrastructure and a dearth of clinical health care providers (HCPs) skilled in geriatric medicine. The demand for geriatricians is projected to outpace supply and result in a national shortage of nearly 27 000 geriatricians by 2025.³ Moreover, the overwhelming majority (90%) of HCPs identifying as geriatric specialists reside in urban areas.⁴ This creates tremendous pressure on the health care system to provide remote care for older veterans contending with complex conditions, and ultimately these veterans may not receive the specialized care they need.

Telehealth modalities bridge these gaps by bringing health care to veterans in rural communities. They may also hold promise for strengthening community care in rural areas through workforce development and dissemination of educational resources. The VHA has been recognized as a leader in the field of telehealth since it began offering telehealth services to veterans in 1977^5-8 and served more than 677 000 Veterans via telehealth in fiscal year (FY) 2015.⁹ The VHA currently employs multiple modes of telehealth to increase veterans’ access to health care, including: (1) synchronous technology like clinical video telehealth (CVT), which provides live encounters between HCPs and patients using videoconferencing software; and (2) asynchronous technology, such as store-and-forward communication that offers remote transmission and clinical interpretation of veteran health data. The VHA has also strengthened its broad telehealth infrastructure by staffing VHA clinical sites with telehealth clinical technicians and providing telehealth hardware throughout.

The Department of Veterans Affairs (VA) Office of Geriatrics and Extended Care (GEC) and Office of Rural Health (ORH) established the Geriatric Research Education and Clinical Centers (GRECC) Connect project in 2014 to leverage the existing telehealth technologies at the VA to meet the health care needs of older veterans. GRECC Connect builds on the VHA network of geriatrics expertise in GRECCs by providing telehealth-based consultative support for rural primary care provider (PCP) teams, older veterans, and their families. This program profile describes this project’s mission, structure, and activities.

Program Overview

GRECC Connect leverages the clinical expertise and administrative infrastructure of participating GRECCs in order to reach clinicians and veterans in primarily rural communities.¹⁰ GRECCs are VA centers of excellence focused on aging and comprise a large network of interdisciplinary geriatrics expertise. All GRECCs have strong affiliations with local universities and are located in urban VA medical centers (VAMCs). GRECC Connect is based on a hub-and-spoke model in which urban GRECC hub sites are connected to community-based outpatient clinic (CBOC) and VAMC spokes that primarily serve veterans in other communities. CBOCs are stand-alone clinics that are geographically separate from a related VA medical center and provide outpatient primary care, mental health care services, and some specialty care services such as cardiology or neurology. They range in size from small, mainly telehealth clinics with 1 technician to large clinics with several specialty providers. Each GRECC hub site partners with an average of 6 CBOCs (range 3-16), each of which is an average distance of 92.8 miles from the related VA medical center (range 20-406 miles).

GRECC Connect was established under the umbrella of the VA Geriatric Scholars Program, which since 2008 integrates geriatrics into rural primary care practices through tailored education for continuing professional development.¹¹ Through intensive courses in geriatrics and quality improvement methods and through participation in local quality improvement projects benefiting older veterans, the Geriatric Scholars Program trains rural PCPs so that they can more effectively and independently diagnose and manage common geriatric syndromes.¹² The network of clinician scholars developed by the Geriatric Scholars Program, all rural frontline clinicians at VA clinics, has given the GRECC Connect project a well-prepared, geriatrics-trained workforce to act as project champions at rural CBOCs and VAMCs. The GRECC Connect project’s goals are to enhance access to geriatric specialty care among older veterans with complex medical problems, geriatric syndromes, and increased risk for institutionalization, and to provide geriatrics-focused educational support to rural HCP teams.

Geriatric Provider Consultations

The first overarching goal of the GRECC Connect project is to improve access to geriatrics specialty care by facilitating linkages between GRECC hub sites and the CBOCs and VAMCs that primarily serve veterans in rural communities. GRECC hub sites offer consultative support from geriatrics specialty team members (eg, geriatricians, nurse practitioners, pharmacists, gero- or neuropsychologists, registered nurses [RNs], and social workers) to rural PCP in their catchment area. This support is offered through a variety of telehealth modalities readily available in the VA (Table 1). These include CVT, in which a veteran located at a rural CBOC is seen using videoconferencing software by a geriatrics specialty provider who is located at a GRECC hub site. At some GRECC hub sites, CVT has also been used to conduct group visits between a GRECC provider at the hub site and several veterans who participate from a rural CBOC. Electronic consultations, or e-consults, involve a rural provider entering a clinical question in the VA Computerized Patient Record System. The question is then triaged, and a geriatrics provider at a GRECC responds, based on review of that veteran’s chart. At some GRECC hub sites, the e-consults are more extensive and may include telephone contact with the veteran or their caregiver.

Consultations between GRECC-based teams and rural PCPs may cover any aspect of geriatrics care, ranging from broad concerns to subspecialty areas of geriatric medicine. For instance, general geriatrics consultation may address polypharmacy, during either care transitions or ongoing care. Consultation may also reflect the specific focus area of a particular GRECC, such as cognitive assessment (eg, Pittsburgh GRECC), management of osteoporosis to address falls (eg, Durham GRECC, Miami GRECC), and continence care (eg, Birmingham/Atlanta GRECC).¹³ Most consultations are initiated by a remote HCP who is seeking geriatrics expertise from the GRECC team.

Some GRECC hub sites, however, employ case finding strategies, or detailed chart reviews, in order to identify older veterans who may benefit from geriatrics consultation. For veterans identified through those mechanisms, the GRECC clinicians suggest that the rural HCP either request or allow an e-consult or evaluation via CVT for those veterans. The geriatric consultations may help identify additional care needs for older veterans and lead to recommendations, orders, or remote provision of a variety of other actions, including VA or non-VA services (eg, home-based primary care, home nursing service, respite service, social support services such as Meals on Wheels); neuropsychological testing; physical or occupational therapy; audiology or optometry referral; falls and fracture risk assessment and interventions to reduce falls (eg, home safety evaluation, physical therapy); osteoporosis risk assessments (eg, densitometry, recommendations for pharmacologic therapy) to reduce the risk of injury or nontraumatic fractures from falls; palliative care for incontinence and hospice; and counseling on geriatric issues such as dementia caregiving, advanced directives, and driving cessation.

More recently, the Miami GRECC has begun evaluating rural veterans at risk for hypoglycemia, providing patient education and counseling about hypoglycemia, and making recommendations to the veterans’ primary care teams.¹⁴ Consultations may also lead to the appropriate use or discontinuation of medications, related to polypharmacy. GRECC-based teams, for example, have helped rural HCPs modify medication doses, start appropriate medications for dementia and depression, and identify and stop potentially inappropriate medications (eg, those that increase fall risk or that have significant anticholinergic properties).¹⁵

GRECC Connect Geriatric Case Conference Series

The second overarching goal of the GRECC Connect project is to provide geriatrics-focused educational support to equip PCPs to better serve their aging veteran patients. This is achieved through twice-monthly, case-based conferences supported by the VA Employee Education System (EES) and delivered through a webinar interface. Case conferences are targeted to members of the health care team who may provide care for rural older adults, including physicians, nurse practitioners, physician assistants, RNs, psychologists, social workers, physical and occupational therapists, and pharmacists. The format of these sessions includes a clinical case presentation, a didactic portion to enhance knowledge of participants, and an open question/answer period. The conferences focus on discussions of challenging clinical cases, addressing common problems (eg, driving concerns), and the assessment/management of geriatric syndromes (eg, cognitive decline, falls, polypharmacy). These conferences aim to improve the knowledge and skills of rural clinical teams in taking care of older veterans and to disseminate best practices in geriatric medicine, using case discussions to highlight practical applications of practices to clinical care. Recent GRECC Connect geriatric case conferences are listed in Table 2 and are recorded and archived to ensure that busy clinicians may access these trainings at the time of their choosing. These materials are catalogued and archived on the EES server.

Early Experience

GRECC Connect tracks on an annual basis the number of unique veterans served, number of participating GRECC hub sites and CBOCs, mileage from veteran homes to teleconsultation sites, and number of clinicians and staff engaged in GRECC Connect education programs.¹⁶ Since its inception in 2014, the GRECC Connect project has provided direct clinical support to more than 4000 unique veterans (eFigure), of whom half were seen for a cognition-related issue. Consultations were made on behalf of 1,622 veterans in FY 2018, of whom 60% were from rural or highly rural communities and 56.8% were served by CVT visits. The number of GRECC hub sites has increased from 4 in FY 2014 to 12 (of 20 total GRECCs) in FY 2018. The locations of current GRECC hub sites can be found on the Geriatric Scholars website: www.gerischolars.org. Through this expansion, GRECC Connect provides geriatric consultative and educational support to > 70 rural VA clinics in 10 of the 18 Veterans Integrated Service Networks (VISNs).

To assess the reduction in commute times from teleconsultation, we calculated the difference between the mileage from veteran homes to teleconsultation sites (ie, rural clinics) and the mileage from veteran homes to VAMCs where geriatric teams are located. We estimate that the 1622 veterans served in FY 2018 saved a total of 179 121 miles in travel through GRECC Connect. Veterans traveled 106 fewer miles and on average saved $58 in out-of-pocket savings (based on US General Services Administration 2018 standard mileage reimbursement rate of $0.545 per mile). However, many of the veterans have reported anecdotally that the reduction in mileage traveled was less important than the elimination of stress involved in urban navigating, driving, and parking.

More difficult to measure, GRECC Connect seeks to enhance veteran safety by reducing driving distances for older veterans whose driving abilities may be influenced by many age-related health conditions (eg, visual changes, cognitive impairment). For these and other reasons, surveyed veterans overwhelmingly reported that they would be likely to recommend teleconsultation services to other veterans, and that they preferred telemedicine consultation over traveling long distances for in-person clinical consultations.¹⁶

Since its inception in 2014, GRECC Connect has provided case-based education to a total of 2335 unique clinicians and staff. Participants have included physicians, nurse practitioners, RNs, social workers, and pharmacists. This distribution reflects the interdisciplinary nature of geriatric care. A plurality of participants (39%) were RNs. Surveyed participants in the GRECC Connect geriatrics case conference series report high overall satisfaction with the learning activity, acquisition of new knowledge and skills, and intention to apply new knowledge and skills to improve job performance.¹⁰ In addition, participants agreed that the online training platform was effective for learning and that they would recommend the education series to other HCPs.^10,16

Discussion

During its rapid 4-year scale up, GRECC Connect has established a national network and enhanced relationships between GRECC-based clinical teams and rural provider teams. In doing so, the program has begun to improve rural veterans’ access to geriatric specialty care. By providing continuing education to members of the interprofessional health care team, GRECC Connect develops rural providers’ clinical competency and promotes geriatrics skills and expertise. These activities are synergistic: Clinical support enables rural HCPs to become better at managing their own patients, while formal educational activities highlight the availability of specialized consultation available through GRECC Connect. Through ongoing creation of handbooks, workflows, and data analytic strategies, GRECC Connect aims to disseminate this model to additional GRECCs as well as other GEC programs to promote “anywhere to anywhere” VA health care.¹⁷

Barriers and Facilitators

GRECC Connect has had notable implementation challenges while new consultation relationships have been forged in order to provide geriatric expertise to rural areas where it is not otherwise available. Many GRECCs had already established connections with rural CBOCs. Among GRECCs that had previously established consultative relationships with rural clinics, the use of telehealth modalities to provide geriatric clinical resources has been a natural extension of these partnerships. GRECCs that lacked these connections, however, often had to obtain buy-in from multiple stakeholders, including rural HCPs and teams, administrative leads, and local telehealth coordinators, and they required VISN- and facility-level leadership to encourage and sustain rural team participation.

Depending on the distance of the GRECC hub-site to the CBOC, efforts to establish and sustain partnerships may require multiple contacts over time (eg, via face-to-face meetings, one-on-one outreach) and large-scale advertising of consultative services. Continuous engagement with CBOC-based teams also involves development of case finding strategies (eg, hospital discharge information, diagnoses, clinical criteria) to better identify veterans who may benefit from GRECC Connect consultation. Owing to the heterogeneity of technological resources, space, scheduling capacity, and staffing at CBOCs, GRECC sites continue to have variable engagement with their CBOC partners.

The inclusion of GRECC Connect within the Geriatric Scholars Program helps ensure that clinician scholars can serve as project champions at their respective rural sites. Rural HCPs with full-time clinical duties initially had difficulty carving out time to participate in GRECC Connect’s case-based conferences. However, the webinar platform has improved and sustained provider participation, and enduring recordings of the presentations allow clinicians to participate in the conferences at their convenience. Finally, the project experienced delays in taking certain administrative steps and hiring staff needed to support the establishment of telehealth modalities—even within a single health care system like the VA, each medical center and regional system has unique policies that complicate how telehealth modalities can be set up.

Conclusion and Future Directions

The GRECC Connect project aims to establish and support meaningful partnerships between urban geriatric specialists and rural HCPs to facilitate veterans’ increased access to geriatric specialty care. VA ORH has recognized it as a Rural Promising Practice, and GRECC Connect is currently being disseminated through an enterprise-wide initiative. Early evidence demonstrates that over 4 years, the expansion of GRECC Connect has helped meet critical aims of improving provider confidence and skills in geriatric management, and of increasing direct service provision. We have also used nationwide education platforms (eg, VA EES) to deliver geriatrics-focused education to health care teams.

Older rural veterans and their caregivers may benefit from this program through decreased travel-associated burden and report high satisfaction with these programs. Through a recently established collaboration with the GEC Data Analysis Center, we will use national data to refine our ability to identify at-risk, older rural veterans and to better evaluate their service needs and the GRECC Connect clinical impact. Because the VA is rapidly expanding use of telehealth and other virtual and digital methods to increase access to care, continued investments in telehealth are central to the VA 5-year strategic plan.¹⁸ In this spirit, GRECC Connect will continue to expand its program offerings and to leverage telehealth technologies to meet the needs of older veterans.

Acknowledgments

The authors wish to acknowledge Lisa Tenover, MD, PhD, (Palo Alto GRECC) for her contributions to this manuscript; the VA Rural Health Resource Center–Western Region; and GRECC Connect team members for their tireless work to ensure this project’s success. The GRECC Teams include Atlanta/Birmingham (Julia [Annette] Tedford, RN; Marquitta Cox, LMSW; Lisa Welch, LMSW; Mark Phillips; Lanie Walters, PharmD; Kroshona Tabb, PhD; Robert Langford, and Jason [Thomas] Sanders, HT, TCT); Bronx/NY Harbor (Ab Brody, RN; PhD, GNP-BC; Nick Koufacos, LMSW; and Shatice Jones); Canandaigua (Gary Kochersberger, MD; Suzanne Gillespie, MD; Gary Warner, PhD; Christie Hylwa, RPh CCP; Sharon Fell, LMSW; and Dorian Savino, MPA); Durham (Mamata Yanamadala, MBBS; Christy Knight, LCSW, MSW; and Julie Vognsen); Eastern Colorado (Larry Bourg, MD; Skotti Church, MD; Morgan Elmore, DO; Stephanie Hartz, LCSW; Carolyn Horney, MD; Steven Huart, AuD; Kathryn Nearing, PhD; Elizabeth O’Brien, PharmD; Laurence Robbins, MD; Robert Schwartz, MD; Karen Shea, MD; and Joleen Sussman, PhD); Little Rock (Prasad Padala, MD; and Tanya Taylor, RN); Madison (Ryan Bartkus, MD; Timothy Howell, MD; Lindsay Clark, PhD; Lauren Welch, PharmD, BCGP; Ellen Wanninger, MSW, CAPSW; Stacie Monson, RN, BSN; and Teresa Swader, MSW, LCSW); Miami (Carlos Gomez Orozo); New England (Malissa Kraft, PsyD); Palo Alto (Terri Huh, PhD, ABPP; Philip Choe, DO; Dawna Dougherty, LCSW; Ashley Scales, MPH); Pittsburgh (Stacey Shaffer, MD; Carol Dolbee, CRNP; Nancy Kovell, LCSW; Paul Bulgarelli, DO; Lauren Jost, PsyD; and Marcia Homer, RN-BC); and San Antonio (Becky Powers, MD; Che Kelly, RN, BSN; Cynthia Stewart, LCSW; Rebecca Rottman-Sagebiel, PharmD, BCPS, CGP; Melody Moris; Daniel MacCarthy; and Chen-pin Wang, PhD).

In the current health care environment, hospitals are constantly challenged to improve quality metrics and deliver better health care outcomes. One means to achieving quality improvement is through the use of order sets, groups of related orders that a health care provider (HCP) can place with either a few keystrokes or mouse clicks.¹

Historically, design of order sets has largely focused on clicking checkboxes containing evidence-based practices. According to Bates and colleagues and the Institute for Safe Medication Practices, incorporating evidence-based medicine (EBM) into order sets is not by itself sufficient.^2,3Execution of proper design coupled with simplicity and provider efficiency is paramount to HCP buy-in, increased likelihood of order set adherence, and to potentially better outcomes.

In this article, we outline advancements in order set design. These improvements increase provider efficiency and ease of use; incorporate human factors engineering (HFE); apply failure mode and effects analysis; and include EBM.

Methods

An inpatient nicotine replacement therapy (NRT) order was developed as part of a multifaceted solution to improve tobacco cessation care at the James A. Haley Veterans’ Hospital (JAHVH) in Tampa, Florida, a complexity level 1a facility. This NRT order set used the 4-step order set design framework the authors’ developed (for additional information about the NRT order set, contact the authors). We distinguish order set design technique between 2 different inpatient NRT order sets. The first order set in the comparison (Figure 1) is an inpatient NRT order set of unknown origin—it is common for US Department of Veterans Affairs (VA) medical facilities to share order sets and other resources. The second order set (Figure 2) is an inpatient NRT order set we designed using our 4-step process for comparison in this article. No institutional review board approval was required as this work met criteria for operational improvement activities exempt from ethics review.

Justin Iannello, DO, MBA, was the team leader and developer of the 4-step order set design technique. The intervention team consisted of 4 internal medicine physicians with expertise in quality improvement and patient safety: 1 certified professional in patient safety and certified as a Lean Six Sigma Black Belt; 2 physicians certified as Lean Six Sigma Black Belts; and 1 physician certified as a Lean Six Sigma Green Belt. Two inpatient clinical pharmacists and 1 quality management specialist also were involved in its development.

Development of a new NRT order set was felt to be an integral part of the tobacco cessation care delivery process. An NRT order set perceived by users as value-added required a solution that merged EBM with standardization and applied quality improvement principles. The result was an approach to order set design that focused on 4 key questions: Is the order set efficient and easy to use/navigate? Is human factors engineering incorporated? Is failure mode and effects analysis applied? Are evidence-based practices included?

Ease of Use and Navigation

Implementing an order set that is efficient and easy to use or navigate seems straightforward but can be difficult to execute. Figure 1 shows many detailed options consisting of different combinations of nicotine patches, lozenges, and gum. Also included are oral tobacco cessation options (bupropion and varenicline). Although more options may seem better, confusion about appropriate medication selection can occur.

According to Heath and Heath, too many options can result in lack of action.⁴ For example, Heath and Heath discuss a food store that offered 6 free samples of different jams on one day and 24 jams the following day. The customers who sampled 6 different types of jam were 10 times more likely to buy jam. The authors concluded that the more options available, the more difficulty a potential buyer has in deciding on a course of action.⁴

In clinical situations where a HCP is using an order set, the number of options can mean the difference between use vs avoidance if the choices are overwhelming. HCPs process layers of detail every day when creating differential diagnoses and treatment plans. While that level of detail is necessary clinically, that same level of detail included in orders sets can create challenges for HCPs.

Figure 2 advances the order set in Figure 1 by providing a simpler and cleaner design, so HCPs can more easily review and process the information. This order set design minimizes the number of options available to help users make the right decision, focusing on value for the appropriate setting and audience. In other words, order sets should not be a “one size fits all” approach.

Order sets should be tailored to the appropriate clinical setting (eg, inpatient acute care, outpatient clinic setting, etc) and HCP (eg, hospitalist, tobacco cessation specialist, etc). We are comparing NRT order sets designed for HCPs who do not routinely prescribe oral tobacco cessation products in the inpatient setting. When possible, autogenerated bundle orders should also be used according to evidence-based recommendations (such as nicotine patch tapers) for ease of use and further simplification of order sets.

Finally, usability testing known as “evaluating a product or service by testing it with representative users” helps further refine an order set.⁵Usability testing should be applied during all phases of order set development with end user(s) as it helps identify problems with order set design prior to implementation. By applying usability testing, the order set becomes more meaningful and valued by the user.

Human Factors Engineering

HFE is “the study of all the factors that make it easier to do the work in the right way.”⁶ HFE seeks to identify, align, and apply processes for people and the world within which they live and work to promote safe and efficient practices, especially in relation to the technology and physical design features in their work environment.⁶

The average American adult makes about 35,000 decisions per day.⁷ Thus, there is potential for error at any moment. Design that does not take HFE into account can be dangerous. For example, when tube feed and IV line connectors look similar and are compatible, patients may inadvertently receive food administered directly into their bloodstream.⁸

HFE can and should be applied to order sets. Everything from the look, feel, and verbiage of an order set affects potential outcomes. For example, consider the impact even seemingly minor modifications can have on outcomes simply by guiding users in a different way: Figure 1 provides NRT options based on cigarette use per day, whereas Figure 2 conveys pack use per day in relation to the equivalent number of cigarettes used daily. These differences may seem small; however, it helps guide users to the right choice when considering that health care providers have been historically trained on social history gathering that emphasizes packs per day and pack-years.

Failure Mode and Effects Analysis

Failure mode and effects analysis (FMEA) is “a structured way to identify and address potential problems, or failures and their resulting effects on the system or process before an adverse event occurs.”⁹ The benefit of an order set must be weighed against the risk during development. FMEA should be applied during order set design to assess and limit risk just as with any other clinical care process.

FMEA examines both level of risk and frequency of risk occurrence associated with a new proposed process. For example, let’s evaluate an order set designed for pain control after surgery that consists of multiple high-risk opioids along with antihistamine medications for as-needed itch relief (a non-life-threatening adverse event (AE) of opioids well known by the medical community). An interdisciplinary FMEA team consisting of subject matter experts may examine how the process should flow in step-by-step detail and then discuss the benefit of a process and risk for potential error. A FMEA team would then analyze what could go wrong with each part of the process and assign a level of risk and risk frequency for various steps in the process, and then decide that certain steps should be modified or eliminated. Perhaps after FMEA, a facility might conclude that the risk of serious complications is high when you combine opioid use with antihistamine medications. The facility could decide to remove antihistamine medications from an order set if it is determined that risks outweigh benefits. While a root cause analysis might identify the cause of an AE after order set use, these situations can be prevented with FMEA.

When applying FMEA to Figure 1, while bupropion is known as an evidence-based oral tobacco cessation option, there is the possibility that bupropion could be inadvertently prescribed from the order set in a hospitalized patient with alcohol withdrawal and withdrawal seizure history. These potentially dangerous situations can be avoided with FMEA. Thus, although bupropion may be evidence-based for NRT, decisions regarding order set design using EBM alone are insufficient.

The practitioner must consider possible unintended consequences within order sets and target treatment options to the appropriate setting and audience. Although Figure 1 may appear to be more inclusive, the interdisciplinary committee designing the inpatient NRT order set felt there was heightened risk with introducing bupropion in Figure 1 and decided the risk would be lowered by removing bupropion from the redesigned NRT order set (Figure 2). In addition to the goal of balancing availability of NRT options with acceptable risk, Figure 2 also focused on building an NRT order set most applicable to the inpatient setting.

Including Evidence-Based Practices

EBM has become a routine part of clinical decision making. Therefore, including EBM in order set design is vital. EBM for NRT has demonstrated that combination therapy is more effective than is monotherapy to help tobacco users quit. Incremental doses of NRT are recommended for patients who use tobacco more frequently.¹⁰

As shown in Figures 1 and 2, both order set designs incorporate EBM for NRT. Although the importance of implementing EBM is evident, critical factors, such as HFE and FMEA make a difference with well-designed order sets.

Results

The 4-step order set design technique was used during development of an inpatient NRT order set at the JAHVH. Results for the inpatient Joint Commission Tobacco Treatment Measures were obtained from the Veterans Health Administration quality metric reporting system known as Strategic Analytics for Improvement and Learning (SAIL). SAIL performance measure outcomes, which include the inpatient Joint Commission Tobacco Treatment Measures, are derived from chart reviews conducted by the External Peer Review Program. Outcomes demonstrated that TOB-2 and TOB-3 (2 inpatient Joint Commission Tobacco Treatment Measures) known as tob20 and tob40, respectively, within SAIL improved by more than 300% after development of an NRT order set using the 4-step order set design framework along with implementation of a multifaceted tobacco cessation care delivery system at JAHVH.

Discussion

While the overall tobacco cessation care delivery system contributed to improved outcomes with the inpatient Joint Commission Tobacco Treatment Measures at JAHVH, the NRT order set was a cornerstone of the design. Although using our order set design technique does not necessarily guarantee successful outcomes, we believe using the 4-step order set design process increases the value of order sets and has potential to improve quality outcomes.

Limitations

Although improved outcomes following implementation of our NRT order set suggest correlation, causation cannot be proven. Also while the NRT order set is believed to have helped tremendously with outcomes, the entire tobacco cessation care delivery system at JAHVH contributed to the results. In addition, the inpatient Joint Commission Tobacco Treatment Measures help improve processes for tobacco cessation care. However, we are uncertain whether the results of our improvement efforts helped patients stop tobacco use. Further studies are needed to determine impact on population health. Finally, our results were based on improvement work done at a single center. Further studies are necessary to see whether results are reproducible.

Conclusion

There was significant improvement with the inpatient Joint Commission Tobacco Treatment Measures outcomes following development of a tobacco cessation care delivery system that included design of an inpatient NRT order set using a 4-step process we developed. This 4-step structure includes emphasis on efficiency and ease of use; human factors engineering; failure mode and effects analysis; and incorporation of evidence-based medicine (Box.) Postimplementation results showed improvement of the inpatient Joint Commission Tobacco Treatment Measures by greater than 3-fold at a single hospital.

The next steps for this initiative include testing the 4-step order set design process in multiple clinical settings to determine the effectiveness of this approach in other areas of clinical care.

In the current health care environment, hospitals are constantly challenged to improve quality metrics and deliver better health care outcomes. One means to achieving quality improvement is through the use of order sets, groups of related orders that a health care provider (HCP) can place with either a few keystrokes or mouse clicks.¹

Historically, design of order sets has largely focused on clicking checkboxes containing evidence-based practices. According to Bates and colleagues and the Institute for Safe Medication Practices, incorporating evidence-based medicine (EBM) into order sets is not by itself sufficient.^2,3Execution of proper design coupled with simplicity and provider efficiency is paramount to HCP buy-in, increased likelihood of order set adherence, and to potentially better outcomes.

In this article, we outline advancements in order set design. These improvements increase provider efficiency and ease of use; incorporate human factors engineering (HFE); apply failure mode and effects analysis; and include EBM.

Methods

An inpatient nicotine replacement therapy (NRT) order was developed as part of a multifaceted solution to improve tobacco cessation care at the James A. Haley Veterans’ Hospital (JAHVH) in Tampa, Florida, a complexity level 1a facility. This NRT order set used the 4-step order set design framework the authors’ developed (for additional information about the NRT order set, contact the authors). We distinguish order set design technique between 2 different inpatient NRT order sets. The first order set in the comparison (Figure 1) is an inpatient NRT order set of unknown origin—it is common for US Department of Veterans Affairs (VA) medical facilities to share order sets and other resources. The second order set (Figure 2) is an inpatient NRT order set we designed using our 4-step process for comparison in this article. No institutional review board approval was required as this work met criteria for operational improvement activities exempt from ethics review.

Justin Iannello, DO, MBA, was the team leader and developer of the 4-step order set design technique. The intervention team consisted of 4 internal medicine physicians with expertise in quality improvement and patient safety: 1 certified professional in patient safety and certified as a Lean Six Sigma Black Belt; 2 physicians certified as Lean Six Sigma Black Belts; and 1 physician certified as a Lean Six Sigma Green Belt. Two inpatient clinical pharmacists and 1 quality management specialist also were involved in its development.

Development of a new NRT order set was felt to be an integral part of the tobacco cessation care delivery process. An NRT order set perceived by users as value-added required a solution that merged EBM with standardization and applied quality improvement principles. The result was an approach to order set design that focused on 4 key questions: Is the order set efficient and easy to use/navigate? Is human factors engineering incorporated? Is failure mode and effects analysis applied? Are evidence-based practices included?

Ease of Use and Navigation

Implementing an order set that is efficient and easy to use or navigate seems straightforward but can be difficult to execute. Figure 1 shows many detailed options consisting of different combinations of nicotine patches, lozenges, and gum. Also included are oral tobacco cessation options (bupropion and varenicline). Although more options may seem better, confusion about appropriate medication selection can occur.

According to Heath and Heath, too many options can result in lack of action.⁴ For example, Heath and Heath discuss a food store that offered 6 free samples of different jams on one day and 24 jams the following day. The customers who sampled 6 different types of jam were 10 times more likely to buy jam. The authors concluded that the more options available, the more difficulty a potential buyer has in deciding on a course of action.⁴

In clinical situations where a HCP is using an order set, the number of options can mean the difference between use vs avoidance if the choices are overwhelming. HCPs process layers of detail every day when creating differential diagnoses and treatment plans. While that level of detail is necessary clinically, that same level of detail included in orders sets can create challenges for HCPs.

Figure 2 advances the order set in Figure 1 by providing a simpler and cleaner design, so HCPs can more easily review and process the information. This order set design minimizes the number of options available to help users make the right decision, focusing on value for the appropriate setting and audience. In other words, order sets should not be a “one size fits all” approach.

Order sets should be tailored to the appropriate clinical setting (eg, inpatient acute care, outpatient clinic setting, etc) and HCP (eg, hospitalist, tobacco cessation specialist, etc). We are comparing NRT order sets designed for HCPs who do not routinely prescribe oral tobacco cessation products in the inpatient setting. When possible, autogenerated bundle orders should also be used according to evidence-based recommendations (such as nicotine patch tapers) for ease of use and further simplification of order sets.

Finally, usability testing known as “evaluating a product or service by testing it with representative users” helps further refine an order set.⁵Usability testing should be applied during all phases of order set development with end user(s) as it helps identify problems with order set design prior to implementation. By applying usability testing, the order set becomes more meaningful and valued by the user.

Human Factors Engineering

HFE is “the study of all the factors that make it easier to do the work in the right way.”⁶ HFE seeks to identify, align, and apply processes for people and the world within which they live and work to promote safe and efficient practices, especially in relation to the technology and physical design features in their work environment.⁶

The average American adult makes about 35,000 decisions per day.⁷ Thus, there is potential for error at any moment. Design that does not take HFE into account can be dangerous. For example, when tube feed and IV line connectors look similar and are compatible, patients may inadvertently receive food administered directly into their bloodstream.⁸

HFE can and should be applied to order sets. Everything from the look, feel, and verbiage of an order set affects potential outcomes. For example, consider the impact even seemingly minor modifications can have on outcomes simply by guiding users in a different way: Figure 1 provides NRT options based on cigarette use per day, whereas Figure 2 conveys pack use per day in relation to the equivalent number of cigarettes used daily. These differences may seem small; however, it helps guide users to the right choice when considering that health care providers have been historically trained on social history gathering that emphasizes packs per day and pack-years.

Failure Mode and Effects Analysis

Failure mode and effects analysis (FMEA) is “a structured way to identify and address potential problems, or failures and their resulting effects on the system or process before an adverse event occurs.”⁹ The benefit of an order set must be weighed against the risk during development. FMEA should be applied during order set design to assess and limit risk just as with any other clinical care process.

FMEA examines both level of risk and frequency of risk occurrence associated with a new proposed process. For example, let’s evaluate an order set designed for pain control after surgery that consists of multiple high-risk opioids along with antihistamine medications for as-needed itch relief (a non-life-threatening adverse event (AE) of opioids well known by the medical community). An interdisciplinary FMEA team consisting of subject matter experts may examine how the process should flow in step-by-step detail and then discuss the benefit of a process and risk for potential error. A FMEA team would then analyze what could go wrong with each part of the process and assign a level of risk and risk frequency for various steps in the process, and then decide that certain steps should be modified or eliminated. Perhaps after FMEA, a facility might conclude that the risk of serious complications is high when you combine opioid use with antihistamine medications. The facility could decide to remove antihistamine medications from an order set if it is determined that risks outweigh benefits. While a root cause analysis might identify the cause of an AE after order set use, these situations can be prevented with FMEA.

When applying FMEA to Figure 1, while bupropion is known as an evidence-based oral tobacco cessation option, there is the possibility that bupropion could be inadvertently prescribed from the order set in a hospitalized patient with alcohol withdrawal and withdrawal seizure history. These potentially dangerous situations can be avoided with FMEA. Thus, although bupropion may be evidence-based for NRT, decisions regarding order set design using EBM alone are insufficient.

The practitioner must consider possible unintended consequences within order sets and target treatment options to the appropriate setting and audience. Although Figure 1 may appear to be more inclusive, the interdisciplinary committee designing the inpatient NRT order set felt there was heightened risk with introducing bupropion in Figure 1 and decided the risk would be lowered by removing bupropion from the redesigned NRT order set (Figure 2). In addition to the goal of balancing availability of NRT options with acceptable risk, Figure 2 also focused on building an NRT order set most applicable to the inpatient setting.

Including Evidence-Based Practices

EBM has become a routine part of clinical decision making. Therefore, including EBM in order set design is vital. EBM for NRT has demonstrated that combination therapy is more effective than is monotherapy to help tobacco users quit. Incremental doses of NRT are recommended for patients who use tobacco more frequently.¹⁰

As shown in Figures 1 and 2, both order set designs incorporate EBM for NRT. Although the importance of implementing EBM is evident, critical factors, such as HFE and FMEA make a difference with well-designed order sets.

Results

The 4-step order set design technique was used during development of an inpatient NRT order set at the JAHVH. Results for the inpatient Joint Commission Tobacco Treatment Measures were obtained from the Veterans Health Administration quality metric reporting system known as Strategic Analytics for Improvement and Learning (SAIL). SAIL performance measure outcomes, which include the inpatient Joint Commission Tobacco Treatment Measures, are derived from chart reviews conducted by the External Peer Review Program. Outcomes demonstrated that TOB-2 and TOB-3 (2 inpatient Joint Commission Tobacco Treatment Measures) known as tob20 and tob40, respectively, within SAIL improved by more than 300% after development of an NRT order set using the 4-step order set design framework along with implementation of a multifaceted tobacco cessation care delivery system at JAHVH.

Discussion

While the overall tobacco cessation care delivery system contributed to improved outcomes with the inpatient Joint Commission Tobacco Treatment Measures at JAHVH, the NRT order set was a cornerstone of the design. Although using our order set design technique does not necessarily guarantee successful outcomes, we believe using the 4-step order set design process increases the value of order sets and has potential to improve quality outcomes.

Limitations

Although improved outcomes following implementation of our NRT order set suggest correlation, causation cannot be proven. Also while the NRT order set is believed to have helped tremendously with outcomes, the entire tobacco cessation care delivery system at JAHVH contributed to the results. In addition, the inpatient Joint Commission Tobacco Treatment Measures help improve processes for tobacco cessation care. However, we are uncertain whether the results of our improvement efforts helped patients stop tobacco use. Further studies are needed to determine impact on population health. Finally, our results were based on improvement work done at a single center. Further studies are necessary to see whether results are reproducible.

Conclusion

There was significant improvement with the inpatient Joint Commission Tobacco Treatment Measures outcomes following development of a tobacco cessation care delivery system that included design of an inpatient NRT order set using a 4-step process we developed. This 4-step structure includes emphasis on efficiency and ease of use; human factors engineering; failure mode and effects analysis; and incorporation of evidence-based medicine (Box.) Postimplementation results showed improvement of the inpatient Joint Commission Tobacco Treatment Measures by greater than 3-fold at a single hospital.

The next steps for this initiative include testing the 4-step order set design process in multiple clinical settings to determine the effectiveness of this approach in other areas of clinical care.

In the current health care environment, hospitals are constantly challenged to improve quality metrics and deliver better health care outcomes. One means to achieving quality improvement is through the use of order sets, groups of related orders that a health care provider (HCP) can place with either a few keystrokes or mouse clicks.¹

Historically, design of order sets has largely focused on clicking checkboxes containing evidence-based practices. According to Bates and colleagues and the Institute for Safe Medication Practices, incorporating evidence-based medicine (EBM) into order sets is not by itself sufficient.^2,3Execution of proper design coupled with simplicity and provider efficiency is paramount to HCP buy-in, increased likelihood of order set adherence, and to potentially better outcomes.

In this article, we outline advancements in order set design. These improvements increase provider efficiency and ease of use; incorporate human factors engineering (HFE); apply failure mode and effects analysis; and include EBM.

Methods

An inpatient nicotine replacement therapy (NRT) order was developed as part of a multifaceted solution to improve tobacco cessation care at the James A. Haley Veterans’ Hospital (JAHVH) in Tampa, Florida, a complexity level 1a facility. This NRT order set used the 4-step order set design framework the authors’ developed (for additional information about the NRT order set, contact the authors). We distinguish order set design technique between 2 different inpatient NRT order sets. The first order set in the comparison (Figure 1) is an inpatient NRT order set of unknown origin—it is common for US Department of Veterans Affairs (VA) medical facilities to share order sets and other resources. The second order set (Figure 2) is an inpatient NRT order set we designed using our 4-step process for comparison in this article. No institutional review board approval was required as this work met criteria for operational improvement activities exempt from ethics review.

Justin Iannello, DO, MBA, was the team leader and developer of the 4-step order set design technique. The intervention team consisted of 4 internal medicine physicians with expertise in quality improvement and patient safety: 1 certified professional in patient safety and certified as a Lean Six Sigma Black Belt; 2 physicians certified as Lean Six Sigma Black Belts; and 1 physician certified as a Lean Six Sigma Green Belt. Two inpatient clinical pharmacists and 1 quality management specialist also were involved in its development.

Development of a new NRT order set was felt to be an integral part of the tobacco cessation care delivery process. An NRT order set perceived by users as value-added required a solution that merged EBM with standardization and applied quality improvement principles. The result was an approach to order set design that focused on 4 key questions: Is the order set efficient and easy to use/navigate? Is human factors engineering incorporated? Is failure mode and effects analysis applied? Are evidence-based practices included?

Ease of Use and Navigation

Implementing an order set that is efficient and easy to use or navigate seems straightforward but can be difficult to execute. Figure 1 shows many detailed options consisting of different combinations of nicotine patches, lozenges, and gum. Also included are oral tobacco cessation options (bupropion and varenicline). Although more options may seem better, confusion about appropriate medication selection can occur.

According to Heath and Heath, too many options can result in lack of action.⁴ For example, Heath and Heath discuss a food store that offered 6 free samples of different jams on one day and 24 jams the following day. The customers who sampled 6 different types of jam were 10 times more likely to buy jam. The authors concluded that the more options available, the more difficulty a potential buyer has in deciding on a course of action.⁴

In clinical situations where a HCP is using an order set, the number of options can mean the difference between use vs avoidance if the choices are overwhelming. HCPs process layers of detail every day when creating differential diagnoses and treatment plans. While that level of detail is necessary clinically, that same level of detail included in orders sets can create challenges for HCPs.

Figure 2 advances the order set in Figure 1 by providing a simpler and cleaner design, so HCPs can more easily review and process the information. This order set design minimizes the number of options available to help users make the right decision, focusing on value for the appropriate setting and audience. In other words, order sets should not be a “one size fits all” approach.

Order sets should be tailored to the appropriate clinical setting (eg, inpatient acute care, outpatient clinic setting, etc) and HCP (eg, hospitalist, tobacco cessation specialist, etc). We are comparing NRT order sets designed for HCPs who do not routinely prescribe oral tobacco cessation products in the inpatient setting. When possible, autogenerated bundle orders should also be used according to evidence-based recommendations (such as nicotine patch tapers) for ease of use and further simplification of order sets.

Finally, usability testing known as “evaluating a product or service by testing it with representative users” helps further refine an order set.⁵Usability testing should be applied during all phases of order set development with end user(s) as it helps identify problems with order set design prior to implementation. By applying usability testing, the order set becomes more meaningful and valued by the user.

Human Factors Engineering

HFE is “the study of all the factors that make it easier to do the work in the right way.”⁶ HFE seeks to identify, align, and apply processes for people and the world within which they live and work to promote safe and efficient practices, especially in relation to the technology and physical design features in their work environment.⁶

The average American adult makes about 35,000 decisions per day.⁷ Thus, there is potential for error at any moment. Design that does not take HFE into account can be dangerous. For example, when tube feed and IV line connectors look similar and are compatible, patients may inadvertently receive food administered directly into their bloodstream.⁸

HFE can and should be applied to order sets. Everything from the look, feel, and verbiage of an order set affects potential outcomes. For example, consider the impact even seemingly minor modifications can have on outcomes simply by guiding users in a different way: Figure 1 provides NRT options based on cigarette use per day, whereas Figure 2 conveys pack use per day in relation to the equivalent number of cigarettes used daily. These differences may seem small; however, it helps guide users to the right choice when considering that health care providers have been historically trained on social history gathering that emphasizes packs per day and pack-years.

Failure Mode and Effects Analysis

Failure mode and effects analysis (FMEA) is “a structured way to identify and address potential problems, or failures and their resulting effects on the system or process before an adverse event occurs.”⁹ The benefit of an order set must be weighed against the risk during development. FMEA should be applied during order set design to assess and limit risk just as with any other clinical care process.

FMEA examines both level of risk and frequency of risk occurrence associated with a new proposed process. For example, let’s evaluate an order set designed for pain control after surgery that consists of multiple high-risk opioids along with antihistamine medications for as-needed itch relief (a non-life-threatening adverse event (AE) of opioids well known by the medical community). An interdisciplinary FMEA team consisting of subject matter experts may examine how the process should flow in step-by-step detail and then discuss the benefit of a process and risk for potential error. A FMEA team would then analyze what could go wrong with each part of the process and assign a level of risk and risk frequency for various steps in the process, and then decide that certain steps should be modified or eliminated. Perhaps after FMEA, a facility might conclude that the risk of serious complications is high when you combine opioid use with antihistamine medications. The facility could decide to remove antihistamine medications from an order set if it is determined that risks outweigh benefits. While a root cause analysis might identify the cause of an AE after order set use, these situations can be prevented with FMEA.

When applying FMEA to Figure 1, while bupropion is known as an evidence-based oral tobacco cessation option, there is the possibility that bupropion could be inadvertently prescribed from the order set in a hospitalized patient with alcohol withdrawal and withdrawal seizure history. These potentially dangerous situations can be avoided with FMEA. Thus, although bupropion may be evidence-based for NRT, decisions regarding order set design using EBM alone are insufficient.

The practitioner must consider possible unintended consequences within order sets and target treatment options to the appropriate setting and audience. Although Figure 1 may appear to be more inclusive, the interdisciplinary committee designing the inpatient NRT order set felt there was heightened risk with introducing bupropion in Figure 1 and decided the risk would be lowered by removing bupropion from the redesigned NRT order set (Figure 2). In addition to the goal of balancing availability of NRT options with acceptable risk, Figure 2 also focused on building an NRT order set most applicable to the inpatient setting.

Including Evidence-Based Practices

EBM has become a routine part of clinical decision making. Therefore, including EBM in order set design is vital. EBM for NRT has demonstrated that combination therapy is more effective than is monotherapy to help tobacco users quit. Incremental doses of NRT are recommended for patients who use tobacco more frequently.¹⁰

As shown in Figures 1 and 2, both order set designs incorporate EBM for NRT. Although the importance of implementing EBM is evident, critical factors, such as HFE and FMEA make a difference with well-designed order sets.

Results

The 4-step order set design technique was used during development of an inpatient NRT order set at the JAHVH. Results for the inpatient Joint Commission Tobacco Treatment Measures were obtained from the Veterans Health Administration quality metric reporting system known as Strategic Analytics for Improvement and Learning (SAIL). SAIL performance measure outcomes, which include the inpatient Joint Commission Tobacco Treatment Measures, are derived from chart reviews conducted by the External Peer Review Program. Outcomes demonstrated that TOB-2 and TOB-3 (2 inpatient Joint Commission Tobacco Treatment Measures) known as tob20 and tob40, respectively, within SAIL improved by more than 300% after development of an NRT order set using the 4-step order set design framework along with implementation of a multifaceted tobacco cessation care delivery system at JAHVH.

Discussion

While the overall tobacco cessation care delivery system contributed to improved outcomes with the inpatient Joint Commission Tobacco Treatment Measures at JAHVH, the NRT order set was a cornerstone of the design. Although using our order set design technique does not necessarily guarantee successful outcomes, we believe using the 4-step order set design process increases the value of order sets and has potential to improve quality outcomes.

Limitations

Although improved outcomes following implementation of our NRT order set suggest correlation, causation cannot be proven. Also while the NRT order set is believed to have helped tremendously with outcomes, the entire tobacco cessation care delivery system at JAHVH contributed to the results. In addition, the inpatient Joint Commission Tobacco Treatment Measures help improve processes for tobacco cessation care. However, we are uncertain whether the results of our improvement efforts helped patients stop tobacco use. Further studies are needed to determine impact on population health. Finally, our results were based on improvement work done at a single center. Further studies are necessary to see whether results are reproducible.

Conclusion

There was significant improvement with the inpatient Joint Commission Tobacco Treatment Measures outcomes following development of a tobacco cessation care delivery system that included design of an inpatient NRT order set using a 4-step process we developed. This 4-step structure includes emphasis on efficiency and ease of use; human factors engineering; failure mode and effects analysis; and incorporation of evidence-based medicine (Box.) Postimplementation results showed improvement of the inpatient Joint Commission Tobacco Treatment Measures by greater than 3-fold at a single hospital.

The next steps for this initiative include testing the 4-step order set design process in multiple clinical settings to determine the effectiveness of this approach in other areas of clinical care.

Artificial intelligence (AI), first described in 1956, encompasses the field of computer science in which machines are trained to learn from experience. The term was popularized by the 1956 Dartmouth College Summer Research Project on Artificial Intelligence.¹ The field of AI is rapidly growing and has the potential to affect many aspects of our lives. The emerging importance of AI is demonstrated by a February 2019 executive order that launched the American AI Initiative, allocating resources and funding for AI development.² The executive order stresses the potential impact of AI in the health care field, including its potential utility to diagnose disease. Federal agencies were directed to invest in AI research and development to promote rapid breakthroughs in AI technology that may impact multiple areas of society.

Machine learning (ML), a subset of AI, was defined in 1959 by Arthur Samuel and is achieved by employing mathematic models to compute sample data sets.³ Originating from statistical linear models, neural networks were conceived to accomplish these tasks.⁴ These pioneering scientific achievements led to recent developments of deep neural networks. These models are developed to recognize patterns and achieve complex computational tasks within a matter of minutes, often far exceeding human ability.⁵ ML can increase efficiency with decreased computation time, high precision, and recall when compared with that of human decision making.⁶

ML has the potential for numerous applications in the health care field.^7-9 One promising application is in the field of anatomic pathology. ML allows representative images to be used to train a computer to recognize patterns from labeled photographs. Based on a set of images selected to represent a specific tissue or disease process, the computer can be trained to evaluate and recognize new and unique images from patients and render a diagnosis.¹⁰ Prior to modern ML models, users would have to import many thousands of training images to produce algorithms that could recognize patterns with high accuracy. Modern ML algorithms allow for a model known as transfer learning, such that far fewer images are required for training.^11-13

Two novel ML platforms available for public use are offered through Google (Mountain View, CA) and Apple (Cupertino, CA).^14,15 They each offer a user-friendly interface with minimal experience required in computer science. Google AutoML uses ML via cloud services to store and retrieve data with ease. No coding knowledge is required. The Apple Create ML Module provides computer-based ML, requiring only a few lines of code.

The Veterans Health Administration (VHA) is the largest single health care system in the US, and nearly 50 000 cancer cases are diagnosed at the VHA annually.¹⁶ Cancers of the lung and colon are among the most common sources of invasive cancer and are the 2 most common causes of cancer deaths in America.¹⁶ We have previously reported using Apple ML in detecting non-small cell lung cancers (NSCLCs), including adenocarcinomas and squamous cell carcinomas (SCCs); and colon cancers with accuracy.^17,18 In the present study, we expand on these findings by comparing Apple and Google ML platforms in a variety of common pathologic scenarios in veteran patients. Using limited training data, both programs are compared for precision and recall in differentiating conditions involving lung and colon pathology.

In the first 4 experiments, we evaluated the ability of the platforms to differentiate normal lung tissue from cancerous lung tissue, to distinguish lung adenocarcinoma from SCC, and to differentiate colon adenocarcinoma from normal colon tissue. Next, cases of colon adenocarcinoma were assessed to determine whether the presence or absence of the KRAS proto-oncogene could be determined histologically using the AI platforms. KRAS is found in a variety of cancers, including about 40% of colon adenocarcinomas.¹⁹ For colon cancers, the presence or absence of the mutation in KRAS has important implications for patients as it determines whether the tumor will respond to specific chemotherapy agents.²⁰ The presence of the KRAS gene is currently determined by complex molecular testing of tumor tissue.²¹ However, we assessed the potential of ML to determine whether the mutation is present by computerized morphologic analysis alone. Our last experiment examined the ability of the Apple and Google platforms to differentiate between adenocarcinomas of lung origin vs colon origin. This has potential utility in determining the site of origin of metastatic carcinoma.²²

Methods

Fifty cases of lung SCC, 50 cases of lung adenocarcinoma, and 50 cases of colon adenocarcinoma were randomly retrieved from our molecular database. Twenty-five colon adenocarcinoma cases were positive for mutation in KRAS, while 25 cases were negative for mutation in KRAS. Seven hundred fifty total images of lung tissue (250 benign lung tissue, 250 lung adenocarcinomas, and 250 lung SCCs) and 500 total images of colon tissue (250 benign colon tissue and 250 colon adenocarcinoma) were obtained using a Leica Microscope MC190 HD Camera (Wetzlar, Germany) connected to an Olympus BX41 microscope (Center Valley, PA) and the Leica Acquire 9072 software for Apple computers. All the images were captured at a resolution of 1024 x 768 pixels using a 60x dry objective. Lung tissue images were captured and saved on a 2012 Apple MacBook Pro computer, and colon images were captured and saved on a 2011 Apple iMac computer. Both computers were running macOS v10.13.

Creating Image Classifier Models Using Apple Create ML

Apple Create ML is a suite of products that use various tools to create and train custom ML models on Apple computers.¹⁵ The suite contains many features, including image classification to train a ML model to classify images, natural language processing to classify natural language text, and tabular data to train models that deal with labeling information or estimating new quantities. We used Create ML Image Classification to create image classifier models for our project (Appendix A).

Creating ML Modules Using Google Cloud AutoML Vision Beta

Google Cloud AutoML is a suite of machine learning products, including AutoML Vision, AutoML Natural Language and AutoML Translation.¹⁴ All Cloud AutoML machine learning products were in beta version at the time of experimentation. We used Cloud AutoML Vision beta to create ML modules for our project. Unlike Apple Create ML, which is run on a local Apple computer, the Google Cloud AutoML is run online using a Google Cloud account. There are no minimum specifications requirements for the local computer since it is using the cloud-based architecture (Appendix B).

Experiment 1

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to detect and subclassify NSCLC based on the histopathologic images. We created 3 classes of images (250 images each): benign lung tissue, lung adenocarcinoma, and lung SCC.

Experiment 2

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between normal lung tissue and NSCLC histopathologic images with 50/50 mixture of lung adenocarcinoma and lung SCC. We created 2 classes of images (250 images each): benign lung tissue and lung NSCLC.

Experiment 3

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between lung adenocarcinoma and lung SCC histopathologic images. We created 2 classes of images (250 images each): adenocarcinoma and SCC.

Experiment 4

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to detect colon cancer histopathologic images regardless of mutation in KRAS status. We created 2 classes of images (250 images each): benign colon tissue and colon adenocarcinoma.

Experiment 5

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between colon adenocarcinoma with mutations in KRAS and colon adenocarcinoma without the mutation in KRAS histopathologic images. We created 2 classes of images (125 images each): colon adenocarcinoma cases with mutation in KRAS and colon adenocarcinoma cases without the mutation in KRAS.

Experiment 6

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between lung adenocarcinoma and colon adenocarcinoma histopathologic images. We created 2 classes of images (250 images each): colon adenocarcinoma lung adenocarcinoma.

Results

Twelve machine learning models were created in 6 experiments using the Apple Create ML and the Google AutoML (Table). To investigate recall and precision differences between the Apple and the Google ML algorithms, we performed 2-tailed distribution, paired t tests. No statistically significant differences were found (P = .52 for recall and .60 for precision).

Overall, each model performed well in distinguishing between normal and neoplastic tissue for both lung and colon cancers. In subclassifying NSCLC into adenocarcinoma and SCC, the models were shown to have high levels of precision and recall. The models also were successful in distinguishing between lung and colonic origin of adenocarcinoma (Figures 1-4). However, both systems had trouble discerning colon adenocarcinoma with mutations in KRAS from adenocarcinoma without mutations in KRAS.

Discussion

Image classifier models using ML algorithms hold a promising future to revolutionize the health care field. ML products, such as those modules offered by Apple and Google, are easy to use and have a simple graphic user interface to allow individuals to train models to perform humanlike tasks in real time. In our experiments, we compared multiple algorithms to determine their ability to differentiate and subclassify histopathologic images with high precision and recall using common scenarios in treating veteran patients.

Analysis of the results revealed high precision and recall values illustrating the models’ ability to differentiate and detect benign lung tissue from lung SCC and lung adenocarcinoma in ML model 1, benign lung from NSCLC carcinoma in ML model 2, and benign colon from colonic adenocarcinoma in ML model 4. In ML model 3 and 6, both ML algorithms performed at a high level to differentiate lung SCC from lung adenocarcinoma and lung adenocarcinoma from colonic adenocarcinoma, respectively. Of note, ML model 5 had the lowest precision and recall values across both algorithms demonstrating the models’ limited utility in predicting molecular profiles, such as mutations in KRAS as tested here. This is not surprising as pathologists currently require complex molecular tests to detect mutations in KRAS reliably in colon cancer.

Both modules require minimal programming experience and are easy to use. In our comparison, we demonstrated critical distinguishing characteristics that differentiate the 2 products.

Apple Create ML image classifier is available for use on local Mac computers that use Xcode version 10 and macOS 10.14 or later, with just 3 lines of code required to perform computations. Although this product is limited to Apple computers, it is free to use, and images are stored on the computer hard drive. Of unique significance on the Apple system platform, images can be augmented to alter their appearance to enhance model training. For example, imported images can be cropped, rotated, blurred, and flipped, in order to optimize the model’s training abilities to recognize test images and perform pattern recognition. This feature is not as readily available on the Google platform. Apple Create ML Image classifier’s default training set consists of 75% of total imported images with 5% of the total images being randomly used as a validation set. The remaining 20% of images comprise the testing set. The module’s computational analysis to train the model is achieved in about 2 minutes on average. The score threshold is set at 50% and cannot be manipulated for each image class as in Google AutoML Vision.

Google AutoML Vision is open and can be accessed from many devices. It stores images on remote Google servers but requires computing fees after a $300 credit for 12 months. On AutoML Vision, random 80% of the total images are used in the training set, 10% are used in the validation set, and 10% are used in the testing set. It is important to highlight the different percentages used in the default settings on the respective modules. The time to train the Google AutoML Vision with default computational power is longer on average than Apple Create ML, with about 8 minutes required to train the machine learning module. However, it is possible to choose more computational power for an additional fee and decrease module training time. The user will receive e-mail alerts when the computer time begins and is completed. The computation time is calculated by subtracting the time of the initial e-mail from the final e-mail.

Based on our calculations, we determined there was no significant difference between the 2 machine learning algorithms tested at the default settings with recall and precision values obtained. These findings demonstrate the promise of using a ML algorithm to assist in the performance of human tasks and behaviors, specifically the diagnosis of histopathologic images. These results have numerous potential uses in clinical medicine. ML algorithms have been successfully applied to diagnostic and prognostic endeavors in pathology,^23-28 dermatology,^29-31 ophthalmology,³² cardiology,³³ and radiology.^34-36

Pathologists often use additional tests, such as special staining of tissues or molecular tests, to assist with accurate classification of tumors. ML platforms offer the potential of an additional tool for pathologists to use along with human microscopic interpretation.^37,38 In addition, the number of pathologists in the US is dramatically decreasing, and many other countries have marked physician shortages, especially in fields of specialized training such as pathology.^39-42 These models could readily assist physicians in underserved countries and impact shortages of pathologists elsewhere by providing more specific diagnoses in an expedited manner.⁴³

Finally, although we have explored the application of these platforms in common cancer scenarios, great potential exists to use similar techniques in the detection of other conditions. These include the potential for classification and risk assessment of precancerous lesions, infectious processes in tissue (eg, detection of tuberculosis or malaria),^24,44 inflammatory conditions (eg, arthritis subtypes, gout),⁴⁵ blood disorders (eg, abnormal blood cell morphology),⁴⁶ and many others. The potential of these technologies to improve health care delivery to veteran patients seems to be limited only by the imagination of the user.⁴⁷

Regarding the limited effectiveness in determining the presence or absence of mutations in KRAS for colon adenocarcinoma, it is mentioned that currently pathologists rely on complex molecular tests to detect the mutations at the DNA level.²¹ It is possible that the use of more extensive training data sets may improve recall and precision in cases such as these and warrants further study. Our experiments were limited to the stipulations placed by the free trial software agreements; no costs were expended to use the algorithms, though an Apple computer was required.

Conclusion

We have demonstrated the successful application of 2 readily available ML platforms in providing diagnostic guidance in differentiation between common cancer conditions in veteran patient populations. Although both platforms performed very well with no statistically significant differences in results, some distinctions are worth noting. Apple Create ML can be used on local computers but is limited to an Apple operating system. Google AutoML is not platform-specific but runs only via Google Cloud with associated computational fees. Using these readily available models, we demonstrated the vast potential of AI in diagnostic pathology. The application of AI to clinical medicine remains in the very early stages. The VA is uniquely poised to provide leadership as AI technologies will continue to dramatically change the future of health care, both in veteran and nonveteran patients nationwide.

Acknowledgments

The authors thank Paul Borkowski for his constructive criticism and proofreading of this manuscript. This material is the result of work supported with resources and the use of facilities at the James A. Haley Veterans’ Hospital.

Artificial intelligence (AI), first described in 1956, encompasses the field of computer science in which machines are trained to learn from experience. The term was popularized by the 1956 Dartmouth College Summer Research Project on Artificial Intelligence.¹ The field of AI is rapidly growing and has the potential to affect many aspects of our lives. The emerging importance of AI is demonstrated by a February 2019 executive order that launched the American AI Initiative, allocating resources and funding for AI development.² The executive order stresses the potential impact of AI in the health care field, including its potential utility to diagnose disease. Federal agencies were directed to invest in AI research and development to promote rapid breakthroughs in AI technology that may impact multiple areas of society.

Machine learning (ML), a subset of AI, was defined in 1959 by Arthur Samuel and is achieved by employing mathematic models to compute sample data sets.³ Originating from statistical linear models, neural networks were conceived to accomplish these tasks.⁴ These pioneering scientific achievements led to recent developments of deep neural networks. These models are developed to recognize patterns and achieve complex computational tasks within a matter of minutes, often far exceeding human ability.⁵ ML can increase efficiency with decreased computation time, high precision, and recall when compared with that of human decision making.⁶

ML has the potential for numerous applications in the health care field.^7-9 One promising application is in the field of anatomic pathology. ML allows representative images to be used to train a computer to recognize patterns from labeled photographs. Based on a set of images selected to represent a specific tissue or disease process, the computer can be trained to evaluate and recognize new and unique images from patients and render a diagnosis.¹⁰ Prior to modern ML models, users would have to import many thousands of training images to produce algorithms that could recognize patterns with high accuracy. Modern ML algorithms allow for a model known as transfer learning, such that far fewer images are required for training.^11-13

Two novel ML platforms available for public use are offered through Google (Mountain View, CA) and Apple (Cupertino, CA).^14,15 They each offer a user-friendly interface with minimal experience required in computer science. Google AutoML uses ML via cloud services to store and retrieve data with ease. No coding knowledge is required. The Apple Create ML Module provides computer-based ML, requiring only a few lines of code.

The Veterans Health Administration (VHA) is the largest single health care system in the US, and nearly 50 000 cancer cases are diagnosed at the VHA annually.¹⁶ Cancers of the lung and colon are among the most common sources of invasive cancer and are the 2 most common causes of cancer deaths in America.¹⁶ We have previously reported using Apple ML in detecting non-small cell lung cancers (NSCLCs), including adenocarcinomas and squamous cell carcinomas (SCCs); and colon cancers with accuracy.^17,18 In the present study, we expand on these findings by comparing Apple and Google ML platforms in a variety of common pathologic scenarios in veteran patients. Using limited training data, both programs are compared for precision and recall in differentiating conditions involving lung and colon pathology.

In the first 4 experiments, we evaluated the ability of the platforms to differentiate normal lung tissue from cancerous lung tissue, to distinguish lung adenocarcinoma from SCC, and to differentiate colon adenocarcinoma from normal colon tissue. Next, cases of colon adenocarcinoma were assessed to determine whether the presence or absence of the KRAS proto-oncogene could be determined histologically using the AI platforms. KRAS is found in a variety of cancers, including about 40% of colon adenocarcinomas.¹⁹ For colon cancers, the presence or absence of the mutation in KRAS has important implications for patients as it determines whether the tumor will respond to specific chemotherapy agents.²⁰ The presence of the KRAS gene is currently determined by complex molecular testing of tumor tissue.²¹ However, we assessed the potential of ML to determine whether the mutation is present by computerized morphologic analysis alone. Our last experiment examined the ability of the Apple and Google platforms to differentiate between adenocarcinomas of lung origin vs colon origin. This has potential utility in determining the site of origin of metastatic carcinoma.²²

Methods

Fifty cases of lung SCC, 50 cases of lung adenocarcinoma, and 50 cases of colon adenocarcinoma were randomly retrieved from our molecular database. Twenty-five colon adenocarcinoma cases were positive for mutation in KRAS, while 25 cases were negative for mutation in KRAS. Seven hundred fifty total images of lung tissue (250 benign lung tissue, 250 lung adenocarcinomas, and 250 lung SCCs) and 500 total images of colon tissue (250 benign colon tissue and 250 colon adenocarcinoma) were obtained using a Leica Microscope MC190 HD Camera (Wetzlar, Germany) connected to an Olympus BX41 microscope (Center Valley, PA) and the Leica Acquire 9072 software for Apple computers. All the images were captured at a resolution of 1024 x 768 pixels using a 60x dry objective. Lung tissue images were captured and saved on a 2012 Apple MacBook Pro computer, and colon images were captured and saved on a 2011 Apple iMac computer. Both computers were running macOS v10.13.

Creating Image Classifier Models Using Apple Create ML

Apple Create ML is a suite of products that use various tools to create and train custom ML models on Apple computers.¹⁵ The suite contains many features, including image classification to train a ML model to classify images, natural language processing to classify natural language text, and tabular data to train models that deal with labeling information or estimating new quantities. We used Create ML Image Classification to create image classifier models for our project (Appendix A).

Creating ML Modules Using Google Cloud AutoML Vision Beta

Google Cloud AutoML is a suite of machine learning products, including AutoML Vision, AutoML Natural Language and AutoML Translation.¹⁴ All Cloud AutoML machine learning products were in beta version at the time of experimentation. We used Cloud AutoML Vision beta to create ML modules for our project. Unlike Apple Create ML, which is run on a local Apple computer, the Google Cloud AutoML is run online using a Google Cloud account. There are no minimum specifications requirements for the local computer since it is using the cloud-based architecture (Appendix B).

Experiment 1

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to detect and subclassify NSCLC based on the histopathologic images. We created 3 classes of images (250 images each): benign lung tissue, lung adenocarcinoma, and lung SCC.

Experiment 2

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between normal lung tissue and NSCLC histopathologic images with 50/50 mixture of lung adenocarcinoma and lung SCC. We created 2 classes of images (250 images each): benign lung tissue and lung NSCLC.

Experiment 3

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between lung adenocarcinoma and lung SCC histopathologic images. We created 2 classes of images (250 images each): adenocarcinoma and SCC.

Experiment 4

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to detect colon cancer histopathologic images regardless of mutation in KRAS status. We created 2 classes of images (250 images each): benign colon tissue and colon adenocarcinoma.

Experiment 5

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between colon adenocarcinoma with mutations in KRAS and colon adenocarcinoma without the mutation in KRAS histopathologic images. We created 2 classes of images (125 images each): colon adenocarcinoma cases with mutation in KRAS and colon adenocarcinoma cases without the mutation in KRAS.

Experiment 6

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between lung adenocarcinoma and colon adenocarcinoma histopathologic images. We created 2 classes of images (250 images each): colon adenocarcinoma lung adenocarcinoma.

Results

Twelve machine learning models were created in 6 experiments using the Apple Create ML and the Google AutoML (Table). To investigate recall and precision differences between the Apple and the Google ML algorithms, we performed 2-tailed distribution, paired t tests. No statistically significant differences were found (P = .52 for recall and .60 for precision).

Overall, each model performed well in distinguishing between normal and neoplastic tissue for both lung and colon cancers. In subclassifying NSCLC into adenocarcinoma and SCC, the models were shown to have high levels of precision and recall. The models also were successful in distinguishing between lung and colonic origin of adenocarcinoma (Figures 1-4). However, both systems had trouble discerning colon adenocarcinoma with mutations in KRAS from adenocarcinoma without mutations in KRAS.

Discussion

Image classifier models using ML algorithms hold a promising future to revolutionize the health care field. ML products, such as those modules offered by Apple and Google, are easy to use and have a simple graphic user interface to allow individuals to train models to perform humanlike tasks in real time. In our experiments, we compared multiple algorithms to determine their ability to differentiate and subclassify histopathologic images with high precision and recall using common scenarios in treating veteran patients.

Analysis of the results revealed high precision and recall values illustrating the models’ ability to differentiate and detect benign lung tissue from lung SCC and lung adenocarcinoma in ML model 1, benign lung from NSCLC carcinoma in ML model 2, and benign colon from colonic adenocarcinoma in ML model 4. In ML model 3 and 6, both ML algorithms performed at a high level to differentiate lung SCC from lung adenocarcinoma and lung adenocarcinoma from colonic adenocarcinoma, respectively. Of note, ML model 5 had the lowest precision and recall values across both algorithms demonstrating the models’ limited utility in predicting molecular profiles, such as mutations in KRAS as tested here. This is not surprising as pathologists currently require complex molecular tests to detect mutations in KRAS reliably in colon cancer.

Both modules require minimal programming experience and are easy to use. In our comparison, we demonstrated critical distinguishing characteristics that differentiate the 2 products.

Apple Create ML image classifier is available for use on local Mac computers that use Xcode version 10 and macOS 10.14 or later, with just 3 lines of code required to perform computations. Although this product is limited to Apple computers, it is free to use, and images are stored on the computer hard drive. Of unique significance on the Apple system platform, images can be augmented to alter their appearance to enhance model training. For example, imported images can be cropped, rotated, blurred, and flipped, in order to optimize the model’s training abilities to recognize test images and perform pattern recognition. This feature is not as readily available on the Google platform. Apple Create ML Image classifier’s default training set consists of 75% of total imported images with 5% of the total images being randomly used as a validation set. The remaining 20% of images comprise the testing set. The module’s computational analysis to train the model is achieved in about 2 minutes on average. The score threshold is set at 50% and cannot be manipulated for each image class as in Google AutoML Vision.

Google AutoML Vision is open and can be accessed from many devices. It stores images on remote Google servers but requires computing fees after a $300 credit for 12 months. On AutoML Vision, random 80% of the total images are used in the training set, 10% are used in the validation set, and 10% are used in the testing set. It is important to highlight the different percentages used in the default settings on the respective modules. The time to train the Google AutoML Vision with default computational power is longer on average than Apple Create ML, with about 8 minutes required to train the machine learning module. However, it is possible to choose more computational power for an additional fee and decrease module training time. The user will receive e-mail alerts when the computer time begins and is completed. The computation time is calculated by subtracting the time of the initial e-mail from the final e-mail.

Based on our calculations, we determined there was no significant difference between the 2 machine learning algorithms tested at the default settings with recall and precision values obtained. These findings demonstrate the promise of using a ML algorithm to assist in the performance of human tasks and behaviors, specifically the diagnosis of histopathologic images. These results have numerous potential uses in clinical medicine. ML algorithms have been successfully applied to diagnostic and prognostic endeavors in pathology,^23-28 dermatology,^29-31 ophthalmology,³² cardiology,³³ and radiology.^34-36

Pathologists often use additional tests, such as special staining of tissues or molecular tests, to assist with accurate classification of tumors. ML platforms offer the potential of an additional tool for pathologists to use along with human microscopic interpretation.^37,38 In addition, the number of pathologists in the US is dramatically decreasing, and many other countries have marked physician shortages, especially in fields of specialized training such as pathology.^39-42 These models could readily assist physicians in underserved countries and impact shortages of pathologists elsewhere by providing more specific diagnoses in an expedited manner.⁴³

Finally, although we have explored the application of these platforms in common cancer scenarios, great potential exists to use similar techniques in the detection of other conditions. These include the potential for classification and risk assessment of precancerous lesions, infectious processes in tissue (eg, detection of tuberculosis or malaria),^24,44 inflammatory conditions (eg, arthritis subtypes, gout),⁴⁵ blood disorders (eg, abnormal blood cell morphology),⁴⁶ and many others. The potential of these technologies to improve health care delivery to veteran patients seems to be limited only by the imagination of the user.⁴⁷

Regarding the limited effectiveness in determining the presence or absence of mutations in KRAS for colon adenocarcinoma, it is mentioned that currently pathologists rely on complex molecular tests to detect the mutations at the DNA level.²¹ It is possible that the use of more extensive training data sets may improve recall and precision in cases such as these and warrants further study. Our experiments were limited to the stipulations placed by the free trial software agreements; no costs were expended to use the algorithms, though an Apple computer was required.

Conclusion

We have demonstrated the successful application of 2 readily available ML platforms in providing diagnostic guidance in differentiation between common cancer conditions in veteran patient populations. Although both platforms performed very well with no statistically significant differences in results, some distinctions are worth noting. Apple Create ML can be used on local computers but is limited to an Apple operating system. Google AutoML is not platform-specific but runs only via Google Cloud with associated computational fees. Using these readily available models, we demonstrated the vast potential of AI in diagnostic pathology. The application of AI to clinical medicine remains in the very early stages. The VA is uniquely poised to provide leadership as AI technologies will continue to dramatically change the future of health care, both in veteran and nonveteran patients nationwide.

Acknowledgments

The authors thank Paul Borkowski for his constructive criticism and proofreading of this manuscript. This material is the result of work supported with resources and the use of facilities at the James A. Haley Veterans’ Hospital.

Artificial intelligence (AI), first described in 1956, encompasses the field of computer science in which machines are trained to learn from experience. The term was popularized by the 1956 Dartmouth College Summer Research Project on Artificial Intelligence.¹ The field of AI is rapidly growing and has the potential to affect many aspects of our lives. The emerging importance of AI is demonstrated by a February 2019 executive order that launched the American AI Initiative, allocating resources and funding for AI development.² The executive order stresses the potential impact of AI in the health care field, including its potential utility to diagnose disease. Federal agencies were directed to invest in AI research and development to promote rapid breakthroughs in AI technology that may impact multiple areas of society.

Machine learning (ML), a subset of AI, was defined in 1959 by Arthur Samuel and is achieved by employing mathematic models to compute sample data sets.³ Originating from statistical linear models, neural networks were conceived to accomplish these tasks.⁴ These pioneering scientific achievements led to recent developments of deep neural networks. These models are developed to recognize patterns and achieve complex computational tasks within a matter of minutes, often far exceeding human ability.⁵ ML can increase efficiency with decreased computation time, high precision, and recall when compared with that of human decision making.⁶

ML has the potential for numerous applications in the health care field.^7-9 One promising application is in the field of anatomic pathology. ML allows representative images to be used to train a computer to recognize patterns from labeled photographs. Based on a set of images selected to represent a specific tissue or disease process, the computer can be trained to evaluate and recognize new and unique images from patients and render a diagnosis.¹⁰ Prior to modern ML models, users would have to import many thousands of training images to produce algorithms that could recognize patterns with high accuracy. Modern ML algorithms allow for a model known as transfer learning, such that far fewer images are required for training.^11-13

Two novel ML platforms available for public use are offered through Google (Mountain View, CA) and Apple (Cupertino, CA).^14,15 They each offer a user-friendly interface with minimal experience required in computer science. Google AutoML uses ML via cloud services to store and retrieve data with ease. No coding knowledge is required. The Apple Create ML Module provides computer-based ML, requiring only a few lines of code.

The Veterans Health Administration (VHA) is the largest single health care system in the US, and nearly 50 000 cancer cases are diagnosed at the VHA annually.¹⁶ Cancers of the lung and colon are among the most common sources of invasive cancer and are the 2 most common causes of cancer deaths in America.¹⁶ We have previously reported using Apple ML in detecting non-small cell lung cancers (NSCLCs), including adenocarcinomas and squamous cell carcinomas (SCCs); and colon cancers with accuracy.^17,18 In the present study, we expand on these findings by comparing Apple and Google ML platforms in a variety of common pathologic scenarios in veteran patients. Using limited training data, both programs are compared for precision and recall in differentiating conditions involving lung and colon pathology.

In the first 4 experiments, we evaluated the ability of the platforms to differentiate normal lung tissue from cancerous lung tissue, to distinguish lung adenocarcinoma from SCC, and to differentiate colon adenocarcinoma from normal colon tissue. Next, cases of colon adenocarcinoma were assessed to determine whether the presence or absence of the KRAS proto-oncogene could be determined histologically using the AI platforms. KRAS is found in a variety of cancers, including about 40% of colon adenocarcinomas.¹⁹ For colon cancers, the presence or absence of the mutation in KRAS has important implications for patients as it determines whether the tumor will respond to specific chemotherapy agents.²⁰ The presence of the KRAS gene is currently determined by complex molecular testing of tumor tissue.²¹ However, we assessed the potential of ML to determine whether the mutation is present by computerized morphologic analysis alone. Our last experiment examined the ability of the Apple and Google platforms to differentiate between adenocarcinomas of lung origin vs colon origin. This has potential utility in determining the site of origin of metastatic carcinoma.²²

Methods

Fifty cases of lung SCC, 50 cases of lung adenocarcinoma, and 50 cases of colon adenocarcinoma were randomly retrieved from our molecular database. Twenty-five colon adenocarcinoma cases were positive for mutation in KRAS, while 25 cases were negative for mutation in KRAS. Seven hundred fifty total images of lung tissue (250 benign lung tissue, 250 lung adenocarcinomas, and 250 lung SCCs) and 500 total images of colon tissue (250 benign colon tissue and 250 colon adenocarcinoma) were obtained using a Leica Microscope MC190 HD Camera (Wetzlar, Germany) connected to an Olympus BX41 microscope (Center Valley, PA) and the Leica Acquire 9072 software for Apple computers. All the images were captured at a resolution of 1024 x 768 pixels using a 60x dry objective. Lung tissue images were captured and saved on a 2012 Apple MacBook Pro computer, and colon images were captured and saved on a 2011 Apple iMac computer. Both computers were running macOS v10.13.

Creating Image Classifier Models Using Apple Create ML

Apple Create ML is a suite of products that use various tools to create and train custom ML models on Apple computers.¹⁵ The suite contains many features, including image classification to train a ML model to classify images, natural language processing to classify natural language text, and tabular data to train models that deal with labeling information or estimating new quantities. We used Create ML Image Classification to create image classifier models for our project (Appendix A).

Creating ML Modules Using Google Cloud AutoML Vision Beta

Google Cloud AutoML is a suite of machine learning products, including AutoML Vision, AutoML Natural Language and AutoML Translation.¹⁴ All Cloud AutoML machine learning products were in beta version at the time of experimentation. We used Cloud AutoML Vision beta to create ML modules for our project. Unlike Apple Create ML, which is run on a local Apple computer, the Google Cloud AutoML is run online using a Google Cloud account. There are no minimum specifications requirements for the local computer since it is using the cloud-based architecture (Appendix B).

Experiment 1

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to detect and subclassify NSCLC based on the histopathologic images. We created 3 classes of images (250 images each): benign lung tissue, lung adenocarcinoma, and lung SCC.

Experiment 2

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between normal lung tissue and NSCLC histopathologic images with 50/50 mixture of lung adenocarcinoma and lung SCC. We created 2 classes of images (250 images each): benign lung tissue and lung NSCLC.

Experiment 3

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between lung adenocarcinoma and lung SCC histopathologic images. We created 2 classes of images (250 images each): adenocarcinoma and SCC.

Experiment 4

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to detect colon cancer histopathologic images regardless of mutation in KRAS status. We created 2 classes of images (250 images each): benign colon tissue and colon adenocarcinoma.

Experiment 5

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between colon adenocarcinoma with mutations in KRAS and colon adenocarcinoma without the mutation in KRAS histopathologic images. We created 2 classes of images (125 images each): colon adenocarcinoma cases with mutation in KRAS and colon adenocarcinoma cases without the mutation in KRAS.

Experiment 6

We compared Apple Create ML Image Classifier and Google AutoML Vision in their ability to differentiate between lung adenocarcinoma and colon adenocarcinoma histopathologic images. We created 2 classes of images (250 images each): colon adenocarcinoma lung adenocarcinoma.

Results

Twelve machine learning models were created in 6 experiments using the Apple Create ML and the Google AutoML (Table). To investigate recall and precision differences between the Apple and the Google ML algorithms, we performed 2-tailed distribution, paired t tests. No statistically significant differences were found (P = .52 for recall and .60 for precision).

Overall, each model performed well in distinguishing between normal and neoplastic tissue for both lung and colon cancers. In subclassifying NSCLC into adenocarcinoma and SCC, the models were shown to have high levels of precision and recall. The models also were successful in distinguishing between lung and colonic origin of adenocarcinoma (Figures 1-4). However, both systems had trouble discerning colon adenocarcinoma with mutations in KRAS from adenocarcinoma without mutations in KRAS.

Discussion

Image classifier models using ML algorithms hold a promising future to revolutionize the health care field. ML products, such as those modules offered by Apple and Google, are easy to use and have a simple graphic user interface to allow individuals to train models to perform humanlike tasks in real time. In our experiments, we compared multiple algorithms to determine their ability to differentiate and subclassify histopathologic images with high precision and recall using common scenarios in treating veteran patients.

Analysis of the results revealed high precision and recall values illustrating the models’ ability to differentiate and detect benign lung tissue from lung SCC and lung adenocarcinoma in ML model 1, benign lung from NSCLC carcinoma in ML model 2, and benign colon from colonic adenocarcinoma in ML model 4. In ML model 3 and 6, both ML algorithms performed at a high level to differentiate lung SCC from lung adenocarcinoma and lung adenocarcinoma from colonic adenocarcinoma, respectively. Of note, ML model 5 had the lowest precision and recall values across both algorithms demonstrating the models’ limited utility in predicting molecular profiles, such as mutations in KRAS as tested here. This is not surprising as pathologists currently require complex molecular tests to detect mutations in KRAS reliably in colon cancer.

Both modules require minimal programming experience and are easy to use. In our comparison, we demonstrated critical distinguishing characteristics that differentiate the 2 products.

Apple Create ML image classifier is available for use on local Mac computers that use Xcode version 10 and macOS 10.14 or later, with just 3 lines of code required to perform computations. Although this product is limited to Apple computers, it is free to use, and images are stored on the computer hard drive. Of unique significance on the Apple system platform, images can be augmented to alter their appearance to enhance model training. For example, imported images can be cropped, rotated, blurred, and flipped, in order to optimize the model’s training abilities to recognize test images and perform pattern recognition. This feature is not as readily available on the Google platform. Apple Create ML Image classifier’s default training set consists of 75% of total imported images with 5% of the total images being randomly used as a validation set. The remaining 20% of images comprise the testing set. The module’s computational analysis to train the model is achieved in about 2 minutes on average. The score threshold is set at 50% and cannot be manipulated for each image class as in Google AutoML Vision.

Google AutoML Vision is open and can be accessed from many devices. It stores images on remote Google servers but requires computing fees after a $300 credit for 12 months. On AutoML Vision, random 80% of the total images are used in the training set, 10% are used in the validation set, and 10% are used in the testing set. It is important to highlight the different percentages used in the default settings on the respective modules. The time to train the Google AutoML Vision with default computational power is longer on average than Apple Create ML, with about 8 minutes required to train the machine learning module. However, it is possible to choose more computational power for an additional fee and decrease module training time. The user will receive e-mail alerts when the computer time begins and is completed. The computation time is calculated by subtracting the time of the initial e-mail from the final e-mail.

Based on our calculations, we determined there was no significant difference between the 2 machine learning algorithms tested at the default settings with recall and precision values obtained. These findings demonstrate the promise of using a ML algorithm to assist in the performance of human tasks and behaviors, specifically the diagnosis of histopathologic images. These results have numerous potential uses in clinical medicine. ML algorithms have been successfully applied to diagnostic and prognostic endeavors in pathology,^23-28 dermatology,^29-31 ophthalmology,³² cardiology,³³ and radiology.^34-36

Pathologists often use additional tests, such as special staining of tissues or molecular tests, to assist with accurate classification of tumors. ML platforms offer the potential of an additional tool for pathologists to use along with human microscopic interpretation.^37,38 In addition, the number of pathologists in the US is dramatically decreasing, and many other countries have marked physician shortages, especially in fields of specialized training such as pathology.^39-42 These models could readily assist physicians in underserved countries and impact shortages of pathologists elsewhere by providing more specific diagnoses in an expedited manner.⁴³

Finally, although we have explored the application of these platforms in common cancer scenarios, great potential exists to use similar techniques in the detection of other conditions. These include the potential for classification and risk assessment of precancerous lesions, infectious processes in tissue (eg, detection of tuberculosis or malaria),^24,44 inflammatory conditions (eg, arthritis subtypes, gout),⁴⁵ blood disorders (eg, abnormal blood cell morphology),⁴⁶ and many others. The potential of these technologies to improve health care delivery to veteran patients seems to be limited only by the imagination of the user.⁴⁷

Regarding the limited effectiveness in determining the presence or absence of mutations in KRAS for colon adenocarcinoma, it is mentioned that currently pathologists rely on complex molecular tests to detect the mutations at the DNA level.²¹ It is possible that the use of more extensive training data sets may improve recall and precision in cases such as these and warrants further study. Our experiments were limited to the stipulations placed by the free trial software agreements; no costs were expended to use the algorithms, though an Apple computer was required.

Conclusion

We have demonstrated the successful application of 2 readily available ML platforms in providing diagnostic guidance in differentiation between common cancer conditions in veteran patient populations. Although both platforms performed very well with no statistically significant differences in results, some distinctions are worth noting. Apple Create ML can be used on local computers but is limited to an Apple operating system. Google AutoML is not platform-specific but runs only via Google Cloud with associated computational fees. Using these readily available models, we demonstrated the vast potential of AI in diagnostic pathology. The application of AI to clinical medicine remains in the very early stages. The VA is uniquely poised to provide leadership as AI technologies will continue to dramatically change the future of health care, both in veteran and nonveteran patients nationwide.

Acknowledgments

The authors thank Paul Borkowski for his constructive criticism and proofreading of this manuscript. This material is the result of work supported with resources and the use of facilities at the James A. Haley Veterans’ Hospital.