User login
Augmented Reality Demonstration Survey Results From a Veteran Affairs Medical Center
Building the health care system of the future requires the thoughtful development and integration of innovative technologies to positively transform care.1-4 Extended reality (XR) represents a spectrum of emerging technologies that have the potential to enhance health care. This includes virtual reality (VR), where a computer-generated visual experience fills the screen; augmented reality (AR), which allows users to see computer-generated images superimposed into an otherwise normal real-world field of view; and mixed reality (MR), which allows users to interact and manipulate computer-generated AR images.
Clinicians and researchers have begun exploring the potential of XR to address a wide variety of health care challenges. A recent systematic review concluded that many clinical studies in this area have small sample sizes and are in the preclinical, proof-of-concept stage, but demonstrate the potential and impact of the underlying VR, AR, and MR technologies.5 Common emerging health care uses for XR include medical education, training, presurgical planning, surgical guidance, distraction therapy for pain and anxiety, and home health indications, including rehabilitation.5-39
Importantly, some researchers have raised concerns regarding the adaptability of the health care workforce with emerging technologies, and their interest in new methods of delivering care.7,39 Successful deployment of any novel health care technology depends on multiple factors, including alignment with staff needs, receptivity to those solutions, customization to specific preferences, and usability.1,3,40-42 Unfortunately, the implementation of some health care technologies, such as electronic health records that did not account for end-user requirements, resulted in employee fatigue, burnout, and negative staffing turnover.42-44 Conversely, elevated employee morale and operational performance have been directly linked to a climate of inclusion and innovation.45-47 In this assessment, we sought to understand US Department of Veterans Affairs (VA) employees’ perceptions and expert opinions related to the introduction of new AR/MR technology.
Methods
The VA Palo Alto Health Care System (VAPAHCS) consists of 3 inpatient hospitals and 7 outpatient clinics, provides a full range of care services to > 90,000 enrolled veterans with 800 hospital beds, 3 nursing homes, and a 100-bed domiciliary. The facility also runs data-driven care projects in research, innovation, and evidence-based practice group under nursing services.48 This project was performed by the VA National Center for Collaborative Healthcare Innovation at the VAPAHCS campus.
The combined technical system used for this assessment included a wireless communication network, AR/MR hardware, and software. Medivis AnatomyX software displayed an interactive human anatomy atlas segmented into about 6000 individual interactive parts. Medivis SurgicalAR received US Food and Drug Administration clearance for presurgical planning and was used to transform and display deidentified diagnostic images (eg, magnetic resonance images and computed tomography) in 3-dimensional (3D) interactive holograms (Figures 1 and 2). 
The wireless Microsoft HoloLens 2 AR/MR headset was used for viewing and sensor-enabled collaborative interaction. Multiple participants in the same physical location simultaneously participated and interacted with 3D holograms. The interactive hologram data were enabled for 3D stereoscopic viewing and manipulation.
Setting and Participants
We reviewed published studies that used questionnaires to evaluate institutions’ level of innovation and new technology user acceptance to develop the questionnaire.49-56 Questions and methods were modified, with a focus on understanding the impact on hospital employees. The questionnaire consisted of 2 predemonstration and 3 postdemonstration sections. The first section included background questions. The second (predemonstration) and third (postdemonstration) sections provided matched questions on feelings about the VA. The fourth section included 2 unmatched questions about how the participant felt this technology would impact veterans and whether the VA should implement similar technologies. We used a 5-point Likert scale for sections 2, 3 and 4 (1 = not at all to 5 = extremely). Two unmatched free-text questions asked how the technology could be used in the participant’s hospital service, and another open-ended question asked for any additional comments. To reduce potential reporting bias, 2 VA employees that did not work at VAPAHCS assisted with the survey distribution and collection. VAPAHCS staff were informed by all employee email and facility intranet of the opportunity to participate; the voluntary demonstration and survey took place on February 10 and 11, 2020.
Data Analysis
All matching pre/post questions were analyzed together to determine statistically significant differences using the Wilcoxon signed rank matched pairs test and pooled t test. Survey respondents were also grouped by employment type to evaluate the impact on subgroups. Results were also grouped by VA tenure into 4 categorical 10-year increments (0-10, 11-20, 21-30, 31-40). Additionally, analysis of variance (ANOVA) was performed on employment types and VA tenure to understand whether there was a statistically significant difference in responses by these subgroups. Respondents’ optional free-text answers were manually reviewed by 2 authors (ZPV and DMA), classified, coded by the common themes, and analyzed for comparison.
Results
A total of 166 participants completed the predemonstration survey, which was a requirement for participating in the AR demonstration. Of those, 159 staff members (95.8%) also completed at least part of the postdemonstration paired structured questions, and their results were included in the analysis. 
Paired Questions
For questions about how innovative the VA is, 108 of 152 participants (71.1%) provided higher scores after the demonstration, 42 (27.6%) had no change, and 2 (1.3%) provided decreased scores. The mean innovative score increased from 3.4 predemonstration to 4.5 postdemonstration on a Likert scale, which is a 1.1 point increase from predemonstration to postdemonstration (95% CI, 0.9- 1.2) or a 22% increase (95% CI, 18%-24%) (P < .001). Respondents level of excitement about VA also increased with 82 of 157 participants (52.2%) providing higher scores after the demonstration, 71 (45.2%) had no change, and 4 scores (2.5%) decreased. The predemonstration mean excitement score of 3.7 increased to 4.3 postdemonstration, which is a 0.6 point increase from before to after the demonstration (95% CI, 0.5-0.7) or a 12% increase (95% CI, 10%-14%) (P < .001). In the survey, 36 of 149 participants (24.2%) had higher scores for their expectation to continue working at VA postdemonstration, 109 (73.2%) had no change, and 4 scores (2.7%) decreased. The mean employee retention score increased from 4.2 predemonstration to 4.5 postdemonstration, which is a 0.3 point increase between pre/post (95% CI, 0.2-0.4) or a 6% increase (95% CI, 4%-8%) (P < .001)
The pre/post questions were analyzed using 1-way ANOVA by hospital department and VA tenure. The responses by department were not statistically significant. Of the 159 employees assessed, 101 respondents (63.5%) had 0 to 10 years VA tenure, 44 (27.7%) had 11 to 20 years, 10 (6.3%) had 21 to 30 years, and 4 (2.5%) had > 31 to 40 years. Length of VA tenure did not impact respondent excitement. Respondents opinions on innovation in the 0 to 10 year and the 11 to 20 year groups rose from 3.2 and 3.7 predemonstration to 4.3 and 4.6 postdemonstration, respectively (P < .001 for both statistical comparisons) (Table 2). Interestingly, the 0 to 10 group saw a 9% rise from a 4.0 score predemonstration to a 4.4 score postdemonstration (P < .001), indicating that the demonstration had a positive impact on their plans to continue employment at VA (Table 3).
Sex did not play a significant role in how respondents answered questions regarding VA excitement or innovation. However, there was a statistically significant difference in how male and female respondents answered the predemonstration question about their plans to continue VA employment, according to the Wilcoxon rank sum test. Predemonstration, female respondents had a mean score of 4.1, which was 6% lower than the 4.4 score of male colleagues (P = .04). Veteran status did have an impact on how respondents felt about VA innovation, and their plans to continue employment at VA. After the demonstration, veteran staff felt the VA was more innovative compared with nonveterans: 4.7 vs 4.4, respectively, a 6% difference (P = .02) Similarly, for the continued VA employment question, veterans had a mean score of 4.8 vs 4.4 for nonveterans, an 8% difference (P = .03) These results suggest that the demonstration had more of an impact on veteran employees vs nonveteran employees.
Unpaired Questions
There were 2 structured unpaired postdemonstration questions. Respondents agreed that similar technology would impact veteran health care with mean (SD) of 4.6 (0.6) and a median score of 5 on a 5-point Likert scale. Respondents also agreed on the importance of implementing similar innovations with mean (SD) of 4.7 (0.5), and a median score of 5.
The survey asked how this technology could benefit their hospital service department and had 64 responses. Forty-six respondents saw applications for education or patient care/surgery. Other responses shared excitement about the technology and its potential to positively impact patient education. There were 37 responses to the open-ended question: 21 respondents expressed excitement for the technology, and 10 respondents reiterated that the demonstration would be of benefit to patient care/surgery and training.
Discussion
Successful development, design, and deployment of any new health care tool depends on leveraging insights from the employees that will be using and supporting these systems. Correspondingly, understanding the impact that advanced technologies have on health care employees’ satisfaction, morale, and retention is critical to our overall institutional strategy. Our findings show that a one-time experience with AR/MR technology elicited positive employee reactions. Of note, the survey revealed statistically significant improvements in staff’s view of the VA, with the greatest positive impact for questions about innovation, followed by excitement to work at the VA, and likelihood to continue work at the VA. It is very disruptive and costly when health care employees leave, and improving employee satisfaction and morale is important for better patient care and patient satisfaction, which is priority for VAPAHCS leadership.57-62
The paired predemonstration and postdemonstration scores were similarly high, nearing the top threshold available for the Likert scale (4.3 to 4.5). Furthermore, the least incremental improvement for these responses was observed for topics that had the highest initial baseline score. Therefore, the improvements observed for the paired questions may have more to do with the high baseline values.
Of additional interest, the self-reported likelihood of continuing to work at the VA increased the most for female employees, veteran employees, and employees with the least number of years at the VA. These demographic differences have important implications for VA staff recruitment and retention strategies.62 The unpaired questions about the impact on veteran care and whether the VA should continue similar work demonstrated extremely high support with median scores of 5 for both questions. The free-text postdemonstration responses also demonstrate similar positive themes, with a disposition for excitement about both the training and patient care applications for this technology. In addition, respondents felt strongly that this and other similar technologies will positively impact the health care for veterans and that the VA should continue these efforts.
Strengths and Limitations
A strength of this assessment is the ability to evaluate survey responses that were systematically collected and matched from the same individual immediately before and after exposure to the new technology. The free-text responses provided additional important information that both confirmed the results and provided additional valued supplementary guidance for future implementation strategies, which is critical for our translational implementation goals. An additional strength is that the voluntary surveys were managed by non-VAPAHCS colleagues, limiting potential bias. Importantly, the number of respondents allowed a statistically significant assessment of important health care employee metrics. These results have emphasized how being part of an innovative organization, and the introduction of advanced AR/MR technology, improve employees’ satisfaction and morale about where they work as well as their intention to stay at their institution.
A limitation of this assessment was the lack of comparative data for employee acceptance of other technologies at VAPAHCS. This limits our ability to differentiate whether the strong positive results observed in this evaluation were a result of the specific technology assessed, or of new and advanced health care technology in general. Nonetheless, our unpaired questions, which received extremely high scores, also included participant questions about comparing the system with other similar technologies. This assessment was also focused on veteran care, which limits generalizability.
Conclusions
One-time exposure to advanced AR technology for health care significantly increased employee morale as measured by excitement about working at the VA as well as employee intention to continue employment at the VA. These collateral benefits of the technology are particularly important in health care because our employees are our most important asset and improving employee morale equates to better patient care. Positive impacts were most pronounced for women employees, newer VA employees, and employees who are also veterans. These more detailed insights are also positioned to have a direct impact on employee recruitment and retention strategies. Additional valuable insights regarding the most applicable use of the technology in the clinical setting were also obtained.
Acknowledgments
We thank Andrew Spiegelman, Hyewon Kim, Jonathan Sills, and Alexander Erickson for their assistance in developing the survey questions. We also thank Jason Rhodes and Mark Bulson for traveling to our facility to assist with managing the anonymous surveys during the demonstration event.
1. World Economic Forum. Health and healthcare in the fourth industrial revolution: Global Future Council on the future of health and healthcare 2016-2018. April 2019. Accessed January 27, 2023. https://www3.weforum.org/docs/WEF__Shaping_the_Future_of_Health_Council_Report.pdf
2. Iveroth E, Fryk P, Rapp B. Information technology strategy and alignment issues in health care organizations. Health Care Manage Rev. 2013;38(3):188-200. doi:10.1097/HMR.0b013e31826119d7
3. Thakur R, Hsu SH, Fontenot G. Innovation in healthcare: issues and future trends. J Bus Res. 2012;65(4):562-569. doi:10.1016/j.jbusres.2011.02.022
4. Thimbleby H. Technology and the future of healthcare. J Public Health Res. 2013;2(3):e28. Published 2013 Dec 1. doi:10.4081/jphr.2013.e28
5. Viglialoro RM, Condino S, Turini G, Carbone M, Ferrari V, Gesi M. augmented reality, mixed reality, and hybrid approach in healthcare simulation: a systematic review. Applied Sciences. 2021;11(5):2338. doi:10.3390/app11052338
6. Rawlins CR, Veigulis Z, Hebert C, Curtin C, Osborne T. Effect of immersive virtual reality on pain and anxiety at a Veterans Affairs health care facility. Front Virt Real. 2021;(2):136. doi:10.3389/frvir.2021.719681
7. Chawdhary G, Shoman N. Emerging artificial intelligence applications in otological imaging. Curr Opin Otolaryngol Head Neck Surg. 2021;29(5):357-364. doi:10.1097/MOO.0000000000000754
8. Asadzadeh A, Samad-Soltani T, Rezaei-Hachesu P. Applications of virtual and augmented reality in infectious disease epidemics with a focus on the COVID-19 outbreak. Inform Med Unlocked. 2021;24:100579. doi:10.1016/j.imu.2021.100579
9. Ashwini KB, Savitha R, Harish A. Application of augmented reality technology for home healthcare product visualization. ECS Transas. 2022;107(1):10921. doi:10.1149/10701.10921ecst
10. Brooks AL. VR/Technologies for Rehabilitation. In: Brooks AL, Brahman S, Kapralos B, Nakajima A, Tyerman J, Jain LC, eds. Recent Advances in Technologies for Inclusive Well-Being Virtual Patients, Gamification and Simulation. Intelligent Systems Reference Library. Springer; 2021:241-252. doi:10.1007/978-3-030-59608-8_13
11. Koulouris D, Menychtas A, Maglogiannis I. An IoT-enabled platform for the assessment of physical and mental activities utilizing augmented reality exergaming. Sensors (Basel). 2022;22(9):3181. Published 2022 Apr 21. doi:10.3390/s22093181
12. Deiss YR, Korkut S, Inglese T. Increase therapy understanding and medication adherence for patients with inflammatory skin diseases through augmented reality. Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022. 2022:21-40. doi:10.1007/978-3-031-06018-2_2
13. Bertino E, Gao W, Steffan B, et al, eds. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer; 2022:21-40.
14. Ruhaiyem NIR, Mazlan NA. Image Modeling Through Augmented Reality for Skin Allergies Recognition. Lecture Notes on Data Engineering and Communications Technologies. 2021:72-79. doi: 10.1007/978-3-030-70713-2_8
15. Park BJ, Perkons NR, Profka E, et al. Three-dimensional augmented reality visualization informs locoregional therapy in a translational model of hepatocellular carcinoma. J Vasc Interv Radiol. 2020;31(10):1612-1618.e1. doi:10.1016/j.jvir.2020.01.028
16. Leo J, Zhou Z, Yang H, et al, eds. Interactive cardiovascular surgical planning via augmented reality. 5th Asian CHI Symposium 2021; 2021. doi:10.1145/3429360.3468195
17. Zuo Y, Jiang T, Dou J, et al. A novel evaluation model for a mixed-reality surgical navigation system: where Microsoft Hololens meets the operating room. Surg Innov. 2020;27(2):193-202. doi:10.1177/1553350619893236
18. Ghaednia H, Fourman MS, Lans A, et al. Augmented and virtual reality in spine surgery, current applications and future potentials. Spine J. 2021;21(10):1617-1625. doi:10.1016/j.spinee.2021.03.018
19. Liu Y, Lee MG, Kim JS. Spine surgery assisted by augmented reality: where have we been?. Yonsei Med J. 2022;63(4):305-316. doi:10.3349/ymj.2022.63.4.305
20. Kimmel S, Cobus V, Heuten W, eds. opticARe—augmented reality mobile patient monitoring in intensive care units. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST; 2021. doi:10.1145/3489849.3489852
21. Voštinár P, Horváthová D, Mitter M, Bako M. The look at the various uses of VR. Open Computer Sci. 2021;11(1):241-250. doi:10.1515/comp-2020-0123
22. Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: a meta-analysis of randomized controlled studies. BMC Med Educ. 2020;20(1):127. Published 2020 Apr 25. doi:10.1186/s12909-020-1994-z
23. Ricci S, Calandrino A, Borgonovo G, Chirico M, Casadio M. Viewpoint: virtual and augmented reality in basic and advanced life support training. JMIR Serious Games. 2022;10(1):e28595. Published 2022 Mar 23. doi:10.2196/28595
24. Ricci S, Mobilio GA, Calandrino A, et al. RiNeo MR: A mixed-reality tool for newborn life support training. Annu Int Conf IEEE Eng Med Biol Soc. 2021;2021:5043-5046. doi:10.1109/EMBC46164.2021.9629612
25. Dhar P, Rocks T, Samarasinghe RM, Stephenson G, Smith C. Augmented reality in medical education: students’ experiences and learning outcomes. Med Educ Online. 2021;26(1):1953953. doi:10.1080/10872981.2021.1953953
26. Pears M, Konstantinidis S. The future of immersive technology in global surgery education [published online ahead of print, 2021 Jul 1]. Indian J Surg. 2021;84(suppl 1):1-5. doi:10.1007/s12262-021-02998-6
27. Liang CJ, Start C, Boley H, Kamat VR, Menassa CC, Aebersold M. Enhancing stroke assessment simulation experience in clinical training using augmented reality. Virt Real. 2021;25(3):575-584. doi:10.1007/s10055-020-00475-1
28. Lacey G, Gozdzielewska L, McAloney-Kocaman K, Ruttle J, Cronin S, Price L. Psychomotor learning theory informing the design and evaluation of an interactive augmented reality hand hygiene training app for healthcare workers. Educ Inf Technol. 2022;27(3):3813-3832. doi:10.1007/s10639-021-10752-4
29. Ryan GV, Callaghan S, Rafferty A, Higgins MF, Mangina E, McAuliffe F. Learning outcomes of immersive technologies in health care student education: systematic review of the literature. J Med Internet Res. 2022;24(2):e30082. Published 2022 Feb 1. doi:10.2196/30082
30. Yu FU, Yan HU, Sundstedt V. A Systematic literature review of virtual, augmented, and mixed reality game applications in healthcare. ACM Trans Comput Healthcare. 2022;3(2);1-27. doi:10.1145/3472303
31. Weeks JK, Amiel JM. Enhancing neuroanatomy education with augmented reality. Med Educ. 2019;53(5):516-517. doi:10.1111/medu.13843
32. Williams MA, McVeigh J, Handa AI, Lee R. Augmented reality in surgical training: a systematic review. Postgrad Med J. 2020;96(1139):537-542. doi:10.1136/postgradmedj-2020-137600
33. Triepels CPR, Smeets CFA, Notten KJB, et al. Does three-dimensional anatomy improve student understanding? Clin Anat. 2020;33(1):25-33. doi:10.1002/ca.23405
34. Pietruski P, Majak M, S´wia¸tek-Najwer E, et al. Supporting fibula free flap harvest with augmented reality: A proof-of-concept study. Laryngoscope. 2020;130(5):1173-1179. doi:10.1002/lary.28090
35. Perkins SL, Krajancich B, Yang CJ, Hargreaves BA, Daniel BL, Berry MF. A patient-specific mixed-reality visualization tool for thoracic surgical planning. Ann Thorac Surg. 2020;110(1):290-295. doi:10.1016/j.athoracsur.2020.01.060
36. Müller F, Roner S, Liebmann F, Spirig JM, Fürnstahl P, Farshad M. Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging. Spine J. 2020;20(4):621-628. doi:10.1016/j.spinee.2019.10.012
37. Kaplan AD, Cruit J, Endsley M, Beers SM, Sawyer BD, Hancock PA. The effects of virtual reality, augmented reality, and mixed reality as training enhancement methods: a meta-analysis. Hum Factors. 2021;63(4):706-726. doi:10.1177/0018720820904229
38. Jud L, Fotouhi J, Andronic O, et al. Applicability of augmented reality in orthopedic surgery - a systematic review. BMC Musculoskelet Disord. 2020;21(1):103. Published 2020 Feb 15. doi:10.1186/s12891-020-3110-2
39. Ara J, Karim FB, Alsubaie MSA, et al. Comprehensive analysis of augmented reality technology in modern healthcare system. Int J Adv Comput Sci Appl. 2021;12(6):845-854. doi:10.14569/IJACSA.2021.0120698
40. Webster A, Gardner J. Aligning technology and institutional readiness: the adoption of innovation. Technol Anal Strateg Manag. 2019;31(10):1229-1241. doi:10.1080/09537325.2019.1601694
41. Hastall MR, Dockweiler C, Mühlhaus J. achieving end user acceptance: building blocks for an evidence-based user-centered framework for health technology development and assessment. In: Antona, M, Stephanidis C, eds. Universal Access in Human–Computer Interaction. Human and Technological Environments. UAHCI 2017. Lecture Notes in Computer Science, vol 10279. Springer, Cham; 2017. doi:10.1007/978-3-319-58700-4_2
42. Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc. 2015;22(6):1179-1182. doi:10.1093/jamia/ocv050
43. Khairat S, Coleman C, Ottmar P, Jayachander DI, Bice T, Carson SS. Association of Electronic Health Record Use With Physician Fatigue and Efficiency. JAMA Netw Open. 2020;3(6):e207385. Published 2020 Jun 1. doi:10.1001/jamanetworkopen.2020.7385
44. Melnick ER, Dyrbye LN, Sinsky CA, et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc. 2020;95(3):476-487. doi:10.1016/j.mayocp.2019.09.024
45. Lee YJ. Comparison of job satisfaction between nonprofit and public employees. Nonprofit Volunt Sect Q. 2016;45(2):295-313. doi:10.1177/0899764015584061
46. Brimhall KC. Inclusion is important... but how do I include? Examining the effects of leader engagement on inclusion, innovation, job satisfaction, and perceived quality of care in a diverse nonprofit health care organization. Nonprofit Volunt Sect Q. 2019;48(4):716-737. doi:10.1177/0899764019829834
47. Moreira MR, Gherman M, Sousa PS. Does innovation influence the performance of healthcare organizations?. Innovation (North Syd). 2017;19(3):335-352. doi:10.1080/14479338.2017.1293489
48. US Department of Veterans Affairs. VA Palo Alto Healthcare System. Updated December 29, 2020. Accessed January 27, 2023. https://www.paloalto.va.gov/about/index.asp
49. Siegel SM, Kaemmerer WF. Measuring the perceived support for innovation in organizations. J Appl Psychol. 1978;63(5):553-562. doi:10.1037/0021-9010.63.5.553
50. Anderson NR, West MA. Measuring climate for work group innovation: development and validation of the team climate inventory. J Organ Behav. 1998;19(3):235-258. doi:10.1002/(SICI)1099-1379(199805)19:3<235::AID-JOB837>3.0.CO;2-C
51. Aarons GA. Measuring provider attitudes toward evidence-based practice: consideration of organizational context and individual differences. Child Adolesc Psychiatr Clin N Am. 2005;14(2):255-viii. doi:10.1016/j.chc.2004.04.008
52. Van der Heijden H. User acceptance of hedonic information systems. MIS Q. 2004;28(4):695-704. doi:10.2307/25148660
53. Venkatesh V, Speier C, Morris MG. User acceptance enablers in individual decision making about technology: Toward an integrated model. Decis Sci. 2002;33(2):297-316. doi:10.1111/j.1540-5915.2002.tb01646.x
54. Puri A, Kim B, Nguyen O, Stolee P, Tung J, Lee J. User acceptance of wrist-worn activity trackers among community-dwelling older adults: mixed method study. JMIR Mhealth Uhealth. 2017;5(11):e173. Published 2017 Nov 15. doi:10.2196/mhealth.8211
55. Huang YC, Backman SJ, Backman KF, Moore D. Exploring user acceptance of 3D virtual worlds in travel and tourism marketing. Tourism Management. 2013;36:490-501. doi:10.1016/j.tourman.2012.09.009
56. Rasimah CM, Ahmad A, Zaman HB. Evaluation of user acceptance of mixed reality technology. AJET. 2011;27(8). doi:10.14742/ajet.899
57. Choi J, Boyle DK. RN workgroup job satisfaction and patient falls in acute care hospital units. J Nurs Adm. 2013;43(11):586-591. doi:10.1097/01.NNA.0000434509.66749.7c58. Tzeng HM, Ketefian S. The relationship between nurses’ job satisfaction and inpatient satisfaction: an exploratory study in a Taiwan teaching hospital. J Nurs Care Qual. 2002;16(2):39-49. doi:10.1097/00001786-200201000-00005
59. Williams ES, Skinner AC. Outcomes of physician job satisfaction: a narrative review, implications, and directions for future research. Health Care Manage Rev. 2003;28(2):119-139. doi:10.1097/00004010-200304000-00004
60. Waldman JD, Kelly F, Arora S, Smith HL. The shocking cost of turnover in health care. Health Care Manage Rev. 2004;29(1):2-7. doi:10.1097/00004010-200401000-00002
61. Hayes LJ, O’Brien-Pallas L, Duffield C, et al. Nurse turnover: a literature review - an update. Int J Nurs Stud. 2012;49(7):887-905. doi:10.1016/j.ijnurstu.2011.10.001
62. US Department of Veterans Affairs. FY 2021/FY 2019 Annual performance plan and report. February 2020. Accessed January 27, 2023. https://www.va.gov/oei/docs/VA2019-2021appr.pdf
Building the health care system of the future requires the thoughtful development and integration of innovative technologies to positively transform care.1-4 Extended reality (XR) represents a spectrum of emerging technologies that have the potential to enhance health care. This includes virtual reality (VR), where a computer-generated visual experience fills the screen; augmented reality (AR), which allows users to see computer-generated images superimposed into an otherwise normal real-world field of view; and mixed reality (MR), which allows users to interact and manipulate computer-generated AR images.
Clinicians and researchers have begun exploring the potential of XR to address a wide variety of health care challenges. A recent systematic review concluded that many clinical studies in this area have small sample sizes and are in the preclinical, proof-of-concept stage, but demonstrate the potential and impact of the underlying VR, AR, and MR technologies.5 Common emerging health care uses for XR include medical education, training, presurgical planning, surgical guidance, distraction therapy for pain and anxiety, and home health indications, including rehabilitation.5-39
Importantly, some researchers have raised concerns regarding the adaptability of the health care workforce with emerging technologies, and their interest in new methods of delivering care.7,39 Successful deployment of any novel health care technology depends on multiple factors, including alignment with staff needs, receptivity to those solutions, customization to specific preferences, and usability.1,3,40-42 Unfortunately, the implementation of some health care technologies, such as electronic health records that did not account for end-user requirements, resulted in employee fatigue, burnout, and negative staffing turnover.42-44 Conversely, elevated employee morale and operational performance have been directly linked to a climate of inclusion and innovation.45-47 In this assessment, we sought to understand US Department of Veterans Affairs (VA) employees’ perceptions and expert opinions related to the introduction of new AR/MR technology.
Methods
The VA Palo Alto Health Care System (VAPAHCS) consists of 3 inpatient hospitals and 7 outpatient clinics, provides a full range of care services to > 90,000 enrolled veterans with 800 hospital beds, 3 nursing homes, and a 100-bed domiciliary. The facility also runs data-driven care projects in research, innovation, and evidence-based practice group under nursing services.48 This project was performed by the VA National Center for Collaborative Healthcare Innovation at the VAPAHCS campus.
The combined technical system used for this assessment included a wireless communication network, AR/MR hardware, and software. Medivis AnatomyX software displayed an interactive human anatomy atlas segmented into about 6000 individual interactive parts. Medivis SurgicalAR received US Food and Drug Administration clearance for presurgical planning and was used to transform and display deidentified diagnostic images (eg, magnetic resonance images and computed tomography) in 3-dimensional (3D) interactive holograms (Figures 1 and 2).
The wireless Microsoft HoloLens 2 AR/MR headset was used for viewing and sensor-enabled collaborative interaction. Multiple participants in the same physical location simultaneously participated and interacted with 3D holograms. The interactive hologram data were enabled for 3D stereoscopic viewing and manipulation.
Setting and Participants
We reviewed published studies that used questionnaires to evaluate institutions’ level of innovation and new technology user acceptance to develop the questionnaire.49-56 Questions and methods were modified, with a focus on understanding the impact on hospital employees. The questionnaire consisted of 2 predemonstration and 3 postdemonstration sections. The first section included background questions. The second (predemonstration) and third (postdemonstration) sections provided matched questions on feelings about the VA. The fourth section included 2 unmatched questions about how the participant felt this technology would impact veterans and whether the VA should implement similar technologies. We used a 5-point Likert scale for sections 2, 3 and 4 (1 = not at all to 5 = extremely). Two unmatched free-text questions asked how the technology could be used in the participant’s hospital service, and another open-ended question asked for any additional comments. To reduce potential reporting bias, 2 VA employees that did not work at VAPAHCS assisted with the survey distribution and collection. VAPAHCS staff were informed by all employee email and facility intranet of the opportunity to participate; the voluntary demonstration and survey took place on February 10 and 11, 2020.
Data Analysis
All matching pre/post questions were analyzed together to determine statistically significant differences using the Wilcoxon signed rank matched pairs test and pooled t test. Survey respondents were also grouped by employment type to evaluate the impact on subgroups. Results were also grouped by VA tenure into 4 categorical 10-year increments (0-10, 11-20, 21-30, 31-40). Additionally, analysis of variance (ANOVA) was performed on employment types and VA tenure to understand whether there was a statistically significant difference in responses by these subgroups. Respondents’ optional free-text answers were manually reviewed by 2 authors (ZPV and DMA), classified, coded by the common themes, and analyzed for comparison.
Results
A total of 166 participants completed the predemonstration survey, which was a requirement for participating in the AR demonstration. Of those, 159 staff members (95.8%) also completed at least part of the postdemonstration paired structured questions, and their results were included in the analysis.
Paired Questions
For questions about how innovative the VA is, 108 of 152 participants (71.1%) provided higher scores after the demonstration, 42 (27.6%) had no change, and 2 (1.3%) provided decreased scores. The mean innovative score increased from 3.4 predemonstration to 4.5 postdemonstration on a Likert scale, which is a 1.1 point increase from predemonstration to postdemonstration (95% CI, 0.9- 1.2) or a 22% increase (95% CI, 18%-24%) (P < .001). Respondents level of excitement about VA also increased with 82 of 157 participants (52.2%) providing higher scores after the demonstration, 71 (45.2%) had no change, and 4 scores (2.5%) decreased. The predemonstration mean excitement score of 3.7 increased to 4.3 postdemonstration, which is a 0.6 point increase from before to after the demonstration (95% CI, 0.5-0.7) or a 12% increase (95% CI, 10%-14%) (P < .001). In the survey, 36 of 149 participants (24.2%) had higher scores for their expectation to continue working at VA postdemonstration, 109 (73.2%) had no change, and 4 scores (2.7%) decreased. The mean employee retention score increased from 4.2 predemonstration to 4.5 postdemonstration, which is a 0.3 point increase between pre/post (95% CI, 0.2-0.4) or a 6% increase (95% CI, 4%-8%) (P < .001)
The pre/post questions were analyzed using 1-way ANOVA by hospital department and VA tenure. The responses by department were not statistically significant. Of the 159 employees assessed, 101 respondents (63.5%) had 0 to 10 years VA tenure, 44 (27.7%) had 11 to 20 years, 10 (6.3%) had 21 to 30 years, and 4 (2.5%) had > 31 to 40 years. Length of VA tenure did not impact respondent excitement. Respondents opinions on innovation in the 0 to 10 year and the 11 to 20 year groups rose from 3.2 and 3.7 predemonstration to 4.3 and 4.6 postdemonstration, respectively (P < .001 for both statistical comparisons) (Table 2). Interestingly, the 0 to 10 group saw a 9% rise from a 4.0 score predemonstration to a 4.4 score postdemonstration (P < .001), indicating that the demonstration had a positive impact on their plans to continue employment at VA (Table 3).
Sex did not play a significant role in how respondents answered questions regarding VA excitement or innovation. However, there was a statistically significant difference in how male and female respondents answered the predemonstration question about their plans to continue VA employment, according to the Wilcoxon rank sum test. Predemonstration, female respondents had a mean score of 4.1, which was 6% lower than the 4.4 score of male colleagues (P = .04). Veteran status did have an impact on how respondents felt about VA innovation, and their plans to continue employment at VA. After the demonstration, veteran staff felt the VA was more innovative compared with nonveterans: 4.7 vs 4.4, respectively, a 6% difference (P = .02) Similarly, for the continued VA employment question, veterans had a mean score of 4.8 vs 4.4 for nonveterans, an 8% difference (P = .03) These results suggest that the demonstration had more of an impact on veteran employees vs nonveteran employees.
Unpaired Questions
There were 2 structured unpaired postdemonstration questions. Respondents agreed that similar technology would impact veteran health care with mean (SD) of 4.6 (0.6) and a median score of 5 on a 5-point Likert scale. Respondents also agreed on the importance of implementing similar innovations with mean (SD) of 4.7 (0.5), and a median score of 5.
The survey asked how this technology could benefit their hospital service department and had 64 responses. Forty-six respondents saw applications for education or patient care/surgery. Other responses shared excitement about the technology and its potential to positively impact patient education. There were 37 responses to the open-ended question: 21 respondents expressed excitement for the technology, and 10 respondents reiterated that the demonstration would be of benefit to patient care/surgery and training.
Discussion
Successful development, design, and deployment of any new health care tool depends on leveraging insights from the employees that will be using and supporting these systems. Correspondingly, understanding the impact that advanced technologies have on health care employees’ satisfaction, morale, and retention is critical to our overall institutional strategy. Our findings show that a one-time experience with AR/MR technology elicited positive employee reactions. Of note, the survey revealed statistically significant improvements in staff’s view of the VA, with the greatest positive impact for questions about innovation, followed by excitement to work at the VA, and likelihood to continue work at the VA. It is very disruptive and costly when health care employees leave, and improving employee satisfaction and morale is important for better patient care and patient satisfaction, which is priority for VAPAHCS leadership.57-62
The paired predemonstration and postdemonstration scores were similarly high, nearing the top threshold available for the Likert scale (4.3 to 4.5). Furthermore, the least incremental improvement for these responses was observed for topics that had the highest initial baseline score. Therefore, the improvements observed for the paired questions may have more to do with the high baseline values.
Of additional interest, the self-reported likelihood of continuing to work at the VA increased the most for female employees, veteran employees, and employees with the least number of years at the VA. These demographic differences have important implications for VA staff recruitment and retention strategies.62 The unpaired questions about the impact on veteran care and whether the VA should continue similar work demonstrated extremely high support with median scores of 5 for both questions. The free-text postdemonstration responses also demonstrate similar positive themes, with a disposition for excitement about both the training and patient care applications for this technology. In addition, respondents felt strongly that this and other similar technologies will positively impact the health care for veterans and that the VA should continue these efforts.
Strengths and Limitations
A strength of this assessment is the ability to evaluate survey responses that were systematically collected and matched from the same individual immediately before and after exposure to the new technology. The free-text responses provided additional important information that both confirmed the results and provided additional valued supplementary guidance for future implementation strategies, which is critical for our translational implementation goals. An additional strength is that the voluntary surveys were managed by non-VAPAHCS colleagues, limiting potential bias. Importantly, the number of respondents allowed a statistically significant assessment of important health care employee metrics. These results have emphasized how being part of an innovative organization, and the introduction of advanced AR/MR technology, improve employees’ satisfaction and morale about where they work as well as their intention to stay at their institution.
A limitation of this assessment was the lack of comparative data for employee acceptance of other technologies at VAPAHCS. This limits our ability to differentiate whether the strong positive results observed in this evaluation were a result of the specific technology assessed, or of new and advanced health care technology in general. Nonetheless, our unpaired questions, which received extremely high scores, also included participant questions about comparing the system with other similar technologies. This assessment was also focused on veteran care, which limits generalizability.
Conclusions
One-time exposure to advanced AR technology for health care significantly increased employee morale as measured by excitement about working at the VA as well as employee intention to continue employment at the VA. These collateral benefits of the technology are particularly important in health care because our employees are our most important asset and improving employee morale equates to better patient care. Positive impacts were most pronounced for women employees, newer VA employees, and employees who are also veterans. These more detailed insights are also positioned to have a direct impact on employee recruitment and retention strategies. Additional valuable insights regarding the most applicable use of the technology in the clinical setting were also obtained.
Acknowledgments
We thank Andrew Spiegelman, Hyewon Kim, Jonathan Sills, and Alexander Erickson for their assistance in developing the survey questions. We also thank Jason Rhodes and Mark Bulson for traveling to our facility to assist with managing the anonymous surveys during the demonstration event.
Building the health care system of the future requires the thoughtful development and integration of innovative technologies to positively transform care.1-4 Extended reality (XR) represents a spectrum of emerging technologies that have the potential to enhance health care. This includes virtual reality (VR), where a computer-generated visual experience fills the screen; augmented reality (AR), which allows users to see computer-generated images superimposed into an otherwise normal real-world field of view; and mixed reality (MR), which allows users to interact and manipulate computer-generated AR images.
Clinicians and researchers have begun exploring the potential of XR to address a wide variety of health care challenges. A recent systematic review concluded that many clinical studies in this area have small sample sizes and are in the preclinical, proof-of-concept stage, but demonstrate the potential and impact of the underlying VR, AR, and MR technologies.5 Common emerging health care uses for XR include medical education, training, presurgical planning, surgical guidance, distraction therapy for pain and anxiety, and home health indications, including rehabilitation.5-39
Importantly, some researchers have raised concerns regarding the adaptability of the health care workforce with emerging technologies, and their interest in new methods of delivering care.7,39 Successful deployment of any novel health care technology depends on multiple factors, including alignment with staff needs, receptivity to those solutions, customization to specific preferences, and usability.1,3,40-42 Unfortunately, the implementation of some health care technologies, such as electronic health records that did not account for end-user requirements, resulted in employee fatigue, burnout, and negative staffing turnover.42-44 Conversely, elevated employee morale and operational performance have been directly linked to a climate of inclusion and innovation.45-47 In this assessment, we sought to understand US Department of Veterans Affairs (VA) employees’ perceptions and expert opinions related to the introduction of new AR/MR technology.
Methods
The VA Palo Alto Health Care System (VAPAHCS) consists of 3 inpatient hospitals and 7 outpatient clinics, provides a full range of care services to > 90,000 enrolled veterans with 800 hospital beds, 3 nursing homes, and a 100-bed domiciliary. The facility also runs data-driven care projects in research, innovation, and evidence-based practice group under nursing services.48 This project was performed by the VA National Center for Collaborative Healthcare Innovation at the VAPAHCS campus.
The combined technical system used for this assessment included a wireless communication network, AR/MR hardware, and software. Medivis AnatomyX software displayed an interactive human anatomy atlas segmented into about 6000 individual interactive parts. Medivis SurgicalAR received US Food and Drug Administration clearance for presurgical planning and was used to transform and display deidentified diagnostic images (eg, magnetic resonance images and computed tomography) in 3-dimensional (3D) interactive holograms (Figures 1 and 2).
The wireless Microsoft HoloLens 2 AR/MR headset was used for viewing and sensor-enabled collaborative interaction. Multiple participants in the same physical location simultaneously participated and interacted with 3D holograms. The interactive hologram data were enabled for 3D stereoscopic viewing and manipulation.
Setting and Participants
We reviewed published studies that used questionnaires to evaluate institutions’ level of innovation and new technology user acceptance to develop the questionnaire.49-56 Questions and methods were modified, with a focus on understanding the impact on hospital employees. The questionnaire consisted of 2 predemonstration and 3 postdemonstration sections. The first section included background questions. The second (predemonstration) and third (postdemonstration) sections provided matched questions on feelings about the VA. The fourth section included 2 unmatched questions about how the participant felt this technology would impact veterans and whether the VA should implement similar technologies. We used a 5-point Likert scale for sections 2, 3 and 4 (1 = not at all to 5 = extremely). Two unmatched free-text questions asked how the technology could be used in the participant’s hospital service, and another open-ended question asked for any additional comments. To reduce potential reporting bias, 2 VA employees that did not work at VAPAHCS assisted with the survey distribution and collection. VAPAHCS staff were informed by all employee email and facility intranet of the opportunity to participate; the voluntary demonstration and survey took place on February 10 and 11, 2020.
Data Analysis
All matching pre/post questions were analyzed together to determine statistically significant differences using the Wilcoxon signed rank matched pairs test and pooled t test. Survey respondents were also grouped by employment type to evaluate the impact on subgroups. Results were also grouped by VA tenure into 4 categorical 10-year increments (0-10, 11-20, 21-30, 31-40). Additionally, analysis of variance (ANOVA) was performed on employment types and VA tenure to understand whether there was a statistically significant difference in responses by these subgroups. Respondents’ optional free-text answers were manually reviewed by 2 authors (ZPV and DMA), classified, coded by the common themes, and analyzed for comparison.
Results
A total of 166 participants completed the predemonstration survey, which was a requirement for participating in the AR demonstration. Of those, 159 staff members (95.8%) also completed at least part of the postdemonstration paired structured questions, and their results were included in the analysis.
Paired Questions
For questions about how innovative the VA is, 108 of 152 participants (71.1%) provided higher scores after the demonstration, 42 (27.6%) had no change, and 2 (1.3%) provided decreased scores. The mean innovative score increased from 3.4 predemonstration to 4.5 postdemonstration on a Likert scale, which is a 1.1 point increase from predemonstration to postdemonstration (95% CI, 0.9- 1.2) or a 22% increase (95% CI, 18%-24%) (P < .001). Respondents level of excitement about VA also increased with 82 of 157 participants (52.2%) providing higher scores after the demonstration, 71 (45.2%) had no change, and 4 scores (2.5%) decreased. The predemonstration mean excitement score of 3.7 increased to 4.3 postdemonstration, which is a 0.6 point increase from before to after the demonstration (95% CI, 0.5-0.7) or a 12% increase (95% CI, 10%-14%) (P < .001). In the survey, 36 of 149 participants (24.2%) had higher scores for their expectation to continue working at VA postdemonstration, 109 (73.2%) had no change, and 4 scores (2.7%) decreased. The mean employee retention score increased from 4.2 predemonstration to 4.5 postdemonstration, which is a 0.3 point increase between pre/post (95% CI, 0.2-0.4) or a 6% increase (95% CI, 4%-8%) (P < .001)
The pre/post questions were analyzed using 1-way ANOVA by hospital department and VA tenure. The responses by department were not statistically significant. Of the 159 employees assessed, 101 respondents (63.5%) had 0 to 10 years VA tenure, 44 (27.7%) had 11 to 20 years, 10 (6.3%) had 21 to 30 years, and 4 (2.5%) had > 31 to 40 years. Length of VA tenure did not impact respondent excitement. Respondents opinions on innovation in the 0 to 10 year and the 11 to 20 year groups rose from 3.2 and 3.7 predemonstration to 4.3 and 4.6 postdemonstration, respectively (P < .001 for both statistical comparisons) (Table 2). Interestingly, the 0 to 10 group saw a 9% rise from a 4.0 score predemonstration to a 4.4 score postdemonstration (P < .001), indicating that the demonstration had a positive impact on their plans to continue employment at VA (Table 3).
Sex did not play a significant role in how respondents answered questions regarding VA excitement or innovation. However, there was a statistically significant difference in how male and female respondents answered the predemonstration question about their plans to continue VA employment, according to the Wilcoxon rank sum test. Predemonstration, female respondents had a mean score of 4.1, which was 6% lower than the 4.4 score of male colleagues (P = .04). Veteran status did have an impact on how respondents felt about VA innovation, and their plans to continue employment at VA. After the demonstration, veteran staff felt the VA was more innovative compared with nonveterans: 4.7 vs 4.4, respectively, a 6% difference (P = .02) Similarly, for the continued VA employment question, veterans had a mean score of 4.8 vs 4.4 for nonveterans, an 8% difference (P = .03) These results suggest that the demonstration had more of an impact on veteran employees vs nonveteran employees.
Unpaired Questions
There were 2 structured unpaired postdemonstration questions. Respondents agreed that similar technology would impact veteran health care with mean (SD) of 4.6 (0.6) and a median score of 5 on a 5-point Likert scale. Respondents also agreed on the importance of implementing similar innovations with mean (SD) of 4.7 (0.5), and a median score of 5.
The survey asked how this technology could benefit their hospital service department and had 64 responses. Forty-six respondents saw applications for education or patient care/surgery. Other responses shared excitement about the technology and its potential to positively impact patient education. There were 37 responses to the open-ended question: 21 respondents expressed excitement for the technology, and 10 respondents reiterated that the demonstration would be of benefit to patient care/surgery and training.
Discussion
Successful development, design, and deployment of any new health care tool depends on leveraging insights from the employees that will be using and supporting these systems. Correspondingly, understanding the impact that advanced technologies have on health care employees’ satisfaction, morale, and retention is critical to our overall institutional strategy. Our findings show that a one-time experience with AR/MR technology elicited positive employee reactions. Of note, the survey revealed statistically significant improvements in staff’s view of the VA, with the greatest positive impact for questions about innovation, followed by excitement to work at the VA, and likelihood to continue work at the VA. It is very disruptive and costly when health care employees leave, and improving employee satisfaction and morale is important for better patient care and patient satisfaction, which is priority for VAPAHCS leadership.57-62
The paired predemonstration and postdemonstration scores were similarly high, nearing the top threshold available for the Likert scale (4.3 to 4.5). Furthermore, the least incremental improvement for these responses was observed for topics that had the highest initial baseline score. Therefore, the improvements observed for the paired questions may have more to do with the high baseline values.
Of additional interest, the self-reported likelihood of continuing to work at the VA increased the most for female employees, veteran employees, and employees with the least number of years at the VA. These demographic differences have important implications for VA staff recruitment and retention strategies.62 The unpaired questions about the impact on veteran care and whether the VA should continue similar work demonstrated extremely high support with median scores of 5 for both questions. The free-text postdemonstration responses also demonstrate similar positive themes, with a disposition for excitement about both the training and patient care applications for this technology. In addition, respondents felt strongly that this and other similar technologies will positively impact the health care for veterans and that the VA should continue these efforts.
Strengths and Limitations
A strength of this assessment is the ability to evaluate survey responses that were systematically collected and matched from the same individual immediately before and after exposure to the new technology. The free-text responses provided additional important information that both confirmed the results and provided additional valued supplementary guidance for future implementation strategies, which is critical for our translational implementation goals. An additional strength is that the voluntary surveys were managed by non-VAPAHCS colleagues, limiting potential bias. Importantly, the number of respondents allowed a statistically significant assessment of important health care employee metrics. These results have emphasized how being part of an innovative organization, and the introduction of advanced AR/MR technology, improve employees’ satisfaction and morale about where they work as well as their intention to stay at their institution.
A limitation of this assessment was the lack of comparative data for employee acceptance of other technologies at VAPAHCS. This limits our ability to differentiate whether the strong positive results observed in this evaluation were a result of the specific technology assessed, or of new and advanced health care technology in general. Nonetheless, our unpaired questions, which received extremely high scores, also included participant questions about comparing the system with other similar technologies. This assessment was also focused on veteran care, which limits generalizability.
Conclusions
One-time exposure to advanced AR technology for health care significantly increased employee morale as measured by excitement about working at the VA as well as employee intention to continue employment at the VA. These collateral benefits of the technology are particularly important in health care because our employees are our most important asset and improving employee morale equates to better patient care. Positive impacts were most pronounced for women employees, newer VA employees, and employees who are also veterans. These more detailed insights are also positioned to have a direct impact on employee recruitment and retention strategies. Additional valuable insights regarding the most applicable use of the technology in the clinical setting were also obtained.
Acknowledgments
We thank Andrew Spiegelman, Hyewon Kim, Jonathan Sills, and Alexander Erickson for their assistance in developing the survey questions. We also thank Jason Rhodes and Mark Bulson for traveling to our facility to assist with managing the anonymous surveys during the demonstration event.
1. World Economic Forum. Health and healthcare in the fourth industrial revolution: Global Future Council on the future of health and healthcare 2016-2018. April 2019. Accessed January 27, 2023. https://www3.weforum.org/docs/WEF__Shaping_the_Future_of_Health_Council_Report.pdf
2. Iveroth E, Fryk P, Rapp B. Information technology strategy and alignment issues in health care organizations. Health Care Manage Rev. 2013;38(3):188-200. doi:10.1097/HMR.0b013e31826119d7
3. Thakur R, Hsu SH, Fontenot G. Innovation in healthcare: issues and future trends. J Bus Res. 2012;65(4):562-569. doi:10.1016/j.jbusres.2011.02.022
4. Thimbleby H. Technology and the future of healthcare. J Public Health Res. 2013;2(3):e28. Published 2013 Dec 1. doi:10.4081/jphr.2013.e28
5. Viglialoro RM, Condino S, Turini G, Carbone M, Ferrari V, Gesi M. augmented reality, mixed reality, and hybrid approach in healthcare simulation: a systematic review. Applied Sciences. 2021;11(5):2338. doi:10.3390/app11052338
6. Rawlins CR, Veigulis Z, Hebert C, Curtin C, Osborne T. Effect of immersive virtual reality on pain and anxiety at a Veterans Affairs health care facility. Front Virt Real. 2021;(2):136. doi:10.3389/frvir.2021.719681
7. Chawdhary G, Shoman N. Emerging artificial intelligence applications in otological imaging. Curr Opin Otolaryngol Head Neck Surg. 2021;29(5):357-364. doi:10.1097/MOO.0000000000000754
8. Asadzadeh A, Samad-Soltani T, Rezaei-Hachesu P. Applications of virtual and augmented reality in infectious disease epidemics with a focus on the COVID-19 outbreak. Inform Med Unlocked. 2021;24:100579. doi:10.1016/j.imu.2021.100579
9. Ashwini KB, Savitha R, Harish A. Application of augmented reality technology for home healthcare product visualization. ECS Transas. 2022;107(1):10921. doi:10.1149/10701.10921ecst
10. Brooks AL. VR/Technologies for Rehabilitation. In: Brooks AL, Brahman S, Kapralos B, Nakajima A, Tyerman J, Jain LC, eds. Recent Advances in Technologies for Inclusive Well-Being Virtual Patients, Gamification and Simulation. Intelligent Systems Reference Library. Springer; 2021:241-252. doi:10.1007/978-3-030-59608-8_13
11. Koulouris D, Menychtas A, Maglogiannis I. An IoT-enabled platform for the assessment of physical and mental activities utilizing augmented reality exergaming. Sensors (Basel). 2022;22(9):3181. Published 2022 Apr 21. doi:10.3390/s22093181
12. Deiss YR, Korkut S, Inglese T. Increase therapy understanding and medication adherence for patients with inflammatory skin diseases through augmented reality. Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022. 2022:21-40. doi:10.1007/978-3-031-06018-2_2
13. Bertino E, Gao W, Steffan B, et al, eds. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer; 2022:21-40.
14. Ruhaiyem NIR, Mazlan NA. Image Modeling Through Augmented Reality for Skin Allergies Recognition. Lecture Notes on Data Engineering and Communications Technologies. 2021:72-79. doi: 10.1007/978-3-030-70713-2_8
15. Park BJ, Perkons NR, Profka E, et al. Three-dimensional augmented reality visualization informs locoregional therapy in a translational model of hepatocellular carcinoma. J Vasc Interv Radiol. 2020;31(10):1612-1618.e1. doi:10.1016/j.jvir.2020.01.028
16. Leo J, Zhou Z, Yang H, et al, eds. Interactive cardiovascular surgical planning via augmented reality. 5th Asian CHI Symposium 2021; 2021. doi:10.1145/3429360.3468195
17. Zuo Y, Jiang T, Dou J, et al. A novel evaluation model for a mixed-reality surgical navigation system: where Microsoft Hololens meets the operating room. Surg Innov. 2020;27(2):193-202. doi:10.1177/1553350619893236
18. Ghaednia H, Fourman MS, Lans A, et al. Augmented and virtual reality in spine surgery, current applications and future potentials. Spine J. 2021;21(10):1617-1625. doi:10.1016/j.spinee.2021.03.018
19. Liu Y, Lee MG, Kim JS. Spine surgery assisted by augmented reality: where have we been?. Yonsei Med J. 2022;63(4):305-316. doi:10.3349/ymj.2022.63.4.305
20. Kimmel S, Cobus V, Heuten W, eds. opticARe—augmented reality mobile patient monitoring in intensive care units. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST; 2021. doi:10.1145/3489849.3489852
21. Voštinár P, Horváthová D, Mitter M, Bako M. The look at the various uses of VR. Open Computer Sci. 2021;11(1):241-250. doi:10.1515/comp-2020-0123
22. Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: a meta-analysis of randomized controlled studies. BMC Med Educ. 2020;20(1):127. Published 2020 Apr 25. doi:10.1186/s12909-020-1994-z
23. Ricci S, Calandrino A, Borgonovo G, Chirico M, Casadio M. Viewpoint: virtual and augmented reality in basic and advanced life support training. JMIR Serious Games. 2022;10(1):e28595. Published 2022 Mar 23. doi:10.2196/28595
24. Ricci S, Mobilio GA, Calandrino A, et al. RiNeo MR: A mixed-reality tool for newborn life support training. Annu Int Conf IEEE Eng Med Biol Soc. 2021;2021:5043-5046. doi:10.1109/EMBC46164.2021.9629612
25. Dhar P, Rocks T, Samarasinghe RM, Stephenson G, Smith C. Augmented reality in medical education: students’ experiences and learning outcomes. Med Educ Online. 2021;26(1):1953953. doi:10.1080/10872981.2021.1953953
26. Pears M, Konstantinidis S. The future of immersive technology in global surgery education [published online ahead of print, 2021 Jul 1]. Indian J Surg. 2021;84(suppl 1):1-5. doi:10.1007/s12262-021-02998-6
27. Liang CJ, Start C, Boley H, Kamat VR, Menassa CC, Aebersold M. Enhancing stroke assessment simulation experience in clinical training using augmented reality. Virt Real. 2021;25(3):575-584. doi:10.1007/s10055-020-00475-1
28. Lacey G, Gozdzielewska L, McAloney-Kocaman K, Ruttle J, Cronin S, Price L. Psychomotor learning theory informing the design and evaluation of an interactive augmented reality hand hygiene training app for healthcare workers. Educ Inf Technol. 2022;27(3):3813-3832. doi:10.1007/s10639-021-10752-4
29. Ryan GV, Callaghan S, Rafferty A, Higgins MF, Mangina E, McAuliffe F. Learning outcomes of immersive technologies in health care student education: systematic review of the literature. J Med Internet Res. 2022;24(2):e30082. Published 2022 Feb 1. doi:10.2196/30082
30. Yu FU, Yan HU, Sundstedt V. A Systematic literature review of virtual, augmented, and mixed reality game applications in healthcare. ACM Trans Comput Healthcare. 2022;3(2);1-27. doi:10.1145/3472303
31. Weeks JK, Amiel JM. Enhancing neuroanatomy education with augmented reality. Med Educ. 2019;53(5):516-517. doi:10.1111/medu.13843
32. Williams MA, McVeigh J, Handa AI, Lee R. Augmented reality in surgical training: a systematic review. Postgrad Med J. 2020;96(1139):537-542. doi:10.1136/postgradmedj-2020-137600
33. Triepels CPR, Smeets CFA, Notten KJB, et al. Does three-dimensional anatomy improve student understanding? Clin Anat. 2020;33(1):25-33. doi:10.1002/ca.23405
34. Pietruski P, Majak M, S´wia¸tek-Najwer E, et al. Supporting fibula free flap harvest with augmented reality: A proof-of-concept study. Laryngoscope. 2020;130(5):1173-1179. doi:10.1002/lary.28090
35. Perkins SL, Krajancich B, Yang CJ, Hargreaves BA, Daniel BL, Berry MF. A patient-specific mixed-reality visualization tool for thoracic surgical planning. Ann Thorac Surg. 2020;110(1):290-295. doi:10.1016/j.athoracsur.2020.01.060
36. Müller F, Roner S, Liebmann F, Spirig JM, Fürnstahl P, Farshad M. Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging. Spine J. 2020;20(4):621-628. doi:10.1016/j.spinee.2019.10.012
37. Kaplan AD, Cruit J, Endsley M, Beers SM, Sawyer BD, Hancock PA. The effects of virtual reality, augmented reality, and mixed reality as training enhancement methods: a meta-analysis. Hum Factors. 2021;63(4):706-726. doi:10.1177/0018720820904229
38. Jud L, Fotouhi J, Andronic O, et al. Applicability of augmented reality in orthopedic surgery - a systematic review. BMC Musculoskelet Disord. 2020;21(1):103. Published 2020 Feb 15. doi:10.1186/s12891-020-3110-2
39. Ara J, Karim FB, Alsubaie MSA, et al. Comprehensive analysis of augmented reality technology in modern healthcare system. Int J Adv Comput Sci Appl. 2021;12(6):845-854. doi:10.14569/IJACSA.2021.0120698
40. Webster A, Gardner J. Aligning technology and institutional readiness: the adoption of innovation. Technol Anal Strateg Manag. 2019;31(10):1229-1241. doi:10.1080/09537325.2019.1601694
41. Hastall MR, Dockweiler C, Mühlhaus J. achieving end user acceptance: building blocks for an evidence-based user-centered framework for health technology development and assessment. In: Antona, M, Stephanidis C, eds. Universal Access in Human–Computer Interaction. Human and Technological Environments. UAHCI 2017. Lecture Notes in Computer Science, vol 10279. Springer, Cham; 2017. doi:10.1007/978-3-319-58700-4_2
42. Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc. 2015;22(6):1179-1182. doi:10.1093/jamia/ocv050
43. Khairat S, Coleman C, Ottmar P, Jayachander DI, Bice T, Carson SS. Association of Electronic Health Record Use With Physician Fatigue and Efficiency. JAMA Netw Open. 2020;3(6):e207385. Published 2020 Jun 1. doi:10.1001/jamanetworkopen.2020.7385
44. Melnick ER, Dyrbye LN, Sinsky CA, et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc. 2020;95(3):476-487. doi:10.1016/j.mayocp.2019.09.024
45. Lee YJ. Comparison of job satisfaction between nonprofit and public employees. Nonprofit Volunt Sect Q. 2016;45(2):295-313. doi:10.1177/0899764015584061
46. Brimhall KC. Inclusion is important... but how do I include? Examining the effects of leader engagement on inclusion, innovation, job satisfaction, and perceived quality of care in a diverse nonprofit health care organization. Nonprofit Volunt Sect Q. 2019;48(4):716-737. doi:10.1177/0899764019829834
47. Moreira MR, Gherman M, Sousa PS. Does innovation influence the performance of healthcare organizations?. Innovation (North Syd). 2017;19(3):335-352. doi:10.1080/14479338.2017.1293489
48. US Department of Veterans Affairs. VA Palo Alto Healthcare System. Updated December 29, 2020. Accessed January 27, 2023. https://www.paloalto.va.gov/about/index.asp
49. Siegel SM, Kaemmerer WF. Measuring the perceived support for innovation in organizations. J Appl Psychol. 1978;63(5):553-562. doi:10.1037/0021-9010.63.5.553
50. Anderson NR, West MA. Measuring climate for work group innovation: development and validation of the team climate inventory. J Organ Behav. 1998;19(3):235-258. doi:10.1002/(SICI)1099-1379(199805)19:3<235::AID-JOB837>3.0.CO;2-C
51. Aarons GA. Measuring provider attitudes toward evidence-based practice: consideration of organizational context and individual differences. Child Adolesc Psychiatr Clin N Am. 2005;14(2):255-viii. doi:10.1016/j.chc.2004.04.008
52. Van der Heijden H. User acceptance of hedonic information systems. MIS Q. 2004;28(4):695-704. doi:10.2307/25148660
53. Venkatesh V, Speier C, Morris MG. User acceptance enablers in individual decision making about technology: Toward an integrated model. Decis Sci. 2002;33(2):297-316. doi:10.1111/j.1540-5915.2002.tb01646.x
54. Puri A, Kim B, Nguyen O, Stolee P, Tung J, Lee J. User acceptance of wrist-worn activity trackers among community-dwelling older adults: mixed method study. JMIR Mhealth Uhealth. 2017;5(11):e173. Published 2017 Nov 15. doi:10.2196/mhealth.8211
55. Huang YC, Backman SJ, Backman KF, Moore D. Exploring user acceptance of 3D virtual worlds in travel and tourism marketing. Tourism Management. 2013;36:490-501. doi:10.1016/j.tourman.2012.09.009
56. Rasimah CM, Ahmad A, Zaman HB. Evaluation of user acceptance of mixed reality technology. AJET. 2011;27(8). doi:10.14742/ajet.899
57. Choi J, Boyle DK. RN workgroup job satisfaction and patient falls in acute care hospital units. J Nurs Adm. 2013;43(11):586-591. doi:10.1097/01.NNA.0000434509.66749.7c58. Tzeng HM, Ketefian S. The relationship between nurses’ job satisfaction and inpatient satisfaction: an exploratory study in a Taiwan teaching hospital. J Nurs Care Qual. 2002;16(2):39-49. doi:10.1097/00001786-200201000-00005
59. Williams ES, Skinner AC. Outcomes of physician job satisfaction: a narrative review, implications, and directions for future research. Health Care Manage Rev. 2003;28(2):119-139. doi:10.1097/00004010-200304000-00004
60. Waldman JD, Kelly F, Arora S, Smith HL. The shocking cost of turnover in health care. Health Care Manage Rev. 2004;29(1):2-7. doi:10.1097/00004010-200401000-00002
61. Hayes LJ, O’Brien-Pallas L, Duffield C, et al. Nurse turnover: a literature review - an update. Int J Nurs Stud. 2012;49(7):887-905. doi:10.1016/j.ijnurstu.2011.10.001
62. US Department of Veterans Affairs. FY 2021/FY 2019 Annual performance plan and report. February 2020. Accessed January 27, 2023. https://www.va.gov/oei/docs/VA2019-2021appr.pdf
1. World Economic Forum. Health and healthcare in the fourth industrial revolution: Global Future Council on the future of health and healthcare 2016-2018. April 2019. Accessed January 27, 2023. https://www3.weforum.org/docs/WEF__Shaping_the_Future_of_Health_Council_Report.pdf
2. Iveroth E, Fryk P, Rapp B. Information technology strategy and alignment issues in health care organizations. Health Care Manage Rev. 2013;38(3):188-200. doi:10.1097/HMR.0b013e31826119d7
3. Thakur R, Hsu SH, Fontenot G. Innovation in healthcare: issues and future trends. J Bus Res. 2012;65(4):562-569. doi:10.1016/j.jbusres.2011.02.022
4. Thimbleby H. Technology and the future of healthcare. J Public Health Res. 2013;2(3):e28. Published 2013 Dec 1. doi:10.4081/jphr.2013.e28
5. Viglialoro RM, Condino S, Turini G, Carbone M, Ferrari V, Gesi M. augmented reality, mixed reality, and hybrid approach in healthcare simulation: a systematic review. Applied Sciences. 2021;11(5):2338. doi:10.3390/app11052338
6. Rawlins CR, Veigulis Z, Hebert C, Curtin C, Osborne T. Effect of immersive virtual reality on pain and anxiety at a Veterans Affairs health care facility. Front Virt Real. 2021;(2):136. doi:10.3389/frvir.2021.719681
7. Chawdhary G, Shoman N. Emerging artificial intelligence applications in otological imaging. Curr Opin Otolaryngol Head Neck Surg. 2021;29(5):357-364. doi:10.1097/MOO.0000000000000754
8. Asadzadeh A, Samad-Soltani T, Rezaei-Hachesu P. Applications of virtual and augmented reality in infectious disease epidemics with a focus on the COVID-19 outbreak. Inform Med Unlocked. 2021;24:100579. doi:10.1016/j.imu.2021.100579
9. Ashwini KB, Savitha R, Harish A. Application of augmented reality technology for home healthcare product visualization. ECS Transas. 2022;107(1):10921. doi:10.1149/10701.10921ecst
10. Brooks AL. VR/Technologies for Rehabilitation. In: Brooks AL, Brahman S, Kapralos B, Nakajima A, Tyerman J, Jain LC, eds. Recent Advances in Technologies for Inclusive Well-Being Virtual Patients, Gamification and Simulation. Intelligent Systems Reference Library. Springer; 2021:241-252. doi:10.1007/978-3-030-59608-8_13
11. Koulouris D, Menychtas A, Maglogiannis I. An IoT-enabled platform for the assessment of physical and mental activities utilizing augmented reality exergaming. Sensors (Basel). 2022;22(9):3181. Published 2022 Apr 21. doi:10.3390/s22093181
12. Deiss YR, Korkut S, Inglese T. Increase therapy understanding and medication adherence for patients with inflammatory skin diseases through augmented reality. Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022. 2022:21-40. doi:10.1007/978-3-031-06018-2_2
13. Bertino E, Gao W, Steffan B, et al, eds. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer; 2022:21-40.
14. Ruhaiyem NIR, Mazlan NA. Image Modeling Through Augmented Reality for Skin Allergies Recognition. Lecture Notes on Data Engineering and Communications Technologies. 2021:72-79. doi: 10.1007/978-3-030-70713-2_8
15. Park BJ, Perkons NR, Profka E, et al. Three-dimensional augmented reality visualization informs locoregional therapy in a translational model of hepatocellular carcinoma. J Vasc Interv Radiol. 2020;31(10):1612-1618.e1. doi:10.1016/j.jvir.2020.01.028
16. Leo J, Zhou Z, Yang H, et al, eds. Interactive cardiovascular surgical planning via augmented reality. 5th Asian CHI Symposium 2021; 2021. doi:10.1145/3429360.3468195
17. Zuo Y, Jiang T, Dou J, et al. A novel evaluation model for a mixed-reality surgical navigation system: where Microsoft Hololens meets the operating room. Surg Innov. 2020;27(2):193-202. doi:10.1177/1553350619893236
18. Ghaednia H, Fourman MS, Lans A, et al. Augmented and virtual reality in spine surgery, current applications and future potentials. Spine J. 2021;21(10):1617-1625. doi:10.1016/j.spinee.2021.03.018
19. Liu Y, Lee MG, Kim JS. Spine surgery assisted by augmented reality: where have we been?. Yonsei Med J. 2022;63(4):305-316. doi:10.3349/ymj.2022.63.4.305
20. Kimmel S, Cobus V, Heuten W, eds. opticARe—augmented reality mobile patient monitoring in intensive care units. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST; 2021. doi:10.1145/3489849.3489852
21. Voštinár P, Horváthová D, Mitter M, Bako M. The look at the various uses of VR. Open Computer Sci. 2021;11(1):241-250. doi:10.1515/comp-2020-0123
22. Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: a meta-analysis of randomized controlled studies. BMC Med Educ. 2020;20(1):127. Published 2020 Apr 25. doi:10.1186/s12909-020-1994-z
23. Ricci S, Calandrino A, Borgonovo G, Chirico M, Casadio M. Viewpoint: virtual and augmented reality in basic and advanced life support training. JMIR Serious Games. 2022;10(1):e28595. Published 2022 Mar 23. doi:10.2196/28595
24. Ricci S, Mobilio GA, Calandrino A, et al. RiNeo MR: A mixed-reality tool for newborn life support training. Annu Int Conf IEEE Eng Med Biol Soc. 2021;2021:5043-5046. doi:10.1109/EMBC46164.2021.9629612
25. Dhar P, Rocks T, Samarasinghe RM, Stephenson G, Smith C. Augmented reality in medical education: students’ experiences and learning outcomes. Med Educ Online. 2021;26(1):1953953. doi:10.1080/10872981.2021.1953953
26. Pears M, Konstantinidis S. The future of immersive technology in global surgery education [published online ahead of print, 2021 Jul 1]. Indian J Surg. 2021;84(suppl 1):1-5. doi:10.1007/s12262-021-02998-6
27. Liang CJ, Start C, Boley H, Kamat VR, Menassa CC, Aebersold M. Enhancing stroke assessment simulation experience in clinical training using augmented reality. Virt Real. 2021;25(3):575-584. doi:10.1007/s10055-020-00475-1
28. Lacey G, Gozdzielewska L, McAloney-Kocaman K, Ruttle J, Cronin S, Price L. Psychomotor learning theory informing the design and evaluation of an interactive augmented reality hand hygiene training app for healthcare workers. Educ Inf Technol. 2022;27(3):3813-3832. doi:10.1007/s10639-021-10752-4
29. Ryan GV, Callaghan S, Rafferty A, Higgins MF, Mangina E, McAuliffe F. Learning outcomes of immersive technologies in health care student education: systematic review of the literature. J Med Internet Res. 2022;24(2):e30082. Published 2022 Feb 1. doi:10.2196/30082
30. Yu FU, Yan HU, Sundstedt V. A Systematic literature review of virtual, augmented, and mixed reality game applications in healthcare. ACM Trans Comput Healthcare. 2022;3(2);1-27. doi:10.1145/3472303
31. Weeks JK, Amiel JM. Enhancing neuroanatomy education with augmented reality. Med Educ. 2019;53(5):516-517. doi:10.1111/medu.13843
32. Williams MA, McVeigh J, Handa AI, Lee R. Augmented reality in surgical training: a systematic review. Postgrad Med J. 2020;96(1139):537-542. doi:10.1136/postgradmedj-2020-137600
33. Triepels CPR, Smeets CFA, Notten KJB, et al. Does three-dimensional anatomy improve student understanding? Clin Anat. 2020;33(1):25-33. doi:10.1002/ca.23405
34. Pietruski P, Majak M, S´wia¸tek-Najwer E, et al. Supporting fibula free flap harvest with augmented reality: A proof-of-concept study. Laryngoscope. 2020;130(5):1173-1179. doi:10.1002/lary.28090
35. Perkins SL, Krajancich B, Yang CJ, Hargreaves BA, Daniel BL, Berry MF. A patient-specific mixed-reality visualization tool for thoracic surgical planning. Ann Thorac Surg. 2020;110(1):290-295. doi:10.1016/j.athoracsur.2020.01.060
36. Müller F, Roner S, Liebmann F, Spirig JM, Fürnstahl P, Farshad M. Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging. Spine J. 2020;20(4):621-628. doi:10.1016/j.spinee.2019.10.012
37. Kaplan AD, Cruit J, Endsley M, Beers SM, Sawyer BD, Hancock PA. The effects of virtual reality, augmented reality, and mixed reality as training enhancement methods: a meta-analysis. Hum Factors. 2021;63(4):706-726. doi:10.1177/0018720820904229
38. Jud L, Fotouhi J, Andronic O, et al. Applicability of augmented reality in orthopedic surgery - a systematic review. BMC Musculoskelet Disord. 2020;21(1):103. Published 2020 Feb 15. doi:10.1186/s12891-020-3110-2
39. Ara J, Karim FB, Alsubaie MSA, et al. Comprehensive analysis of augmented reality technology in modern healthcare system. Int J Adv Comput Sci Appl. 2021;12(6):845-854. doi:10.14569/IJACSA.2021.0120698
40. Webster A, Gardner J. Aligning technology and institutional readiness: the adoption of innovation. Technol Anal Strateg Manag. 2019;31(10):1229-1241. doi:10.1080/09537325.2019.1601694
41. Hastall MR, Dockweiler C, Mühlhaus J. achieving end user acceptance: building blocks for an evidence-based user-centered framework for health technology development and assessment. In: Antona, M, Stephanidis C, eds. Universal Access in Human–Computer Interaction. Human and Technological Environments. UAHCI 2017. Lecture Notes in Computer Science, vol 10279. Springer, Cham; 2017. doi:10.1007/978-3-319-58700-4_2
42. Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc. 2015;22(6):1179-1182. doi:10.1093/jamia/ocv050
43. Khairat S, Coleman C, Ottmar P, Jayachander DI, Bice T, Carson SS. Association of Electronic Health Record Use With Physician Fatigue and Efficiency. JAMA Netw Open. 2020;3(6):e207385. Published 2020 Jun 1. doi:10.1001/jamanetworkopen.2020.7385
44. Melnick ER, Dyrbye LN, Sinsky CA, et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc. 2020;95(3):476-487. doi:10.1016/j.mayocp.2019.09.024
45. Lee YJ. Comparison of job satisfaction between nonprofit and public employees. Nonprofit Volunt Sect Q. 2016;45(2):295-313. doi:10.1177/0899764015584061
46. Brimhall KC. Inclusion is important... but how do I include? Examining the effects of leader engagement on inclusion, innovation, job satisfaction, and perceived quality of care in a diverse nonprofit health care organization. Nonprofit Volunt Sect Q. 2019;48(4):716-737. doi:10.1177/0899764019829834
47. Moreira MR, Gherman M, Sousa PS. Does innovation influence the performance of healthcare organizations?. Innovation (North Syd). 2017;19(3):335-352. doi:10.1080/14479338.2017.1293489
48. US Department of Veterans Affairs. VA Palo Alto Healthcare System. Updated December 29, 2020. Accessed January 27, 2023. https://www.paloalto.va.gov/about/index.asp
49. Siegel SM, Kaemmerer WF. Measuring the perceived support for innovation in organizations. J Appl Psychol. 1978;63(5):553-562. doi:10.1037/0021-9010.63.5.553
50. Anderson NR, West MA. Measuring climate for work group innovation: development and validation of the team climate inventory. J Organ Behav. 1998;19(3):235-258. doi:10.1002/(SICI)1099-1379(199805)19:3<235::AID-JOB837>3.0.CO;2-C
51. Aarons GA. Measuring provider attitudes toward evidence-based practice: consideration of organizational context and individual differences. Child Adolesc Psychiatr Clin N Am. 2005;14(2):255-viii. doi:10.1016/j.chc.2004.04.008
52. Van der Heijden H. User acceptance of hedonic information systems. MIS Q. 2004;28(4):695-704. doi:10.2307/25148660
53. Venkatesh V, Speier C, Morris MG. User acceptance enablers in individual decision making about technology: Toward an integrated model. Decis Sci. 2002;33(2):297-316. doi:10.1111/j.1540-5915.2002.tb01646.x
54. Puri A, Kim B, Nguyen O, Stolee P, Tung J, Lee J. User acceptance of wrist-worn activity trackers among community-dwelling older adults: mixed method study. JMIR Mhealth Uhealth. 2017;5(11):e173. Published 2017 Nov 15. doi:10.2196/mhealth.8211
55. Huang YC, Backman SJ, Backman KF, Moore D. Exploring user acceptance of 3D virtual worlds in travel and tourism marketing. Tourism Management. 2013;36:490-501. doi:10.1016/j.tourman.2012.09.009
56. Rasimah CM, Ahmad A, Zaman HB. Evaluation of user acceptance of mixed reality technology. AJET. 2011;27(8). doi:10.14742/ajet.899
57. Choi J, Boyle DK. RN workgroup job satisfaction and patient falls in acute care hospital units. J Nurs Adm. 2013;43(11):586-591. doi:10.1097/01.NNA.0000434509.66749.7c58. Tzeng HM, Ketefian S. The relationship between nurses’ job satisfaction and inpatient satisfaction: an exploratory study in a Taiwan teaching hospital. J Nurs Care Qual. 2002;16(2):39-49. doi:10.1097/00001786-200201000-00005
59. Williams ES, Skinner AC. Outcomes of physician job satisfaction: a narrative review, implications, and directions for future research. Health Care Manage Rev. 2003;28(2):119-139. doi:10.1097/00004010-200304000-00004
60. Waldman JD, Kelly F, Arora S, Smith HL. The shocking cost of turnover in health care. Health Care Manage Rev. 2004;29(1):2-7. doi:10.1097/00004010-200401000-00002
61. Hayes LJ, O’Brien-Pallas L, Duffield C, et al. Nurse turnover: a literature review - an update. Int J Nurs Stud. 2012;49(7):887-905. doi:10.1016/j.ijnurstu.2011.10.001
62. US Department of Veterans Affairs. FY 2021/FY 2019 Annual performance plan and report. February 2020. Accessed January 27, 2023. https://www.va.gov/oei/docs/VA2019-2021appr.pdf
