User login
3D Printing for the Development of Palatal Defect Prosthetics
Three-dimensional (3D) printing has become a promising area of innovation in biomedical research.1,2 Previous research in orthopedic surgery has found that customized 3D printed implants, casts, orthoses, and prosthetics (eg, prosthetic hands) matched to an individual’s unique anatomy can result in more precise placement and better surgical outcomes.3-5 Customized prosthetics have also been found to lead to fewer complications.3,6
Recent advances in 3D printing technology has prompted investigation from surgeons to identify how this new tool may be incorporated into patient care.1,7 One of the most common applications of 3D printing is during preoperative planning in which surgeons gain better insight into patient-specific anatomy by using patient-specific printed models.8 Another promising application is the production of customized prosthetics suited to each patient’s unique anatomy.9 As a result, 3D printing has significantly impacted bone and cartilage restoration procedures and has the potential to completely transform the treatment of patients with debilitating musculoskeletal injuries.3,10
The potential surrounding 3D printed prosthetics has led to their adoption by several other specialties, including otolaryngology.11 The most widely used application of 3D printing among otolaryngologists is preoperative planning, and the incorporation of printed prosthetics intoreconstruction of the orbit, nasal septum, auricle, and palate has also been reported.2,12,13 Patient-specific implants might allow otolaryngologists to better rehabilitate, reconstruct, and/or regenerate craniofacial defects using more humane procedures.14
Patients with palatomaxillary cancers are treated by prosthodontists or otolaryngologists. An impression is made with a resin–which can be painful for postoperative patients–and a prosthetic is manufactured and implanted.15-17 Patients with cancer often see many specialists, though reconstructive care is a low priority. Many of these individuals also experience dynamic anatomic functional changes over time, leading to the need for multiple prothesis.
palatomaxillary prosthetics
This program aims to use patients’ previous computed tomography (CT) to tailor customized 3D printed palatomaxillary prosthetics to specifically fit their anatomy. Palatomaxillary defects are a source of profound disability for patients with head and neck cancers who are left with large anatomic defects as a direct result of treatment. Reconstruction of palatal defects poses unique challenges due to the complexity of patient anatomy.18,19
3D printed prosthetics for palatomaxillary defects have not been incorporated into patient care. We reviewed previous imaging research to determine if it could be used to assist patients who struggle with their function and appearance following treatment for head and neck cancers. The primary aim was to investigate whether 3D printing was a feasible strategy for creating patient-specific palatomaxillary prosthetics. The secondary aim is to determine whether these prosthetics should be tested in the future for use in reconstruction of maxillary defects.
Data Acquisition
This study was conducted at the Veterans Affairs Palo Alto Health Care System (VAPAHCS) and was approved by the Stanford University Institutional Review Board (approval #28958, informed consent and patient contact excluded). A retrospective chart review was conducted on all patients with head and neck cancers who were treated at VAPAHCS from 2010 to 2022. Patients aged ≥ 18 years who had a palatomaxillary defect due to cancer treatment, had undergone a palatal resection, and who received treatment at any point from 2010 to 2022 were included in the review. CTs were not a specific inclusion criterion, though the quality of the scans was analyzed for eligible patients. Younger patients and those treated at VAPAHCS prior to 2010 were excluded.
There was no control group; all data was sourced from the US Department of Veterans Affairs (VA) imaging system database. Among the 3595 patients reviewed, 5 met inclusion criteria and the quality of their craniofacial anatomy CTs were analyzed. To maintain accurate craniofacial 3D modeling, CTs require a maximum of 1 mm slice thickness. Of the 5 patients who met the inclusion criteria, 4 were found to have variability in the quality of their CTs and severe defects not suitable for prosthetic reconstruction, which led to their exclusion from the study. One patient was investigated to demonstrate if making these prostheses was feasible. This patient was diagnosed with a malignant neoplasm of the hard palate, underwent a partial maxillectomy, and a palatal obturator was placed to cover the defect.
The primary data collected was patient identifiers as well as the gross anatomy and dimensions of the patients’ craniofacial anatomy, as seen in previous imaging research.20 Before the imaging analysis, all personal health information was removed and the dataset was deidentified to ensure patient anonymity and noninvolvement.
CT Segmentation and 3D Printing
Using CTs of the patient’s craniofacial anatomy, we developed a model of the defects. This was achieved with deidentified CTs imported into the Food and Drug Administration (FDA)-approved computerized aid design (CAD) software, Materialise Mimics. The hard palate was segmented and isolated based off the presented scan and any holes in the image were filled using the CAD software. The model was subsequently mirrored in Materialise 3-matic to replicate an original anatomical hard palate prosthesis. The final product was converted into a 3D model and imported into Formlabs preform software to generate 3D printing supports and orient it for printing. The prosthetic was printed using FDA-approved Biocompatible Denture Base Resin by a Formlabs 3B+ printer at the Palo Alto VA Simulation Center. The 3D printed prosthesis was washed using Formlabs Form Wash 80% ethyl alcohol to remove excess resin and subsequently cured to harden the malleable resin. Supports were later removed, and the prosthesis was sanded.
The primary aim of this study was to investigate whether using CTs to create patient-specific prosthetic renderings for patients with head and neck cancer could be a feasible strategy. The CTs from the patient were successfully used to generate a 3D printed prosthesis, and the prosthesis matched the original craniofacial anatomy seen in the patient's imaging (Figure). These results demonstrate that high quality CTs can be used as a template for 3D printed prostheses for mild to moderate palatomaxillary defects.
3D Printing Costs
One liter of Denture Base Resin costs $299; prostheses use about 5 mL of resin. The average annual salary of a 3D printing technician in the United States is $42,717, or $20.54 per hour.21 For an experienced 3D printing technician, the time required to segment the hard palate and prepare it for 3D printing is 1 to 2 hours. The process may exceed 2 hours if the technician is presented with a lower quality CT or if the patient has a complex craniofacial anatomy.
The average time it takes to print a palatal prosthetic is 5 hours. An additional hour is needed for postprocessing, which includes washing and sanding. Therefore, the cost of the materials and labor for an average 3D printed prosthetic is about $150. A Formlabs 3B+ printer is competitively priced around $10,000. The cost for Materialise Mimics software varies, but is estimated at $16,000 at VAPAHCS. The prices for these 2 items are not included in our price estimation but should be taken into consideration.
Prosthodontist Process and Cost
The typical process of creating a palatal prosthesis by a prosthodontist begins by examining the patient, creating a stone model, then creating a wax model. Biocompatible materials are selected and processed into a mold that is trimmed and polished to the desired shape. This is followed by another patient visit to ensure the prosthesis fits properly. Follow-up care is also necessary for maintenance and comfort.
The average cost of a palatal prosthesis varies depending on the type needed (ie, metal implant, teeth replacement), the materials used, the region in which the patient is receiving care, and the complexity of the case. For complex and customizable options like those required for patients with cancer, the prostheses typically cost several thousands of dollars. The Healthcare Common Procedure Coding System code for a palatal lift prosthesis (D5955) lists prices ranging from $4000 to $8000 per prosthetic, not including the cost of the prosthodontist visits.22,23
Discussion
This program sought to determine whether imaging studies of maxillary defects are effective templates for developing 3D printed prosthetics and whether these prosthetics should be tested for future use in reconstruction of palatomaxillary defects. Our program illustrated that CTs served as feasible templates for developing hard palate prostheses for patients with palatomaxillary defects. It is important to note the CTs used were from a newer and more modern scanner and therefore yielded detailed palatal structures with higher accuracy more suitable for 3D modeling. Lower-quality CTs from the 4 patients excluded from the program were not suitable for 3D modeling. This suggests that with high-quality imaging, 3D printed prosthesis may be a viable strategy to help patients who struggle with their function following treatment for head and neck cancers.
3D printed prosthesis may also be a more patient centered and convenient option. In the traditional prosthesis creation workflow, the patient must physically bite down onto a resin (alginate or silicone) to make an impression, a very painful postoperative process that is irritating to the raw edges of the surgical bed.15,16 Prosthodontists then create a prosthetic minus the tumor and typically secure it with clips or glue.17 Many patients also experience changes in their anatomy over time requiring them to have a new protheses created. This is particularly important in veterans with palatomaxillary defects since many VA medical centers do not have a prosthodontist on staff, making accessibility to these specialists difficult. 3D printing provides a contactless prosthetic creation process. This convenience may reduce a patient’s pain and the number of visits for which they need a specialist.
Future Directions
Additional research is needed to determine the full potential of 3D printed prosthetics. 3D printed prostheses have been effectively used for patient education in areas of presurgical planning, prosthesis creation, and trainee education.24 This research represents an early step in the development of a new technology for use in otolaryngology. Specifically, many veterans with a history of head and neck cancers have sustained changes to their craniofacial anatomy following treatment. Using imaging to create 3D printed prosthetics could be very effective for these patients. Prosthetics could improve a patient’s quality of life by restoring/approximating their anatomy after cancer treatment.
Significant time and care must be taken by cancer and reconstructive surgeons to properly fit a prosthesis. Improperly fitting prosthetics leads to mucosal ulceration that then may lead to a need for fitting a new prosthetic. The advantage of 3D printed prosthetics is that they may more precisely fit the anatomy of each patient using CT results, thus potentially reducing the time needed to fit the prosthetic as well as the risk associated with an improperly fit prosthetic. 3D printed prosthesis could be used directly in the future, however, clinical trials are needed to verify its efficacy vs prosthodontic options.
Another consideration for potential future use of 3D printed prosthetics is cost. We estimated that the cost of the materials and labor of our 3D printed prosthetic to be about $150. Pricing of current molded prosthetics varies, but is often listed at several thousand dollars. Another consideration is the durability of 3D printed prosthetics vs standard prosthetics. Since we were unable to use the prosthetic in the patient, it was difficult to determine its durability. The significant cost of the 3D printer and software necessary for 3D printed prosthetics must also be considered and may be prohibitive. While many academic hospitals are considering the purchase of 3D printers and licenses, this may be challenging for resource-constrained institutions. 3D printing may also be difficult for groups without any prior experience in the field. Outsourcing to a third party is possible, though doing so adds more cost to the project. While we recognize there is a learning curve associated with adopting any new technology, it’s equally important to note that 3D printing is being rapidly integrated and has already made significant advancements in personalized medicine.8,25,26
Limitations
This program had several limitations. First, we only obtained CTs of sufficient quality from 1 patient to generate a 3D printed prosthesis. Further research with additional patients is necessary to validate this process. Second, we were unable to trial the prosthesis in the patient because we did not have FDA approval. Additionally, it is difficult to calculate a true cost estimate for this process as materials and software costs vary dramatically across institutions as well as over time.
Conclusions
The purpose of this study was to demonstrate the possibility to develop prosthetics for the hard palate for patients suffering from palatomaxillary defects. A 3D printed prosthetic was generated that matched the patient’s craniofacial anatomy. Future research should test the feasibility of these prosthetics in patient care against a traditional prosthodontic impression. Though this is a proof-of-concept study and no prosthetics were implanted as part of this investigation, we showcase the feasibility of printing prosthetics for palatomaxillary defects. The use of 3D printed prosthetics may be a more humane process, potentially lower cost, and be more accessible to veterans.
1. Crafts TD, Ellsperman SE, Wannemuehler TJ, Bellicchi TD, Shipchandler TZ, Mantravadi AV. Three-dimensional printing and its applications in otorhinolaryngology-head and neck surgery. Otolaryngol Head Neck Surg. 2017;156(6):999-1010. doi:10.1177/0194599816678372
2. Virani FR, Chua EC, Timbang MR, Hsieh TY, Senders CW. Three-dimensional printing in cleft care: a systematic review. Cleft Palate Craniofac J. 2022;59(4):484-496. doi:10.1177/10556656211013175
3. Lal H, Patralekh MK. 3D printing and its applications in orthopaedic trauma: A technological marvel. J Clin Orthop Trauma. 2018;9(3):260-268. doi:10.1016/j.jcot.2018.07.022
4. Vujaklija I, Farina D. 3D printed upper limb prosthetics. Expert Rev Med Devices. 2018;15(7):505-512. doi:10.1080/17434440.2018.1494568
5. Ten Kate J, Smit G, Breedveld P. 3D-printed upper limb prostheses: a review. Disabil Rehabil Assist Technol. 2017;12(3):300-314. doi:10.1080/17483107.2016.1253117
6. Thomas CN, Mavrommatis S, Schroder LK, Cole PA. An overview of 3D printing and the orthopaedic application of patient-specific models in malunion surgery. Injury. 2022;53(3):977-983. doi:10.1016/j.injury.2021.11.019
7. Colaco M, Igel DA, Atala A. The potential of 3D printing in urological research and patient care. Nat Rev Urol. 2018;15(4):213-221. doi:10.1038/nrurol.2018.6
8. Meyer-Szary J, Luis MS, Mikulski S, et al. The role of 3D printing in planning complex medical procedures and training of medical professionals-cross-sectional multispecialty review. Int J Environ Res Public Health. 2022;19(6):3331. Published 2022 Mar 11. doi:10.3390/ijerph19063331
9. Moya D, Gobbato B, Valente S, Roca R. Use of preoperative planning and 3D printing in orthopedics and traumatology: entering a new era. Acta Ortop Mex. 2022;36(1):39-47.
10. Wixted CM, Peterson JR, Kadakia RJ, Adams SB. Three-dimensional printing in orthopaedic surgery: current applications and future developments. J Am Acad Orthop Surg Glob Res Rev. 2021;5(4):e20.00230-11. Published 2021 Apr 20. doi:10.5435/JAAOSGlobal-D-20-00230
11. Hong CJ, Giannopoulos AA, Hong BY, et al. Clinical applications of three-dimensional printing in otolaryngology-head and neck surgery: a systematic review. Laryngoscope. 2019;129(9):2045-2052. doi:10.1002/lary.2783112. Sigron GR, Barba M, Chammartin F, Msallem B, Berg BI, Thieringer FM. Functional and cosmetic outcome after reconstruction of isolated, unilateral orbital floor fractures (blow-out fractures) with and without the support of 3D-printed orbital anatomical models. J Clin Med. 2021;10(16):3509. Published 2021 Aug 9. doi:10.3390/jcm10163509
13. Kimura K, Davis S, Thomas E, et al. 3D Customization for microtia repair in hemifacial microsomia. Laryngoscope. 2022;132(3):545-549. doi:10.1002/lary.29823
14. Nyberg EL, Farris AL, Hung BP, et al. 3D-printing technologies for craniofacial rehabilitation, reconstruction, and regeneration. Ann Biomed Eng. 2017;45(1):45-57. doi:10.1007/s10439-016-1668-5
15. Flores-Ruiz R, Castellanos-Cosano L, Serrera-Figallo MA, et al. Evolution of oral cancer treatment in an andalusian population sample: rehabilitation with prosthetic obturation and removable partial prosthesis. J Clin Exp Dent. 2017;9(8):e1008-e1014. doi:10.4317/jced.54023
16. Rogers SN, Lowe D, McNally D, Brown JS, Vaughan ED. Health-related quality of life after maxillectomy: a comparison between prosthetic obturation and free flap. J Oral Maxillofac Surg. 2003;61(2):174-181. doi:10.1053/joms.2003.50044
17. Pool C, Shokri T, Vincent A, Wang W, Kadakia S, Ducic Y. Prosthetic reconstruction of the maxilla and palate. Semin Plast Surg. 2020;34(2):114-119. doi:10.1055/s-0040-1709143
18. Badhey AK, Khan MN. Palatomaxillary reconstruction: fibula or scapula. Semin Plast Surg. 2020;34(2):86-91. doi:10.1055/s-0040-1709431
19. Jategaonkar AA, Kaul VF, Lee E, Genden EM. Surgery of the palatomaxillary structure. Semin Plast Surg. 2020;34(2):71-76. doi:10.1055/s-0040-1709430
20. Lobb DC, Cottler P, Dart D, Black JS. The use of patient-specific three-dimensional printed surgical models enhances plastic surgery resident education in craniofacial surgery. J Craniofac Surg. 2019;30(2):339-341. doi:10.1097/SCS.0000000000005322
21. 3D printing technician salary in the United States. Accessed February 27, 2024. https://www.salary.com/research/salary/posting/3d-printing-technician-salary22. US Dept of Veterans Affairs. Healthcare Common Procedure Coding System. Outpatient dental professional nationwide charges by HCPCS code. January-December 2020. Accessed February 27, 2024. https://www.va.gov/COMMUNITYCARE/docs/RO/Outpatient-DataTables/v3-27_Table-I.pdf23. Washington State Department of Labor and Industries. Professional services fee schedule HCPCS level II fees. October 1, 2020. Accessed February 27, 2024. https://lni.wa.gov/patient-care/billing-payments/marfsdocs/2020/2020FSHCPCS.pdf24. Low CM, Morris JM, Price DL, et al. Three-dimensional printing: current use in rhinology and endoscopic skull base surgery. Am J Rhinol Allergy. 2019;33(6):770-781. doi:10.1177/1945892419866319
25. Aimar A, Palermo A, Innocenti B. The role of 3D printing in medical applications: a state of the art. J Healthc Eng. 2019;2019:5340616. Published 2019 Mar 21. doi:10.1155/2019/5340616
26. Garcia J, Yang Z, Mongrain R, Leask RL, Lachapelle K. 3D printing materials and their use in medical education: a review of current technology and trends for the future. BMJ Simul Technol Enhanc Learn. 2018;4(1):27-40. doi:10.1136/bmjstel-2017-000234
Three-dimensional (3D) printing has become a promising area of innovation in biomedical research.1,2 Previous research in orthopedic surgery has found that customized 3D printed implants, casts, orthoses, and prosthetics (eg, prosthetic hands) matched to an individual’s unique anatomy can result in more precise placement and better surgical outcomes.3-5 Customized prosthetics have also been found to lead to fewer complications.3,6
Recent advances in 3D printing technology has prompted investigation from surgeons to identify how this new tool may be incorporated into patient care.1,7 One of the most common applications of 3D printing is during preoperative planning in which surgeons gain better insight into patient-specific anatomy by using patient-specific printed models.8 Another promising application is the production of customized prosthetics suited to each patient’s unique anatomy.9 As a result, 3D printing has significantly impacted bone and cartilage restoration procedures and has the potential to completely transform the treatment of patients with debilitating musculoskeletal injuries.3,10
The potential surrounding 3D printed prosthetics has led to their adoption by several other specialties, including otolaryngology.11 The most widely used application of 3D printing among otolaryngologists is preoperative planning, and the incorporation of printed prosthetics intoreconstruction of the orbit, nasal septum, auricle, and palate has also been reported.2,12,13 Patient-specific implants might allow otolaryngologists to better rehabilitate, reconstruct, and/or regenerate craniofacial defects using more humane procedures.14
Patients with palatomaxillary cancers are treated by prosthodontists or otolaryngologists. An impression is made with a resin–which can be painful for postoperative patients–and a prosthetic is manufactured and implanted.15-17 Patients with cancer often see many specialists, though reconstructive care is a low priority. Many of these individuals also experience dynamic anatomic functional changes over time, leading to the need for multiple prothesis.
palatomaxillary prosthetics
This program aims to use patients’ previous computed tomography (CT) to tailor customized 3D printed palatomaxillary prosthetics to specifically fit their anatomy. Palatomaxillary defects are a source of profound disability for patients with head and neck cancers who are left with large anatomic defects as a direct result of treatment. Reconstruction of palatal defects poses unique challenges due to the complexity of patient anatomy.18,19
3D printed prosthetics for palatomaxillary defects have not been incorporated into patient care. We reviewed previous imaging research to determine if it could be used to assist patients who struggle with their function and appearance following treatment for head and neck cancers. The primary aim was to investigate whether 3D printing was a feasible strategy for creating patient-specific palatomaxillary prosthetics. The secondary aim is to determine whether these prosthetics should be tested in the future for use in reconstruction of maxillary defects.
Data Acquisition
This study was conducted at the Veterans Affairs Palo Alto Health Care System (VAPAHCS) and was approved by the Stanford University Institutional Review Board (approval #28958, informed consent and patient contact excluded). A retrospective chart review was conducted on all patients with head and neck cancers who were treated at VAPAHCS from 2010 to 2022. Patients aged ≥ 18 years who had a palatomaxillary defect due to cancer treatment, had undergone a palatal resection, and who received treatment at any point from 2010 to 2022 were included in the review. CTs were not a specific inclusion criterion, though the quality of the scans was analyzed for eligible patients. Younger patients and those treated at VAPAHCS prior to 2010 were excluded.
There was no control group; all data was sourced from the US Department of Veterans Affairs (VA) imaging system database. Among the 3595 patients reviewed, 5 met inclusion criteria and the quality of their craniofacial anatomy CTs were analyzed. To maintain accurate craniofacial 3D modeling, CTs require a maximum of 1 mm slice thickness. Of the 5 patients who met the inclusion criteria, 4 were found to have variability in the quality of their CTs and severe defects not suitable for prosthetic reconstruction, which led to their exclusion from the study. One patient was investigated to demonstrate if making these prostheses was feasible. This patient was diagnosed with a malignant neoplasm of the hard palate, underwent a partial maxillectomy, and a palatal obturator was placed to cover the defect.
The primary data collected was patient identifiers as well as the gross anatomy and dimensions of the patients’ craniofacial anatomy, as seen in previous imaging research.20 Before the imaging analysis, all personal health information was removed and the dataset was deidentified to ensure patient anonymity and noninvolvement.
CT Segmentation and 3D Printing
Using CTs of the patient’s craniofacial anatomy, we developed a model of the defects. This was achieved with deidentified CTs imported into the Food and Drug Administration (FDA)-approved computerized aid design (CAD) software, Materialise Mimics. The hard palate was segmented and isolated based off the presented scan and any holes in the image were filled using the CAD software. The model was subsequently mirrored in Materialise 3-matic to replicate an original anatomical hard palate prosthesis. The final product was converted into a 3D model and imported into Formlabs preform software to generate 3D printing supports and orient it for printing. The prosthetic was printed using FDA-approved Biocompatible Denture Base Resin by a Formlabs 3B+ printer at the Palo Alto VA Simulation Center. The 3D printed prosthesis was washed using Formlabs Form Wash 80% ethyl alcohol to remove excess resin and subsequently cured to harden the malleable resin. Supports were later removed, and the prosthesis was sanded.
The primary aim of this study was to investigate whether using CTs to create patient-specific prosthetic renderings for patients with head and neck cancer could be a feasible strategy. The CTs from the patient were successfully used to generate a 3D printed prosthesis, and the prosthesis matched the original craniofacial anatomy seen in the patient's imaging (Figure). These results demonstrate that high quality CTs can be used as a template for 3D printed prostheses for mild to moderate palatomaxillary defects.
3D Printing Costs
One liter of Denture Base Resin costs $299; prostheses use about 5 mL of resin. The average annual salary of a 3D printing technician in the United States is $42,717, or $20.54 per hour.21 For an experienced 3D printing technician, the time required to segment the hard palate and prepare it for 3D printing is 1 to 2 hours. The process may exceed 2 hours if the technician is presented with a lower quality CT or if the patient has a complex craniofacial anatomy.
The average time it takes to print a palatal prosthetic is 5 hours. An additional hour is needed for postprocessing, which includes washing and sanding. Therefore, the cost of the materials and labor for an average 3D printed prosthetic is about $150. A Formlabs 3B+ printer is competitively priced around $10,000. The cost for Materialise Mimics software varies, but is estimated at $16,000 at VAPAHCS. The prices for these 2 items are not included in our price estimation but should be taken into consideration.
Prosthodontist Process and Cost
The typical process of creating a palatal prosthesis by a prosthodontist begins by examining the patient, creating a stone model, then creating a wax model. Biocompatible materials are selected and processed into a mold that is trimmed and polished to the desired shape. This is followed by another patient visit to ensure the prosthesis fits properly. Follow-up care is also necessary for maintenance and comfort.
The average cost of a palatal prosthesis varies depending on the type needed (ie, metal implant, teeth replacement), the materials used, the region in which the patient is receiving care, and the complexity of the case. For complex and customizable options like those required for patients with cancer, the prostheses typically cost several thousands of dollars. The Healthcare Common Procedure Coding System code for a palatal lift prosthesis (D5955) lists prices ranging from $4000 to $8000 per prosthetic, not including the cost of the prosthodontist visits.22,23
Discussion
This program sought to determine whether imaging studies of maxillary defects are effective templates for developing 3D printed prosthetics and whether these prosthetics should be tested for future use in reconstruction of palatomaxillary defects. Our program illustrated that CTs served as feasible templates for developing hard palate prostheses for patients with palatomaxillary defects. It is important to note the CTs used were from a newer and more modern scanner and therefore yielded detailed palatal structures with higher accuracy more suitable for 3D modeling. Lower-quality CTs from the 4 patients excluded from the program were not suitable for 3D modeling. This suggests that with high-quality imaging, 3D printed prosthesis may be a viable strategy to help patients who struggle with their function following treatment for head and neck cancers.
3D printed prosthesis may also be a more patient centered and convenient option. In the traditional prosthesis creation workflow, the patient must physically bite down onto a resin (alginate or silicone) to make an impression, a very painful postoperative process that is irritating to the raw edges of the surgical bed.15,16 Prosthodontists then create a prosthetic minus the tumor and typically secure it with clips or glue.17 Many patients also experience changes in their anatomy over time requiring them to have a new protheses created. This is particularly important in veterans with palatomaxillary defects since many VA medical centers do not have a prosthodontist on staff, making accessibility to these specialists difficult. 3D printing provides a contactless prosthetic creation process. This convenience may reduce a patient’s pain and the number of visits for which they need a specialist.
Future Directions
Additional research is needed to determine the full potential of 3D printed prosthetics. 3D printed prostheses have been effectively used for patient education in areas of presurgical planning, prosthesis creation, and trainee education.24 This research represents an early step in the development of a new technology for use in otolaryngology. Specifically, many veterans with a history of head and neck cancers have sustained changes to their craniofacial anatomy following treatment. Using imaging to create 3D printed prosthetics could be very effective for these patients. Prosthetics could improve a patient’s quality of life by restoring/approximating their anatomy after cancer treatment.
Significant time and care must be taken by cancer and reconstructive surgeons to properly fit a prosthesis. Improperly fitting prosthetics leads to mucosal ulceration that then may lead to a need for fitting a new prosthetic. The advantage of 3D printed prosthetics is that they may more precisely fit the anatomy of each patient using CT results, thus potentially reducing the time needed to fit the prosthetic as well as the risk associated with an improperly fit prosthetic. 3D printed prosthesis could be used directly in the future, however, clinical trials are needed to verify its efficacy vs prosthodontic options.
Another consideration for potential future use of 3D printed prosthetics is cost. We estimated that the cost of the materials and labor of our 3D printed prosthetic to be about $150. Pricing of current molded prosthetics varies, but is often listed at several thousand dollars. Another consideration is the durability of 3D printed prosthetics vs standard prosthetics. Since we were unable to use the prosthetic in the patient, it was difficult to determine its durability. The significant cost of the 3D printer and software necessary for 3D printed prosthetics must also be considered and may be prohibitive. While many academic hospitals are considering the purchase of 3D printers and licenses, this may be challenging for resource-constrained institutions. 3D printing may also be difficult for groups without any prior experience in the field. Outsourcing to a third party is possible, though doing so adds more cost to the project. While we recognize there is a learning curve associated with adopting any new technology, it’s equally important to note that 3D printing is being rapidly integrated and has already made significant advancements in personalized medicine.8,25,26
Limitations
This program had several limitations. First, we only obtained CTs of sufficient quality from 1 patient to generate a 3D printed prosthesis. Further research with additional patients is necessary to validate this process. Second, we were unable to trial the prosthesis in the patient because we did not have FDA approval. Additionally, it is difficult to calculate a true cost estimate for this process as materials and software costs vary dramatically across institutions as well as over time.
Conclusions
The purpose of this study was to demonstrate the possibility to develop prosthetics for the hard palate for patients suffering from palatomaxillary defects. A 3D printed prosthetic was generated that matched the patient’s craniofacial anatomy. Future research should test the feasibility of these prosthetics in patient care against a traditional prosthodontic impression. Though this is a proof-of-concept study and no prosthetics were implanted as part of this investigation, we showcase the feasibility of printing prosthetics for palatomaxillary defects. The use of 3D printed prosthetics may be a more humane process, potentially lower cost, and be more accessible to veterans.
Three-dimensional (3D) printing has become a promising area of innovation in biomedical research.1,2 Previous research in orthopedic surgery has found that customized 3D printed implants, casts, orthoses, and prosthetics (eg, prosthetic hands) matched to an individual’s unique anatomy can result in more precise placement and better surgical outcomes.3-5 Customized prosthetics have also been found to lead to fewer complications.3,6
Recent advances in 3D printing technology has prompted investigation from surgeons to identify how this new tool may be incorporated into patient care.1,7 One of the most common applications of 3D printing is during preoperative planning in which surgeons gain better insight into patient-specific anatomy by using patient-specific printed models.8 Another promising application is the production of customized prosthetics suited to each patient’s unique anatomy.9 As a result, 3D printing has significantly impacted bone and cartilage restoration procedures and has the potential to completely transform the treatment of patients with debilitating musculoskeletal injuries.3,10
The potential surrounding 3D printed prosthetics has led to their adoption by several other specialties, including otolaryngology.11 The most widely used application of 3D printing among otolaryngologists is preoperative planning, and the incorporation of printed prosthetics intoreconstruction of the orbit, nasal septum, auricle, and palate has also been reported.2,12,13 Patient-specific implants might allow otolaryngologists to better rehabilitate, reconstruct, and/or regenerate craniofacial defects using more humane procedures.14
Patients with palatomaxillary cancers are treated by prosthodontists or otolaryngologists. An impression is made with a resin–which can be painful for postoperative patients–and a prosthetic is manufactured and implanted.15-17 Patients with cancer often see many specialists, though reconstructive care is a low priority. Many of these individuals also experience dynamic anatomic functional changes over time, leading to the need for multiple prothesis.
palatomaxillary prosthetics
This program aims to use patients’ previous computed tomography (CT) to tailor customized 3D printed palatomaxillary prosthetics to specifically fit their anatomy. Palatomaxillary defects are a source of profound disability for patients with head and neck cancers who are left with large anatomic defects as a direct result of treatment. Reconstruction of palatal defects poses unique challenges due to the complexity of patient anatomy.18,19
3D printed prosthetics for palatomaxillary defects have not been incorporated into patient care. We reviewed previous imaging research to determine if it could be used to assist patients who struggle with their function and appearance following treatment for head and neck cancers. The primary aim was to investigate whether 3D printing was a feasible strategy for creating patient-specific palatomaxillary prosthetics. The secondary aim is to determine whether these prosthetics should be tested in the future for use in reconstruction of maxillary defects.
Data Acquisition
This study was conducted at the Veterans Affairs Palo Alto Health Care System (VAPAHCS) and was approved by the Stanford University Institutional Review Board (approval #28958, informed consent and patient contact excluded). A retrospective chart review was conducted on all patients with head and neck cancers who were treated at VAPAHCS from 2010 to 2022. Patients aged ≥ 18 years who had a palatomaxillary defect due to cancer treatment, had undergone a palatal resection, and who received treatment at any point from 2010 to 2022 were included in the review. CTs were not a specific inclusion criterion, though the quality of the scans was analyzed for eligible patients. Younger patients and those treated at VAPAHCS prior to 2010 were excluded.
There was no control group; all data was sourced from the US Department of Veterans Affairs (VA) imaging system database. Among the 3595 patients reviewed, 5 met inclusion criteria and the quality of their craniofacial anatomy CTs were analyzed. To maintain accurate craniofacial 3D modeling, CTs require a maximum of 1 mm slice thickness. Of the 5 patients who met the inclusion criteria, 4 were found to have variability in the quality of their CTs and severe defects not suitable for prosthetic reconstruction, which led to their exclusion from the study. One patient was investigated to demonstrate if making these prostheses was feasible. This patient was diagnosed with a malignant neoplasm of the hard palate, underwent a partial maxillectomy, and a palatal obturator was placed to cover the defect.
The primary data collected was patient identifiers as well as the gross anatomy and dimensions of the patients’ craniofacial anatomy, as seen in previous imaging research.20 Before the imaging analysis, all personal health information was removed and the dataset was deidentified to ensure patient anonymity and noninvolvement.
CT Segmentation and 3D Printing
Using CTs of the patient’s craniofacial anatomy, we developed a model of the defects. This was achieved with deidentified CTs imported into the Food and Drug Administration (FDA)-approved computerized aid design (CAD) software, Materialise Mimics. The hard palate was segmented and isolated based off the presented scan and any holes in the image were filled using the CAD software. The model was subsequently mirrored in Materialise 3-matic to replicate an original anatomical hard palate prosthesis. The final product was converted into a 3D model and imported into Formlabs preform software to generate 3D printing supports and orient it for printing. The prosthetic was printed using FDA-approved Biocompatible Denture Base Resin by a Formlabs 3B+ printer at the Palo Alto VA Simulation Center. The 3D printed prosthesis was washed using Formlabs Form Wash 80% ethyl alcohol to remove excess resin and subsequently cured to harden the malleable resin. Supports were later removed, and the prosthesis was sanded.
The primary aim of this study was to investigate whether using CTs to create patient-specific prosthetic renderings for patients with head and neck cancer could be a feasible strategy. The CTs from the patient were successfully used to generate a 3D printed prosthesis, and the prosthesis matched the original craniofacial anatomy seen in the patient's imaging (Figure). These results demonstrate that high quality CTs can be used as a template for 3D printed prostheses for mild to moderate palatomaxillary defects.
3D Printing Costs
One liter of Denture Base Resin costs $299; prostheses use about 5 mL of resin. The average annual salary of a 3D printing technician in the United States is $42,717, or $20.54 per hour.21 For an experienced 3D printing technician, the time required to segment the hard palate and prepare it for 3D printing is 1 to 2 hours. The process may exceed 2 hours if the technician is presented with a lower quality CT or if the patient has a complex craniofacial anatomy.
The average time it takes to print a palatal prosthetic is 5 hours. An additional hour is needed for postprocessing, which includes washing and sanding. Therefore, the cost of the materials and labor for an average 3D printed prosthetic is about $150. A Formlabs 3B+ printer is competitively priced around $10,000. The cost for Materialise Mimics software varies, but is estimated at $16,000 at VAPAHCS. The prices for these 2 items are not included in our price estimation but should be taken into consideration.
Prosthodontist Process and Cost
The typical process of creating a palatal prosthesis by a prosthodontist begins by examining the patient, creating a stone model, then creating a wax model. Biocompatible materials are selected and processed into a mold that is trimmed and polished to the desired shape. This is followed by another patient visit to ensure the prosthesis fits properly. Follow-up care is also necessary for maintenance and comfort.
The average cost of a palatal prosthesis varies depending on the type needed (ie, metal implant, teeth replacement), the materials used, the region in which the patient is receiving care, and the complexity of the case. For complex and customizable options like those required for patients with cancer, the prostheses typically cost several thousands of dollars. The Healthcare Common Procedure Coding System code for a palatal lift prosthesis (D5955) lists prices ranging from $4000 to $8000 per prosthetic, not including the cost of the prosthodontist visits.22,23
Discussion
This program sought to determine whether imaging studies of maxillary defects are effective templates for developing 3D printed prosthetics and whether these prosthetics should be tested for future use in reconstruction of palatomaxillary defects. Our program illustrated that CTs served as feasible templates for developing hard palate prostheses for patients with palatomaxillary defects. It is important to note the CTs used were from a newer and more modern scanner and therefore yielded detailed palatal structures with higher accuracy more suitable for 3D modeling. Lower-quality CTs from the 4 patients excluded from the program were not suitable for 3D modeling. This suggests that with high-quality imaging, 3D printed prosthesis may be a viable strategy to help patients who struggle with their function following treatment for head and neck cancers.
3D printed prosthesis may also be a more patient centered and convenient option. In the traditional prosthesis creation workflow, the patient must physically bite down onto a resin (alginate or silicone) to make an impression, a very painful postoperative process that is irritating to the raw edges of the surgical bed.15,16 Prosthodontists then create a prosthetic minus the tumor and typically secure it with clips or glue.17 Many patients also experience changes in their anatomy over time requiring them to have a new protheses created. This is particularly important in veterans with palatomaxillary defects since many VA medical centers do not have a prosthodontist on staff, making accessibility to these specialists difficult. 3D printing provides a contactless prosthetic creation process. This convenience may reduce a patient’s pain and the number of visits for which they need a specialist.
Future Directions
Additional research is needed to determine the full potential of 3D printed prosthetics. 3D printed prostheses have been effectively used for patient education in areas of presurgical planning, prosthesis creation, and trainee education.24 This research represents an early step in the development of a new technology for use in otolaryngology. Specifically, many veterans with a history of head and neck cancers have sustained changes to their craniofacial anatomy following treatment. Using imaging to create 3D printed prosthetics could be very effective for these patients. Prosthetics could improve a patient’s quality of life by restoring/approximating their anatomy after cancer treatment.
Significant time and care must be taken by cancer and reconstructive surgeons to properly fit a prosthesis. Improperly fitting prosthetics leads to mucosal ulceration that then may lead to a need for fitting a new prosthetic. The advantage of 3D printed prosthetics is that they may more precisely fit the anatomy of each patient using CT results, thus potentially reducing the time needed to fit the prosthetic as well as the risk associated with an improperly fit prosthetic. 3D printed prosthesis could be used directly in the future, however, clinical trials are needed to verify its efficacy vs prosthodontic options.
Another consideration for potential future use of 3D printed prosthetics is cost. We estimated that the cost of the materials and labor of our 3D printed prosthetic to be about $150. Pricing of current molded prosthetics varies, but is often listed at several thousand dollars. Another consideration is the durability of 3D printed prosthetics vs standard prosthetics. Since we were unable to use the prosthetic in the patient, it was difficult to determine its durability. The significant cost of the 3D printer and software necessary for 3D printed prosthetics must also be considered and may be prohibitive. While many academic hospitals are considering the purchase of 3D printers and licenses, this may be challenging for resource-constrained institutions. 3D printing may also be difficult for groups without any prior experience in the field. Outsourcing to a third party is possible, though doing so adds more cost to the project. While we recognize there is a learning curve associated with adopting any new technology, it’s equally important to note that 3D printing is being rapidly integrated and has already made significant advancements in personalized medicine.8,25,26
Limitations
This program had several limitations. First, we only obtained CTs of sufficient quality from 1 patient to generate a 3D printed prosthesis. Further research with additional patients is necessary to validate this process. Second, we were unable to trial the prosthesis in the patient because we did not have FDA approval. Additionally, it is difficult to calculate a true cost estimate for this process as materials and software costs vary dramatically across institutions as well as over time.
Conclusions
The purpose of this study was to demonstrate the possibility to develop prosthetics for the hard palate for patients suffering from palatomaxillary defects. A 3D printed prosthetic was generated that matched the patient’s craniofacial anatomy. Future research should test the feasibility of these prosthetics in patient care against a traditional prosthodontic impression. Though this is a proof-of-concept study and no prosthetics were implanted as part of this investigation, we showcase the feasibility of printing prosthetics for palatomaxillary defects. The use of 3D printed prosthetics may be a more humane process, potentially lower cost, and be more accessible to veterans.
1. Crafts TD, Ellsperman SE, Wannemuehler TJ, Bellicchi TD, Shipchandler TZ, Mantravadi AV. Three-dimensional printing and its applications in otorhinolaryngology-head and neck surgery. Otolaryngol Head Neck Surg. 2017;156(6):999-1010. doi:10.1177/0194599816678372
2. Virani FR, Chua EC, Timbang MR, Hsieh TY, Senders CW. Three-dimensional printing in cleft care: a systematic review. Cleft Palate Craniofac J. 2022;59(4):484-496. doi:10.1177/10556656211013175
3. Lal H, Patralekh MK. 3D printing and its applications in orthopaedic trauma: A technological marvel. J Clin Orthop Trauma. 2018;9(3):260-268. doi:10.1016/j.jcot.2018.07.022
4. Vujaklija I, Farina D. 3D printed upper limb prosthetics. Expert Rev Med Devices. 2018;15(7):505-512. doi:10.1080/17434440.2018.1494568
5. Ten Kate J, Smit G, Breedveld P. 3D-printed upper limb prostheses: a review. Disabil Rehabil Assist Technol. 2017;12(3):300-314. doi:10.1080/17483107.2016.1253117
6. Thomas CN, Mavrommatis S, Schroder LK, Cole PA. An overview of 3D printing and the orthopaedic application of patient-specific models in malunion surgery. Injury. 2022;53(3):977-983. doi:10.1016/j.injury.2021.11.019
7. Colaco M, Igel DA, Atala A. The potential of 3D printing in urological research and patient care. Nat Rev Urol. 2018;15(4):213-221. doi:10.1038/nrurol.2018.6
8. Meyer-Szary J, Luis MS, Mikulski S, et al. The role of 3D printing in planning complex medical procedures and training of medical professionals-cross-sectional multispecialty review. Int J Environ Res Public Health. 2022;19(6):3331. Published 2022 Mar 11. doi:10.3390/ijerph19063331
9. Moya D, Gobbato B, Valente S, Roca R. Use of preoperative planning and 3D printing in orthopedics and traumatology: entering a new era. Acta Ortop Mex. 2022;36(1):39-47.
10. Wixted CM, Peterson JR, Kadakia RJ, Adams SB. Three-dimensional printing in orthopaedic surgery: current applications and future developments. J Am Acad Orthop Surg Glob Res Rev. 2021;5(4):e20.00230-11. Published 2021 Apr 20. doi:10.5435/JAAOSGlobal-D-20-00230
11. Hong CJ, Giannopoulos AA, Hong BY, et al. Clinical applications of three-dimensional printing in otolaryngology-head and neck surgery: a systematic review. Laryngoscope. 2019;129(9):2045-2052. doi:10.1002/lary.2783112. Sigron GR, Barba M, Chammartin F, Msallem B, Berg BI, Thieringer FM. Functional and cosmetic outcome after reconstruction of isolated, unilateral orbital floor fractures (blow-out fractures) with and without the support of 3D-printed orbital anatomical models. J Clin Med. 2021;10(16):3509. Published 2021 Aug 9. doi:10.3390/jcm10163509
13. Kimura K, Davis S, Thomas E, et al. 3D Customization for microtia repair in hemifacial microsomia. Laryngoscope. 2022;132(3):545-549. doi:10.1002/lary.29823
14. Nyberg EL, Farris AL, Hung BP, et al. 3D-printing technologies for craniofacial rehabilitation, reconstruction, and regeneration. Ann Biomed Eng. 2017;45(1):45-57. doi:10.1007/s10439-016-1668-5
15. Flores-Ruiz R, Castellanos-Cosano L, Serrera-Figallo MA, et al. Evolution of oral cancer treatment in an andalusian population sample: rehabilitation with prosthetic obturation and removable partial prosthesis. J Clin Exp Dent. 2017;9(8):e1008-e1014. doi:10.4317/jced.54023
16. Rogers SN, Lowe D, McNally D, Brown JS, Vaughan ED. Health-related quality of life after maxillectomy: a comparison between prosthetic obturation and free flap. J Oral Maxillofac Surg. 2003;61(2):174-181. doi:10.1053/joms.2003.50044
17. Pool C, Shokri T, Vincent A, Wang W, Kadakia S, Ducic Y. Prosthetic reconstruction of the maxilla and palate. Semin Plast Surg. 2020;34(2):114-119. doi:10.1055/s-0040-1709143
18. Badhey AK, Khan MN. Palatomaxillary reconstruction: fibula or scapula. Semin Plast Surg. 2020;34(2):86-91. doi:10.1055/s-0040-1709431
19. Jategaonkar AA, Kaul VF, Lee E, Genden EM. Surgery of the palatomaxillary structure. Semin Plast Surg. 2020;34(2):71-76. doi:10.1055/s-0040-1709430
20. Lobb DC, Cottler P, Dart D, Black JS. The use of patient-specific three-dimensional printed surgical models enhances plastic surgery resident education in craniofacial surgery. J Craniofac Surg. 2019;30(2):339-341. doi:10.1097/SCS.0000000000005322
21. 3D printing technician salary in the United States. Accessed February 27, 2024. https://www.salary.com/research/salary/posting/3d-printing-technician-salary22. US Dept of Veterans Affairs. Healthcare Common Procedure Coding System. Outpatient dental professional nationwide charges by HCPCS code. January-December 2020. Accessed February 27, 2024. https://www.va.gov/COMMUNITYCARE/docs/RO/Outpatient-DataTables/v3-27_Table-I.pdf23. Washington State Department of Labor and Industries. Professional services fee schedule HCPCS level II fees. October 1, 2020. Accessed February 27, 2024. https://lni.wa.gov/patient-care/billing-payments/marfsdocs/2020/2020FSHCPCS.pdf24. Low CM, Morris JM, Price DL, et al. Three-dimensional printing: current use in rhinology and endoscopic skull base surgery. Am J Rhinol Allergy. 2019;33(6):770-781. doi:10.1177/1945892419866319
25. Aimar A, Palermo A, Innocenti B. The role of 3D printing in medical applications: a state of the art. J Healthc Eng. 2019;2019:5340616. Published 2019 Mar 21. doi:10.1155/2019/5340616
26. Garcia J, Yang Z, Mongrain R, Leask RL, Lachapelle K. 3D printing materials and their use in medical education: a review of current technology and trends for the future. BMJ Simul Technol Enhanc Learn. 2018;4(1):27-40. doi:10.1136/bmjstel-2017-000234
1. Crafts TD, Ellsperman SE, Wannemuehler TJ, Bellicchi TD, Shipchandler TZ, Mantravadi AV. Three-dimensional printing and its applications in otorhinolaryngology-head and neck surgery. Otolaryngol Head Neck Surg. 2017;156(6):999-1010. doi:10.1177/0194599816678372
2. Virani FR, Chua EC, Timbang MR, Hsieh TY, Senders CW. Three-dimensional printing in cleft care: a systematic review. Cleft Palate Craniofac J. 2022;59(4):484-496. doi:10.1177/10556656211013175
3. Lal H, Patralekh MK. 3D printing and its applications in orthopaedic trauma: A technological marvel. J Clin Orthop Trauma. 2018;9(3):260-268. doi:10.1016/j.jcot.2018.07.022
4. Vujaklija I, Farina D. 3D printed upper limb prosthetics. Expert Rev Med Devices. 2018;15(7):505-512. doi:10.1080/17434440.2018.1494568
5. Ten Kate J, Smit G, Breedveld P. 3D-printed upper limb prostheses: a review. Disabil Rehabil Assist Technol. 2017;12(3):300-314. doi:10.1080/17483107.2016.1253117
6. Thomas CN, Mavrommatis S, Schroder LK, Cole PA. An overview of 3D printing and the orthopaedic application of patient-specific models in malunion surgery. Injury. 2022;53(3):977-983. doi:10.1016/j.injury.2021.11.019
7. Colaco M, Igel DA, Atala A. The potential of 3D printing in urological research and patient care. Nat Rev Urol. 2018;15(4):213-221. doi:10.1038/nrurol.2018.6
8. Meyer-Szary J, Luis MS, Mikulski S, et al. The role of 3D printing in planning complex medical procedures and training of medical professionals-cross-sectional multispecialty review. Int J Environ Res Public Health. 2022;19(6):3331. Published 2022 Mar 11. doi:10.3390/ijerph19063331
9. Moya D, Gobbato B, Valente S, Roca R. Use of preoperative planning and 3D printing in orthopedics and traumatology: entering a new era. Acta Ortop Mex. 2022;36(1):39-47.
10. Wixted CM, Peterson JR, Kadakia RJ, Adams SB. Three-dimensional printing in orthopaedic surgery: current applications and future developments. J Am Acad Orthop Surg Glob Res Rev. 2021;5(4):e20.00230-11. Published 2021 Apr 20. doi:10.5435/JAAOSGlobal-D-20-00230
11. Hong CJ, Giannopoulos AA, Hong BY, et al. Clinical applications of three-dimensional printing in otolaryngology-head and neck surgery: a systematic review. Laryngoscope. 2019;129(9):2045-2052. doi:10.1002/lary.2783112. Sigron GR, Barba M, Chammartin F, Msallem B, Berg BI, Thieringer FM. Functional and cosmetic outcome after reconstruction of isolated, unilateral orbital floor fractures (blow-out fractures) with and without the support of 3D-printed orbital anatomical models. J Clin Med. 2021;10(16):3509. Published 2021 Aug 9. doi:10.3390/jcm10163509
13. Kimura K, Davis S, Thomas E, et al. 3D Customization for microtia repair in hemifacial microsomia. Laryngoscope. 2022;132(3):545-549. doi:10.1002/lary.29823
14. Nyberg EL, Farris AL, Hung BP, et al. 3D-printing technologies for craniofacial rehabilitation, reconstruction, and regeneration. Ann Biomed Eng. 2017;45(1):45-57. doi:10.1007/s10439-016-1668-5
15. Flores-Ruiz R, Castellanos-Cosano L, Serrera-Figallo MA, et al. Evolution of oral cancer treatment in an andalusian population sample: rehabilitation with prosthetic obturation and removable partial prosthesis. J Clin Exp Dent. 2017;9(8):e1008-e1014. doi:10.4317/jced.54023
16. Rogers SN, Lowe D, McNally D, Brown JS, Vaughan ED. Health-related quality of life after maxillectomy: a comparison between prosthetic obturation and free flap. J Oral Maxillofac Surg. 2003;61(2):174-181. doi:10.1053/joms.2003.50044
17. Pool C, Shokri T, Vincent A, Wang W, Kadakia S, Ducic Y. Prosthetic reconstruction of the maxilla and palate. Semin Plast Surg. 2020;34(2):114-119. doi:10.1055/s-0040-1709143
18. Badhey AK, Khan MN. Palatomaxillary reconstruction: fibula or scapula. Semin Plast Surg. 2020;34(2):86-91. doi:10.1055/s-0040-1709431
19. Jategaonkar AA, Kaul VF, Lee E, Genden EM. Surgery of the palatomaxillary structure. Semin Plast Surg. 2020;34(2):71-76. doi:10.1055/s-0040-1709430
20. Lobb DC, Cottler P, Dart D, Black JS. The use of patient-specific three-dimensional printed surgical models enhances plastic surgery resident education in craniofacial surgery. J Craniofac Surg. 2019;30(2):339-341. doi:10.1097/SCS.0000000000005322
21. 3D printing technician salary in the United States. Accessed February 27, 2024. https://www.salary.com/research/salary/posting/3d-printing-technician-salary22. US Dept of Veterans Affairs. Healthcare Common Procedure Coding System. Outpatient dental professional nationwide charges by HCPCS code. January-December 2020. Accessed February 27, 2024. https://www.va.gov/COMMUNITYCARE/docs/RO/Outpatient-DataTables/v3-27_Table-I.pdf23. Washington State Department of Labor and Industries. Professional services fee schedule HCPCS level II fees. October 1, 2020. Accessed February 27, 2024. https://lni.wa.gov/patient-care/billing-payments/marfsdocs/2020/2020FSHCPCS.pdf24. Low CM, Morris JM, Price DL, et al. Three-dimensional printing: current use in rhinology and endoscopic skull base surgery. Am J Rhinol Allergy. 2019;33(6):770-781. doi:10.1177/1945892419866319
25. Aimar A, Palermo A, Innocenti B. The role of 3D printing in medical applications: a state of the art. J Healthc Eng. 2019;2019:5340616. Published 2019 Mar 21. doi:10.1155/2019/5340616
26. Garcia J, Yang Z, Mongrain R, Leask RL, Lachapelle K. 3D printing materials and their use in medical education: a review of current technology and trends for the future. BMJ Simul Technol Enhanc Learn. 2018;4(1):27-40. doi:10.1136/bmjstel-2017-000234
Augmented Reality Demonstration Survey Results From a Veteran Affairs Medical Center
Building the health care system of the future requires the thoughtful development and integration of innovative technologies to positively transform care.1-4 Extended reality (XR) represents a spectrum of emerging technologies that have the potential to enhance health care. This includes virtual reality (VR), where a computer-generated visual experience fills the screen; augmented reality (AR), which allows users to see computer-generated images superimposed into an otherwise normal real-world field of view; and mixed reality (MR), which allows users to interact and manipulate computer-generated AR images.
Clinicians and researchers have begun exploring the potential of XR to address a wide variety of health care challenges. A recent systematic review concluded that many clinical studies in this area have small sample sizes and are in the preclinical, proof-of-concept stage, but demonstrate the potential and impact of the underlying VR, AR, and MR technologies.5 Common emerging health care uses for XR include medical education, training, presurgical planning, surgical guidance, distraction therapy for pain and anxiety, and home health indications, including rehabilitation.5-39
Importantly, some researchers have raised concerns regarding the adaptability of the health care workforce with emerging technologies, and their interest in new methods of delivering care.7,39 Successful deployment of any novel health care technology depends on multiple factors, including alignment with staff needs, receptivity to those solutions, customization to specific preferences, and usability.1,3,40-42 Unfortunately, the implementation of some health care technologies, such as electronic health records that did not account for end-user requirements, resulted in employee fatigue, burnout, and negative staffing turnover.42-44 Conversely, elevated employee morale and operational performance have been directly linked to a climate of inclusion and innovation.45-47 In this assessment, we sought to understand US Department of Veterans Affairs (VA) employees’ perceptions and expert opinions related to the introduction of new AR/MR technology.
Methods
The VA Palo Alto Health Care System (VAPAHCS) consists of 3 inpatient hospitals and 7 outpatient clinics, provides a full range of care services to > 90,000 enrolled veterans with 800 hospital beds, 3 nursing homes, and a 100-bed domiciliary. The facility also runs data-driven care projects in research, innovation, and evidence-based practice group under nursing services.48 This project was performed by the VA National Center for Collaborative Healthcare Innovation at the VAPAHCS campus.
The combined technical system used for this assessment included a wireless communication network, AR/MR hardware, and software. Medivis AnatomyX software displayed an interactive human anatomy atlas segmented into about 6000 individual interactive parts. Medivis SurgicalAR received US Food and Drug Administration clearance for presurgical planning and was used to transform and display deidentified diagnostic images (eg, magnetic resonance images and computed tomography) in 3-dimensional (3D) interactive holograms (Figures 1 and 2). 
The wireless Microsoft HoloLens 2 AR/MR headset was used for viewing and sensor-enabled collaborative interaction. Multiple participants in the same physical location simultaneously participated and interacted with 3D holograms. The interactive hologram data were enabled for 3D stereoscopic viewing and manipulation.
Setting and Participants
We reviewed published studies that used questionnaires to evaluate institutions’ level of innovation and new technology user acceptance to develop the questionnaire.49-56 Questions and methods were modified, with a focus on understanding the impact on hospital employees. The questionnaire consisted of 2 predemonstration and 3 postdemonstration sections. The first section included background questions. The second (predemonstration) and third (postdemonstration) sections provided matched questions on feelings about the VA. The fourth section included 2 unmatched questions about how the participant felt this technology would impact veterans and whether the VA should implement similar technologies. We used a 5-point Likert scale for sections 2, 3 and 4 (1 = not at all to 5 = extremely). Two unmatched free-text questions asked how the technology could be used in the participant’s hospital service, and another open-ended question asked for any additional comments. To reduce potential reporting bias, 2 VA employees that did not work at VAPAHCS assisted with the survey distribution and collection. VAPAHCS staff were informed by all employee email and facility intranet of the opportunity to participate; the voluntary demonstration and survey took place on February 10 and 11, 2020.
Data Analysis
All matching pre/post questions were analyzed together to determine statistically significant differences using the Wilcoxon signed rank matched pairs test and pooled t test. Survey respondents were also grouped by employment type to evaluate the impact on subgroups. Results were also grouped by VA tenure into 4 categorical 10-year increments (0-10, 11-20, 21-30, 31-40). Additionally, analysis of variance (ANOVA) was performed on employment types and VA tenure to understand whether there was a statistically significant difference in responses by these subgroups. Respondents’ optional free-text answers were manually reviewed by 2 authors (ZPV and DMA), classified, coded by the common themes, and analyzed for comparison.
Results
A total of 166 participants completed the predemonstration survey, which was a requirement for participating in the AR demonstration. Of those, 159 staff members (95.8%) also completed at least part of the postdemonstration paired structured questions, and their results were included in the analysis. 
Paired Questions
For questions about how innovative the VA is, 108 of 152 participants (71.1%) provided higher scores after the demonstration, 42 (27.6%) had no change, and 2 (1.3%) provided decreased scores. The mean innovative score increased from 3.4 predemonstration to 4.5 postdemonstration on a Likert scale, which is a 1.1 point increase from predemonstration to postdemonstration (95% CI, 0.9- 1.2) or a 22% increase (95% CI, 18%-24%) (P < .001). Respondents level of excitement about VA also increased with 82 of 157 participants (52.2%) providing higher scores after the demonstration, 71 (45.2%) had no change, and 4 scores (2.5%) decreased. The predemonstration mean excitement score of 3.7 increased to 4.3 postdemonstration, which is a 0.6 point increase from before to after the demonstration (95% CI, 0.5-0.7) or a 12% increase (95% CI, 10%-14%) (P < .001). In the survey, 36 of 149 participants (24.2%) had higher scores for their expectation to continue working at VA postdemonstration, 109 (73.2%) had no change, and 4 scores (2.7%) decreased. The mean employee retention score increased from 4.2 predemonstration to 4.5 postdemonstration, which is a 0.3 point increase between pre/post (95% CI, 0.2-0.4) or a 6% increase (95% CI, 4%-8%) (P < .001)
The pre/post questions were analyzed using 1-way ANOVA by hospital department and VA tenure. The responses by department were not statistically significant. Of the 159 employees assessed, 101 respondents (63.5%) had 0 to 10 years VA tenure, 44 (27.7%) had 11 to 20 years, 10 (6.3%) had 21 to 30 years, and 4 (2.5%) had > 31 to 40 years. Length of VA tenure did not impact respondent excitement. Respondents opinions on innovation in the 0 to 10 year and the 11 to 20 year groups rose from 3.2 and 3.7 predemonstration to 4.3 and 4.6 postdemonstration, respectively (P < .001 for both statistical comparisons) (Table 2). Interestingly, the 0 to 10 group saw a 9% rise from a 4.0 score predemonstration to a 4.4 score postdemonstration (P < .001), indicating that the demonstration had a positive impact on their plans to continue employment at VA (Table 3).
Sex did not play a significant role in how respondents answered questions regarding VA excitement or innovation. However, there was a statistically significant difference in how male and female respondents answered the predemonstration question about their plans to continue VA employment, according to the Wilcoxon rank sum test. Predemonstration, female respondents had a mean score of 4.1, which was 6% lower than the 4.4 score of male colleagues (P = .04). Veteran status did have an impact on how respondents felt about VA innovation, and their plans to continue employment at VA. After the demonstration, veteran staff felt the VA was more innovative compared with nonveterans: 4.7 vs 4.4, respectively, a 6% difference (P = .02) Similarly, for the continued VA employment question, veterans had a mean score of 4.8 vs 4.4 for nonveterans, an 8% difference (P = .03) These results suggest that the demonstration had more of an impact on veteran employees vs nonveteran employees.
Unpaired Questions
There were 2 structured unpaired postdemonstration questions. Respondents agreed that similar technology would impact veteran health care with mean (SD) of 4.6 (0.6) and a median score of 5 on a 5-point Likert scale. Respondents also agreed on the importance of implementing similar innovations with mean (SD) of 4.7 (0.5), and a median score of 5.
The survey asked how this technology could benefit their hospital service department and had 64 responses. Forty-six respondents saw applications for education or patient care/surgery. Other responses shared excitement about the technology and its potential to positively impact patient education. There were 37 responses to the open-ended question: 21 respondents expressed excitement for the technology, and 10 respondents reiterated that the demonstration would be of benefit to patient care/surgery and training.
Discussion
Successful development, design, and deployment of any new health care tool depends on leveraging insights from the employees that will be using and supporting these systems. Correspondingly, understanding the impact that advanced technologies have on health care employees’ satisfaction, morale, and retention is critical to our overall institutional strategy. Our findings show that a one-time experience with AR/MR technology elicited positive employee reactions. Of note, the survey revealed statistically significant improvements in staff’s view of the VA, with the greatest positive impact for questions about innovation, followed by excitement to work at the VA, and likelihood to continue work at the VA. It is very disruptive and costly when health care employees leave, and improving employee satisfaction and morale is important for better patient care and patient satisfaction, which is priority for VAPAHCS leadership.57-62
The paired predemonstration and postdemonstration scores were similarly high, nearing the top threshold available for the Likert scale (4.3 to 4.5). Furthermore, the least incremental improvement for these responses was observed for topics that had the highest initial baseline score. Therefore, the improvements observed for the paired questions may have more to do with the high baseline values.
Of additional interest, the self-reported likelihood of continuing to work at the VA increased the most for female employees, veteran employees, and employees with the least number of years at the VA. These demographic differences have important implications for VA staff recruitment and retention strategies.62 The unpaired questions about the impact on veteran care and whether the VA should continue similar work demonstrated extremely high support with median scores of 5 for both questions. The free-text postdemonstration responses also demonstrate similar positive themes, with a disposition for excitement about both the training and patient care applications for this technology. In addition, respondents felt strongly that this and other similar technologies will positively impact the health care for veterans and that the VA should continue these efforts.
Strengths and Limitations
A strength of this assessment is the ability to evaluate survey responses that were systematically collected and matched from the same individual immediately before and after exposure to the new technology. The free-text responses provided additional important information that both confirmed the results and provided additional valued supplementary guidance for future implementation strategies, which is critical for our translational implementation goals. An additional strength is that the voluntary surveys were managed by non-VAPAHCS colleagues, limiting potential bias. Importantly, the number of respondents allowed a statistically significant assessment of important health care employee metrics. These results have emphasized how being part of an innovative organization, and the introduction of advanced AR/MR technology, improve employees’ satisfaction and morale about where they work as well as their intention to stay at their institution.
A limitation of this assessment was the lack of comparative data for employee acceptance of other technologies at VAPAHCS. This limits our ability to differentiate whether the strong positive results observed in this evaluation were a result of the specific technology assessed, or of new and advanced health care technology in general. Nonetheless, our unpaired questions, which received extremely high scores, also included participant questions about comparing the system with other similar technologies. This assessment was also focused on veteran care, which limits generalizability.
Conclusions
One-time exposure to advanced AR technology for health care significantly increased employee morale as measured by excitement about working at the VA as well as employee intention to continue employment at the VA. These collateral benefits of the technology are particularly important in health care because our employees are our most important asset and improving employee morale equates to better patient care. Positive impacts were most pronounced for women employees, newer VA employees, and employees who are also veterans. These more detailed insights are also positioned to have a direct impact on employee recruitment and retention strategies. Additional valuable insights regarding the most applicable use of the technology in the clinical setting were also obtained.
Acknowledgments
We thank Andrew Spiegelman, Hyewon Kim, Jonathan Sills, and Alexander Erickson for their assistance in developing the survey questions. We also thank Jason Rhodes and Mark Bulson for traveling to our facility to assist with managing the anonymous surveys during the demonstration event.
1. World Economic Forum. Health and healthcare in the fourth industrial revolution: Global Future Council on the future of health and healthcare 2016-2018. April 2019. Accessed January 27, 2023. https://www3.weforum.org/docs/WEF__Shaping_the_Future_of_Health_Council_Report.pdf
2. Iveroth E, Fryk P, Rapp B. Information technology strategy and alignment issues in health care organizations. Health Care Manage Rev. 2013;38(3):188-200. doi:10.1097/HMR.0b013e31826119d7
3. Thakur R, Hsu SH, Fontenot G. Innovation in healthcare: issues and future trends. J Bus Res. 2012;65(4):562-569. doi:10.1016/j.jbusres.2011.02.022
4. Thimbleby H. Technology and the future of healthcare. J Public Health Res. 2013;2(3):e28. Published 2013 Dec 1. doi:10.4081/jphr.2013.e28
5. Viglialoro RM, Condino S, Turini G, Carbone M, Ferrari V, Gesi M. augmented reality, mixed reality, and hybrid approach in healthcare simulation: a systematic review. Applied Sciences. 2021;11(5):2338. doi:10.3390/app11052338
6. Rawlins CR, Veigulis Z, Hebert C, Curtin C, Osborne T. Effect of immersive virtual reality on pain and anxiety at a Veterans Affairs health care facility. Front Virt Real. 2021;(2):136. doi:10.3389/frvir.2021.719681
7. Chawdhary G, Shoman N. Emerging artificial intelligence applications in otological imaging. Curr Opin Otolaryngol Head Neck Surg. 2021;29(5):357-364. doi:10.1097/MOO.0000000000000754
8. Asadzadeh A, Samad-Soltani T, Rezaei-Hachesu P. Applications of virtual and augmented reality in infectious disease epidemics with a focus on the COVID-19 outbreak. Inform Med Unlocked. 2021;24:100579. doi:10.1016/j.imu.2021.100579
9. Ashwini KB, Savitha R, Harish A. Application of augmented reality technology for home healthcare product visualization. ECS Transas. 2022;107(1):10921. doi:10.1149/10701.10921ecst
10. Brooks AL. VR/Technologies for Rehabilitation. In: Brooks AL, Brahman S, Kapralos B, Nakajima A, Tyerman J, Jain LC, eds. Recent Advances in Technologies for Inclusive Well-Being Virtual Patients, Gamification and Simulation. Intelligent Systems Reference Library. Springer; 2021:241-252. doi:10.1007/978-3-030-59608-8_13
11. Koulouris D, Menychtas A, Maglogiannis I. An IoT-enabled platform for the assessment of physical and mental activities utilizing augmented reality exergaming. Sensors (Basel). 2022;22(9):3181. Published 2022 Apr 21. doi:10.3390/s22093181
12. Deiss YR, Korkut S, Inglese T. Increase therapy understanding and medication adherence for patients with inflammatory skin diseases through augmented reality. Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022. 2022:21-40. doi:10.1007/978-3-031-06018-2_2
13. Bertino E, Gao W, Steffan B, et al, eds. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer; 2022:21-40.
14. Ruhaiyem NIR, Mazlan NA. Image Modeling Through Augmented Reality for Skin Allergies Recognition. Lecture Notes on Data Engineering and Communications Technologies. 2021:72-79. doi: 10.1007/978-3-030-70713-2_8
15. Park BJ, Perkons NR, Profka E, et al. Three-dimensional augmented reality visualization informs locoregional therapy in a translational model of hepatocellular carcinoma. J Vasc Interv Radiol. 2020;31(10):1612-1618.e1. doi:10.1016/j.jvir.2020.01.028
16. Leo J, Zhou Z, Yang H, et al, eds. Interactive cardiovascular surgical planning via augmented reality. 5th Asian CHI Symposium 2021; 2021. doi:10.1145/3429360.3468195
17. Zuo Y, Jiang T, Dou J, et al. A novel evaluation model for a mixed-reality surgical navigation system: where Microsoft Hololens meets the operating room. Surg Innov. 2020;27(2):193-202. doi:10.1177/1553350619893236
18. Ghaednia H, Fourman MS, Lans A, et al. Augmented and virtual reality in spine surgery, current applications and future potentials. Spine J. 2021;21(10):1617-1625. doi:10.1016/j.spinee.2021.03.018
19. Liu Y, Lee MG, Kim JS. Spine surgery assisted by augmented reality: where have we been?. Yonsei Med J. 2022;63(4):305-316. doi:10.3349/ymj.2022.63.4.305
20. Kimmel S, Cobus V, Heuten W, eds. opticARe—augmented reality mobile patient monitoring in intensive care units. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST; 2021. doi:10.1145/3489849.3489852
21. Voštinár P, Horváthová D, Mitter M, Bako M. The look at the various uses of VR. Open Computer Sci. 2021;11(1):241-250. doi:10.1515/comp-2020-0123
22. Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: a meta-analysis of randomized controlled studies. BMC Med Educ. 2020;20(1):127. Published 2020 Apr 25. doi:10.1186/s12909-020-1994-z
23. Ricci S, Calandrino A, Borgonovo G, Chirico M, Casadio M. Viewpoint: virtual and augmented reality in basic and advanced life support training. JMIR Serious Games. 2022;10(1):e28595. Published 2022 Mar 23. doi:10.2196/28595
24. Ricci S, Mobilio GA, Calandrino A, et al. RiNeo MR: A mixed-reality tool for newborn life support training. Annu Int Conf IEEE Eng Med Biol Soc. 2021;2021:5043-5046. doi:10.1109/EMBC46164.2021.9629612
25. Dhar P, Rocks T, Samarasinghe RM, Stephenson G, Smith C. Augmented reality in medical education: students’ experiences and learning outcomes. Med Educ Online. 2021;26(1):1953953. doi:10.1080/10872981.2021.1953953
26. Pears M, Konstantinidis S. The future of immersive technology in global surgery education [published online ahead of print, 2021 Jul 1]. Indian J Surg. 2021;84(suppl 1):1-5. doi:10.1007/s12262-021-02998-6
27. Liang CJ, Start C, Boley H, Kamat VR, Menassa CC, Aebersold M. Enhancing stroke assessment simulation experience in clinical training using augmented reality. Virt Real. 2021;25(3):575-584. doi:10.1007/s10055-020-00475-1
28. Lacey G, Gozdzielewska L, McAloney-Kocaman K, Ruttle J, Cronin S, Price L. Psychomotor learning theory informing the design and evaluation of an interactive augmented reality hand hygiene training app for healthcare workers. Educ Inf Technol. 2022;27(3):3813-3832. doi:10.1007/s10639-021-10752-4
29. Ryan GV, Callaghan S, Rafferty A, Higgins MF, Mangina E, McAuliffe F. Learning outcomes of immersive technologies in health care student education: systematic review of the literature. J Med Internet Res. 2022;24(2):e30082. Published 2022 Feb 1. doi:10.2196/30082
30. Yu FU, Yan HU, Sundstedt V. A Systematic literature review of virtual, augmented, and mixed reality game applications in healthcare. ACM Trans Comput Healthcare. 2022;3(2);1-27. doi:10.1145/3472303
31. Weeks JK, Amiel JM. Enhancing neuroanatomy education with augmented reality. Med Educ. 2019;53(5):516-517. doi:10.1111/medu.13843
32. Williams MA, McVeigh J, Handa AI, Lee R. Augmented reality in surgical training: a systematic review. Postgrad Med J. 2020;96(1139):537-542. doi:10.1136/postgradmedj-2020-137600
33. Triepels CPR, Smeets CFA, Notten KJB, et al. Does three-dimensional anatomy improve student understanding? Clin Anat. 2020;33(1):25-33. doi:10.1002/ca.23405
34. Pietruski P, Majak M, S´wia¸tek-Najwer E, et al. Supporting fibula free flap harvest with augmented reality: A proof-of-concept study. Laryngoscope. 2020;130(5):1173-1179. doi:10.1002/lary.28090
35. Perkins SL, Krajancich B, Yang CJ, Hargreaves BA, Daniel BL, Berry MF. A patient-specific mixed-reality visualization tool for thoracic surgical planning. Ann Thorac Surg. 2020;110(1):290-295. doi:10.1016/j.athoracsur.2020.01.060
36. Müller F, Roner S, Liebmann F, Spirig JM, Fürnstahl P, Farshad M. Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging. Spine J. 2020;20(4):621-628. doi:10.1016/j.spinee.2019.10.012
37. Kaplan AD, Cruit J, Endsley M, Beers SM, Sawyer BD, Hancock PA. The effects of virtual reality, augmented reality, and mixed reality as training enhancement methods: a meta-analysis. Hum Factors. 2021;63(4):706-726. doi:10.1177/0018720820904229
38. Jud L, Fotouhi J, Andronic O, et al. Applicability of augmented reality in orthopedic surgery - a systematic review. BMC Musculoskelet Disord. 2020;21(1):103. Published 2020 Feb 15. doi:10.1186/s12891-020-3110-2
39. Ara J, Karim FB, Alsubaie MSA, et al. Comprehensive analysis of augmented reality technology in modern healthcare system. Int J Adv Comput Sci Appl. 2021;12(6):845-854. doi:10.14569/IJACSA.2021.0120698
40. Webster A, Gardner J. Aligning technology and institutional readiness: the adoption of innovation. Technol Anal Strateg Manag. 2019;31(10):1229-1241. doi:10.1080/09537325.2019.1601694
41. Hastall MR, Dockweiler C, Mühlhaus J. achieving end user acceptance: building blocks for an evidence-based user-centered framework for health technology development and assessment. In: Antona, M, Stephanidis C, eds. Universal Access in Human–Computer Interaction. Human and Technological Environments. UAHCI 2017. Lecture Notes in Computer Science, vol 10279. Springer, Cham; 2017. doi:10.1007/978-3-319-58700-4_2
42. Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc. 2015;22(6):1179-1182. doi:10.1093/jamia/ocv050
43. Khairat S, Coleman C, Ottmar P, Jayachander DI, Bice T, Carson SS. Association of Electronic Health Record Use With Physician Fatigue and Efficiency. JAMA Netw Open. 2020;3(6):e207385. Published 2020 Jun 1. doi:10.1001/jamanetworkopen.2020.7385
44. Melnick ER, Dyrbye LN, Sinsky CA, et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc. 2020;95(3):476-487. doi:10.1016/j.mayocp.2019.09.024
45. Lee YJ. Comparison of job satisfaction between nonprofit and public employees. Nonprofit Volunt Sect Q. 2016;45(2):295-313. doi:10.1177/0899764015584061
46. Brimhall KC. Inclusion is important... but how do I include? Examining the effects of leader engagement on inclusion, innovation, job satisfaction, and perceived quality of care in a diverse nonprofit health care organization. Nonprofit Volunt Sect Q. 2019;48(4):716-737. doi:10.1177/0899764019829834
47. Moreira MR, Gherman M, Sousa PS. Does innovation influence the performance of healthcare organizations?. Innovation (North Syd). 2017;19(3):335-352. doi:10.1080/14479338.2017.1293489
48. US Department of Veterans Affairs. VA Palo Alto Healthcare System. Updated December 29, 2020. Accessed January 27, 2023. https://www.paloalto.va.gov/about/index.asp
49. Siegel SM, Kaemmerer WF. Measuring the perceived support for innovation in organizations. J Appl Psychol. 1978;63(5):553-562. doi:10.1037/0021-9010.63.5.553
50. Anderson NR, West MA. Measuring climate for work group innovation: development and validation of the team climate inventory. J Organ Behav. 1998;19(3):235-258. doi:10.1002/(SICI)1099-1379(199805)19:3<235::AID-JOB837>3.0.CO;2-C
51. Aarons GA. Measuring provider attitudes toward evidence-based practice: consideration of organizational context and individual differences. Child Adolesc Psychiatr Clin N Am. 2005;14(2):255-viii. doi:10.1016/j.chc.2004.04.008
52. Van der Heijden H. User acceptance of hedonic information systems. MIS Q. 2004;28(4):695-704. doi:10.2307/25148660
53. Venkatesh V, Speier C, Morris MG. User acceptance enablers in individual decision making about technology: Toward an integrated model. Decis Sci. 2002;33(2):297-316. doi:10.1111/j.1540-5915.2002.tb01646.x
54. Puri A, Kim B, Nguyen O, Stolee P, Tung J, Lee J. User acceptance of wrist-worn activity trackers among community-dwelling older adults: mixed method study. JMIR Mhealth Uhealth. 2017;5(11):e173. Published 2017 Nov 15. doi:10.2196/mhealth.8211
55. Huang YC, Backman SJ, Backman KF, Moore D. Exploring user acceptance of 3D virtual worlds in travel and tourism marketing. Tourism Management. 2013;36:490-501. doi:10.1016/j.tourman.2012.09.009
56. Rasimah CM, Ahmad A, Zaman HB. Evaluation of user acceptance of mixed reality technology. AJET. 2011;27(8). doi:10.14742/ajet.899
57. Choi J, Boyle DK. RN workgroup job satisfaction and patient falls in acute care hospital units. J Nurs Adm. 2013;43(11):586-591. doi:10.1097/01.NNA.0000434509.66749.7c58. Tzeng HM, Ketefian S. The relationship between nurses’ job satisfaction and inpatient satisfaction: an exploratory study in a Taiwan teaching hospital. J Nurs Care Qual. 2002;16(2):39-49. doi:10.1097/00001786-200201000-00005
59. Williams ES, Skinner AC. Outcomes of physician job satisfaction: a narrative review, implications, and directions for future research. Health Care Manage Rev. 2003;28(2):119-139. doi:10.1097/00004010-200304000-00004
60. Waldman JD, Kelly F, Arora S, Smith HL. The shocking cost of turnover in health care. Health Care Manage Rev. 2004;29(1):2-7. doi:10.1097/00004010-200401000-00002
61. Hayes LJ, O’Brien-Pallas L, Duffield C, et al. Nurse turnover: a literature review - an update. Int J Nurs Stud. 2012;49(7):887-905. doi:10.1016/j.ijnurstu.2011.10.001
62. US Department of Veterans Affairs. FY 2021/FY 2019 Annual performance plan and report. February 2020. Accessed January 27, 2023. https://www.va.gov/oei/docs/VA2019-2021appr.pdf
Building the health care system of the future requires the thoughtful development and integration of innovative technologies to positively transform care.1-4 Extended reality (XR) represents a spectrum of emerging technologies that have the potential to enhance health care. This includes virtual reality (VR), where a computer-generated visual experience fills the screen; augmented reality (AR), which allows users to see computer-generated images superimposed into an otherwise normal real-world field of view; and mixed reality (MR), which allows users to interact and manipulate computer-generated AR images.
Clinicians and researchers have begun exploring the potential of XR to address a wide variety of health care challenges. A recent systematic review concluded that many clinical studies in this area have small sample sizes and are in the preclinical, proof-of-concept stage, but demonstrate the potential and impact of the underlying VR, AR, and MR technologies.5 Common emerging health care uses for XR include medical education, training, presurgical planning, surgical guidance, distraction therapy for pain and anxiety, and home health indications, including rehabilitation.5-39
Importantly, some researchers have raised concerns regarding the adaptability of the health care workforce with emerging technologies, and their interest in new methods of delivering care.7,39 Successful deployment of any novel health care technology depends on multiple factors, including alignment with staff needs, receptivity to those solutions, customization to specific preferences, and usability.1,3,40-42 Unfortunately, the implementation of some health care technologies, such as electronic health records that did not account for end-user requirements, resulted in employee fatigue, burnout, and negative staffing turnover.42-44 Conversely, elevated employee morale and operational performance have been directly linked to a climate of inclusion and innovation.45-47 In this assessment, we sought to understand US Department of Veterans Affairs (VA) employees’ perceptions and expert opinions related to the introduction of new AR/MR technology.
Methods
The VA Palo Alto Health Care System (VAPAHCS) consists of 3 inpatient hospitals and 7 outpatient clinics, provides a full range of care services to > 90,000 enrolled veterans with 800 hospital beds, 3 nursing homes, and a 100-bed domiciliary. The facility also runs data-driven care projects in research, innovation, and evidence-based practice group under nursing services.48 This project was performed by the VA National Center for Collaborative Healthcare Innovation at the VAPAHCS campus.
The combined technical system used for this assessment included a wireless communication network, AR/MR hardware, and software. Medivis AnatomyX software displayed an interactive human anatomy atlas segmented into about 6000 individual interactive parts. Medivis SurgicalAR received US Food and Drug Administration clearance for presurgical planning and was used to transform and display deidentified diagnostic images (eg, magnetic resonance images and computed tomography) in 3-dimensional (3D) interactive holograms (Figures 1 and 2).
The wireless Microsoft HoloLens 2 AR/MR headset was used for viewing and sensor-enabled collaborative interaction. Multiple participants in the same physical location simultaneously participated and interacted with 3D holograms. The interactive hologram data were enabled for 3D stereoscopic viewing and manipulation.
Setting and Participants
We reviewed published studies that used questionnaires to evaluate institutions’ level of innovation and new technology user acceptance to develop the questionnaire.49-56 Questions and methods were modified, with a focus on understanding the impact on hospital employees. The questionnaire consisted of 2 predemonstration and 3 postdemonstration sections. The first section included background questions. The second (predemonstration) and third (postdemonstration) sections provided matched questions on feelings about the VA. The fourth section included 2 unmatched questions about how the participant felt this technology would impact veterans and whether the VA should implement similar technologies. We used a 5-point Likert scale for sections 2, 3 and 4 (1 = not at all to 5 = extremely). Two unmatched free-text questions asked how the technology could be used in the participant’s hospital service, and another open-ended question asked for any additional comments. To reduce potential reporting bias, 2 VA employees that did not work at VAPAHCS assisted with the survey distribution and collection. VAPAHCS staff were informed by all employee email and facility intranet of the opportunity to participate; the voluntary demonstration and survey took place on February 10 and 11, 2020.
Data Analysis
All matching pre/post questions were analyzed together to determine statistically significant differences using the Wilcoxon signed rank matched pairs test and pooled t test. Survey respondents were also grouped by employment type to evaluate the impact on subgroups. Results were also grouped by VA tenure into 4 categorical 10-year increments (0-10, 11-20, 21-30, 31-40). Additionally, analysis of variance (ANOVA) was performed on employment types and VA tenure to understand whether there was a statistically significant difference in responses by these subgroups. Respondents’ optional free-text answers were manually reviewed by 2 authors (ZPV and DMA), classified, coded by the common themes, and analyzed for comparison.
Results
A total of 166 participants completed the predemonstration survey, which was a requirement for participating in the AR demonstration. Of those, 159 staff members (95.8%) also completed at least part of the postdemonstration paired structured questions, and their results were included in the analysis.
Paired Questions
For questions about how innovative the VA is, 108 of 152 participants (71.1%) provided higher scores after the demonstration, 42 (27.6%) had no change, and 2 (1.3%) provided decreased scores. The mean innovative score increased from 3.4 predemonstration to 4.5 postdemonstration on a Likert scale, which is a 1.1 point increase from predemonstration to postdemonstration (95% CI, 0.9- 1.2) or a 22% increase (95% CI, 18%-24%) (P < .001). Respondents level of excitement about VA also increased with 82 of 157 participants (52.2%) providing higher scores after the demonstration, 71 (45.2%) had no change, and 4 scores (2.5%) decreased. The predemonstration mean excitement score of 3.7 increased to 4.3 postdemonstration, which is a 0.6 point increase from before to after the demonstration (95% CI, 0.5-0.7) or a 12% increase (95% CI, 10%-14%) (P < .001). In the survey, 36 of 149 participants (24.2%) had higher scores for their expectation to continue working at VA postdemonstration, 109 (73.2%) had no change, and 4 scores (2.7%) decreased. The mean employee retention score increased from 4.2 predemonstration to 4.5 postdemonstration, which is a 0.3 point increase between pre/post (95% CI, 0.2-0.4) or a 6% increase (95% CI, 4%-8%) (P < .001)
The pre/post questions were analyzed using 1-way ANOVA by hospital department and VA tenure. The responses by department were not statistically significant. Of the 159 employees assessed, 101 respondents (63.5%) had 0 to 10 years VA tenure, 44 (27.7%) had 11 to 20 years, 10 (6.3%) had 21 to 30 years, and 4 (2.5%) had > 31 to 40 years. Length of VA tenure did not impact respondent excitement. Respondents opinions on innovation in the 0 to 10 year and the 11 to 20 year groups rose from 3.2 and 3.7 predemonstration to 4.3 and 4.6 postdemonstration, respectively (P < .001 for both statistical comparisons) (Table 2). Interestingly, the 0 to 10 group saw a 9% rise from a 4.0 score predemonstration to a 4.4 score postdemonstration (P < .001), indicating that the demonstration had a positive impact on their plans to continue employment at VA (Table 3).
Sex did not play a significant role in how respondents answered questions regarding VA excitement or innovation. However, there was a statistically significant difference in how male and female respondents answered the predemonstration question about their plans to continue VA employment, according to the Wilcoxon rank sum test. Predemonstration, female respondents had a mean score of 4.1, which was 6% lower than the 4.4 score of male colleagues (P = .04). Veteran status did have an impact on how respondents felt about VA innovation, and their plans to continue employment at VA. After the demonstration, veteran staff felt the VA was more innovative compared with nonveterans: 4.7 vs 4.4, respectively, a 6% difference (P = .02) Similarly, for the continued VA employment question, veterans had a mean score of 4.8 vs 4.4 for nonveterans, an 8% difference (P = .03) These results suggest that the demonstration had more of an impact on veteran employees vs nonveteran employees.
Unpaired Questions
There were 2 structured unpaired postdemonstration questions. Respondents agreed that similar technology would impact veteran health care with mean (SD) of 4.6 (0.6) and a median score of 5 on a 5-point Likert scale. Respondents also agreed on the importance of implementing similar innovations with mean (SD) of 4.7 (0.5), and a median score of 5.
The survey asked how this technology could benefit their hospital service department and had 64 responses. Forty-six respondents saw applications for education or patient care/surgery. Other responses shared excitement about the technology and its potential to positively impact patient education. There were 37 responses to the open-ended question: 21 respondents expressed excitement for the technology, and 10 respondents reiterated that the demonstration would be of benefit to patient care/surgery and training.
Discussion
Successful development, design, and deployment of any new health care tool depends on leveraging insights from the employees that will be using and supporting these systems. Correspondingly, understanding the impact that advanced technologies have on health care employees’ satisfaction, morale, and retention is critical to our overall institutional strategy. Our findings show that a one-time experience with AR/MR technology elicited positive employee reactions. Of note, the survey revealed statistically significant improvements in staff’s view of the VA, with the greatest positive impact for questions about innovation, followed by excitement to work at the VA, and likelihood to continue work at the VA. It is very disruptive and costly when health care employees leave, and improving employee satisfaction and morale is important for better patient care and patient satisfaction, which is priority for VAPAHCS leadership.57-62
The paired predemonstration and postdemonstration scores were similarly high, nearing the top threshold available for the Likert scale (4.3 to 4.5). Furthermore, the least incremental improvement for these responses was observed for topics that had the highest initial baseline score. Therefore, the improvements observed for the paired questions may have more to do with the high baseline values.
Of additional interest, the self-reported likelihood of continuing to work at the VA increased the most for female employees, veteran employees, and employees with the least number of years at the VA. These demographic differences have important implications for VA staff recruitment and retention strategies.62 The unpaired questions about the impact on veteran care and whether the VA should continue similar work demonstrated extremely high support with median scores of 5 for both questions. The free-text postdemonstration responses also demonstrate similar positive themes, with a disposition for excitement about both the training and patient care applications for this technology. In addition, respondents felt strongly that this and other similar technologies will positively impact the health care for veterans and that the VA should continue these efforts.
Strengths and Limitations
A strength of this assessment is the ability to evaluate survey responses that were systematically collected and matched from the same individual immediately before and after exposure to the new technology. The free-text responses provided additional important information that both confirmed the results and provided additional valued supplementary guidance for future implementation strategies, which is critical for our translational implementation goals. An additional strength is that the voluntary surveys were managed by non-VAPAHCS colleagues, limiting potential bias. Importantly, the number of respondents allowed a statistically significant assessment of important health care employee metrics. These results have emphasized how being part of an innovative organization, and the introduction of advanced AR/MR technology, improve employees’ satisfaction and morale about where they work as well as their intention to stay at their institution.
A limitation of this assessment was the lack of comparative data for employee acceptance of other technologies at VAPAHCS. This limits our ability to differentiate whether the strong positive results observed in this evaluation were a result of the specific technology assessed, or of new and advanced health care technology in general. Nonetheless, our unpaired questions, which received extremely high scores, also included participant questions about comparing the system with other similar technologies. This assessment was also focused on veteran care, which limits generalizability.
Conclusions
One-time exposure to advanced AR technology for health care significantly increased employee morale as measured by excitement about working at the VA as well as employee intention to continue employment at the VA. These collateral benefits of the technology are particularly important in health care because our employees are our most important asset and improving employee morale equates to better patient care. Positive impacts were most pronounced for women employees, newer VA employees, and employees who are also veterans. These more detailed insights are also positioned to have a direct impact on employee recruitment and retention strategies. Additional valuable insights regarding the most applicable use of the technology in the clinical setting were also obtained.
Acknowledgments
We thank Andrew Spiegelman, Hyewon Kim, Jonathan Sills, and Alexander Erickson for their assistance in developing the survey questions. We also thank Jason Rhodes and Mark Bulson for traveling to our facility to assist with managing the anonymous surveys during the demonstration event.
Building the health care system of the future requires the thoughtful development and integration of innovative technologies to positively transform care.1-4 Extended reality (XR) represents a spectrum of emerging technologies that have the potential to enhance health care. This includes virtual reality (VR), where a computer-generated visual experience fills the screen; augmented reality (AR), which allows users to see computer-generated images superimposed into an otherwise normal real-world field of view; and mixed reality (MR), which allows users to interact and manipulate computer-generated AR images.
Clinicians and researchers have begun exploring the potential of XR to address a wide variety of health care challenges. A recent systematic review concluded that many clinical studies in this area have small sample sizes and are in the preclinical, proof-of-concept stage, but demonstrate the potential and impact of the underlying VR, AR, and MR technologies.5 Common emerging health care uses for XR include medical education, training, presurgical planning, surgical guidance, distraction therapy for pain and anxiety, and home health indications, including rehabilitation.5-39
Importantly, some researchers have raised concerns regarding the adaptability of the health care workforce with emerging technologies, and their interest in new methods of delivering care.7,39 Successful deployment of any novel health care technology depends on multiple factors, including alignment with staff needs, receptivity to those solutions, customization to specific preferences, and usability.1,3,40-42 Unfortunately, the implementation of some health care technologies, such as electronic health records that did not account for end-user requirements, resulted in employee fatigue, burnout, and negative staffing turnover.42-44 Conversely, elevated employee morale and operational performance have been directly linked to a climate of inclusion and innovation.45-47 In this assessment, we sought to understand US Department of Veterans Affairs (VA) employees’ perceptions and expert opinions related to the introduction of new AR/MR technology.
Methods
The VA Palo Alto Health Care System (VAPAHCS) consists of 3 inpatient hospitals and 7 outpatient clinics, provides a full range of care services to > 90,000 enrolled veterans with 800 hospital beds, 3 nursing homes, and a 100-bed domiciliary. The facility also runs data-driven care projects in research, innovation, and evidence-based practice group under nursing services.48 This project was performed by the VA National Center for Collaborative Healthcare Innovation at the VAPAHCS campus.
The combined technical system used for this assessment included a wireless communication network, AR/MR hardware, and software. Medivis AnatomyX software displayed an interactive human anatomy atlas segmented into about 6000 individual interactive parts. Medivis SurgicalAR received US Food and Drug Administration clearance for presurgical planning and was used to transform and display deidentified diagnostic images (eg, magnetic resonance images and computed tomography) in 3-dimensional (3D) interactive holograms (Figures 1 and 2).
The wireless Microsoft HoloLens 2 AR/MR headset was used for viewing and sensor-enabled collaborative interaction. Multiple participants in the same physical location simultaneously participated and interacted with 3D holograms. The interactive hologram data were enabled for 3D stereoscopic viewing and manipulation.
Setting and Participants
We reviewed published studies that used questionnaires to evaluate institutions’ level of innovation and new technology user acceptance to develop the questionnaire.49-56 Questions and methods were modified, with a focus on understanding the impact on hospital employees. The questionnaire consisted of 2 predemonstration and 3 postdemonstration sections. The first section included background questions. The second (predemonstration) and third (postdemonstration) sections provided matched questions on feelings about the VA. The fourth section included 2 unmatched questions about how the participant felt this technology would impact veterans and whether the VA should implement similar technologies. We used a 5-point Likert scale for sections 2, 3 and 4 (1 = not at all to 5 = extremely). Two unmatched free-text questions asked how the technology could be used in the participant’s hospital service, and another open-ended question asked for any additional comments. To reduce potential reporting bias, 2 VA employees that did not work at VAPAHCS assisted with the survey distribution and collection. VAPAHCS staff were informed by all employee email and facility intranet of the opportunity to participate; the voluntary demonstration and survey took place on February 10 and 11, 2020.
Data Analysis
All matching pre/post questions were analyzed together to determine statistically significant differences using the Wilcoxon signed rank matched pairs test and pooled t test. Survey respondents were also grouped by employment type to evaluate the impact on subgroups. Results were also grouped by VA tenure into 4 categorical 10-year increments (0-10, 11-20, 21-30, 31-40). Additionally, analysis of variance (ANOVA) was performed on employment types and VA tenure to understand whether there was a statistically significant difference in responses by these subgroups. Respondents’ optional free-text answers were manually reviewed by 2 authors (ZPV and DMA), classified, coded by the common themes, and analyzed for comparison.
Results
A total of 166 participants completed the predemonstration survey, which was a requirement for participating in the AR demonstration. Of those, 159 staff members (95.8%) also completed at least part of the postdemonstration paired structured questions, and their results were included in the analysis.
Paired Questions
For questions about how innovative the VA is, 108 of 152 participants (71.1%) provided higher scores after the demonstration, 42 (27.6%) had no change, and 2 (1.3%) provided decreased scores. The mean innovative score increased from 3.4 predemonstration to 4.5 postdemonstration on a Likert scale, which is a 1.1 point increase from predemonstration to postdemonstration (95% CI, 0.9- 1.2) or a 22% increase (95% CI, 18%-24%) (P < .001). Respondents level of excitement about VA also increased with 82 of 157 participants (52.2%) providing higher scores after the demonstration, 71 (45.2%) had no change, and 4 scores (2.5%) decreased. The predemonstration mean excitement score of 3.7 increased to 4.3 postdemonstration, which is a 0.6 point increase from before to after the demonstration (95% CI, 0.5-0.7) or a 12% increase (95% CI, 10%-14%) (P < .001). In the survey, 36 of 149 participants (24.2%) had higher scores for their expectation to continue working at VA postdemonstration, 109 (73.2%) had no change, and 4 scores (2.7%) decreased. The mean employee retention score increased from 4.2 predemonstration to 4.5 postdemonstration, which is a 0.3 point increase between pre/post (95% CI, 0.2-0.4) or a 6% increase (95% CI, 4%-8%) (P < .001)
The pre/post questions were analyzed using 1-way ANOVA by hospital department and VA tenure. The responses by department were not statistically significant. Of the 159 employees assessed, 101 respondents (63.5%) had 0 to 10 years VA tenure, 44 (27.7%) had 11 to 20 years, 10 (6.3%) had 21 to 30 years, and 4 (2.5%) had > 31 to 40 years. Length of VA tenure did not impact respondent excitement. Respondents opinions on innovation in the 0 to 10 year and the 11 to 20 year groups rose from 3.2 and 3.7 predemonstration to 4.3 and 4.6 postdemonstration, respectively (P < .001 for both statistical comparisons) (Table 2). Interestingly, the 0 to 10 group saw a 9% rise from a 4.0 score predemonstration to a 4.4 score postdemonstration (P < .001), indicating that the demonstration had a positive impact on their plans to continue employment at VA (Table 3).
Sex did not play a significant role in how respondents answered questions regarding VA excitement or innovation. However, there was a statistically significant difference in how male and female respondents answered the predemonstration question about their plans to continue VA employment, according to the Wilcoxon rank sum test. Predemonstration, female respondents had a mean score of 4.1, which was 6% lower than the 4.4 score of male colleagues (P = .04). Veteran status did have an impact on how respondents felt about VA innovation, and their plans to continue employment at VA. After the demonstration, veteran staff felt the VA was more innovative compared with nonveterans: 4.7 vs 4.4, respectively, a 6% difference (P = .02) Similarly, for the continued VA employment question, veterans had a mean score of 4.8 vs 4.4 for nonveterans, an 8% difference (P = .03) These results suggest that the demonstration had more of an impact on veteran employees vs nonveteran employees.
Unpaired Questions
There were 2 structured unpaired postdemonstration questions. Respondents agreed that similar technology would impact veteran health care with mean (SD) of 4.6 (0.6) and a median score of 5 on a 5-point Likert scale. Respondents also agreed on the importance of implementing similar innovations with mean (SD) of 4.7 (0.5), and a median score of 5.
The survey asked how this technology could benefit their hospital service department and had 64 responses. Forty-six respondents saw applications for education or patient care/surgery. Other responses shared excitement about the technology and its potential to positively impact patient education. There were 37 responses to the open-ended question: 21 respondents expressed excitement for the technology, and 10 respondents reiterated that the demonstration would be of benefit to patient care/surgery and training.
Discussion
Successful development, design, and deployment of any new health care tool depends on leveraging insights from the employees that will be using and supporting these systems. Correspondingly, understanding the impact that advanced technologies have on health care employees’ satisfaction, morale, and retention is critical to our overall institutional strategy. Our findings show that a one-time experience with AR/MR technology elicited positive employee reactions. Of note, the survey revealed statistically significant improvements in staff’s view of the VA, with the greatest positive impact for questions about innovation, followed by excitement to work at the VA, and likelihood to continue work at the VA. It is very disruptive and costly when health care employees leave, and improving employee satisfaction and morale is important for better patient care and patient satisfaction, which is priority for VAPAHCS leadership.57-62
The paired predemonstration and postdemonstration scores were similarly high, nearing the top threshold available for the Likert scale (4.3 to 4.5). Furthermore, the least incremental improvement for these responses was observed for topics that had the highest initial baseline score. Therefore, the improvements observed for the paired questions may have more to do with the high baseline values.
Of additional interest, the self-reported likelihood of continuing to work at the VA increased the most for female employees, veteran employees, and employees with the least number of years at the VA. These demographic differences have important implications for VA staff recruitment and retention strategies.62 The unpaired questions about the impact on veteran care and whether the VA should continue similar work demonstrated extremely high support with median scores of 5 for both questions. The free-text postdemonstration responses also demonstrate similar positive themes, with a disposition for excitement about both the training and patient care applications for this technology. In addition, respondents felt strongly that this and other similar technologies will positively impact the health care for veterans and that the VA should continue these efforts.
Strengths and Limitations
A strength of this assessment is the ability to evaluate survey responses that were systematically collected and matched from the same individual immediately before and after exposure to the new technology. The free-text responses provided additional important information that both confirmed the results and provided additional valued supplementary guidance for future implementation strategies, which is critical for our translational implementation goals. An additional strength is that the voluntary surveys were managed by non-VAPAHCS colleagues, limiting potential bias. Importantly, the number of respondents allowed a statistically significant assessment of important health care employee metrics. These results have emphasized how being part of an innovative organization, and the introduction of advanced AR/MR technology, improve employees’ satisfaction and morale about where they work as well as their intention to stay at their institution.
A limitation of this assessment was the lack of comparative data for employee acceptance of other technologies at VAPAHCS. This limits our ability to differentiate whether the strong positive results observed in this evaluation were a result of the specific technology assessed, or of new and advanced health care technology in general. Nonetheless, our unpaired questions, which received extremely high scores, also included participant questions about comparing the system with other similar technologies. This assessment was also focused on veteran care, which limits generalizability.
Conclusions
One-time exposure to advanced AR technology for health care significantly increased employee morale as measured by excitement about working at the VA as well as employee intention to continue employment at the VA. These collateral benefits of the technology are particularly important in health care because our employees are our most important asset and improving employee morale equates to better patient care. Positive impacts were most pronounced for women employees, newer VA employees, and employees who are also veterans. These more detailed insights are also positioned to have a direct impact on employee recruitment and retention strategies. Additional valuable insights regarding the most applicable use of the technology in the clinical setting were also obtained.
Acknowledgments
We thank Andrew Spiegelman, Hyewon Kim, Jonathan Sills, and Alexander Erickson for their assistance in developing the survey questions. We also thank Jason Rhodes and Mark Bulson for traveling to our facility to assist with managing the anonymous surveys during the demonstration event.
1. World Economic Forum. Health and healthcare in the fourth industrial revolution: Global Future Council on the future of health and healthcare 2016-2018. April 2019. Accessed January 27, 2023. https://www3.weforum.org/docs/WEF__Shaping_the_Future_of_Health_Council_Report.pdf
2. Iveroth E, Fryk P, Rapp B. Information technology strategy and alignment issues in health care organizations. Health Care Manage Rev. 2013;38(3):188-200. doi:10.1097/HMR.0b013e31826119d7
3. Thakur R, Hsu SH, Fontenot G. Innovation in healthcare: issues and future trends. J Bus Res. 2012;65(4):562-569. doi:10.1016/j.jbusres.2011.02.022
4. Thimbleby H. Technology and the future of healthcare. J Public Health Res. 2013;2(3):e28. Published 2013 Dec 1. doi:10.4081/jphr.2013.e28
5. Viglialoro RM, Condino S, Turini G, Carbone M, Ferrari V, Gesi M. augmented reality, mixed reality, and hybrid approach in healthcare simulation: a systematic review. Applied Sciences. 2021;11(5):2338. doi:10.3390/app11052338
6. Rawlins CR, Veigulis Z, Hebert C, Curtin C, Osborne T. Effect of immersive virtual reality on pain and anxiety at a Veterans Affairs health care facility. Front Virt Real. 2021;(2):136. doi:10.3389/frvir.2021.719681
7. Chawdhary G, Shoman N. Emerging artificial intelligence applications in otological imaging. Curr Opin Otolaryngol Head Neck Surg. 2021;29(5):357-364. doi:10.1097/MOO.0000000000000754
8. Asadzadeh A, Samad-Soltani T, Rezaei-Hachesu P. Applications of virtual and augmented reality in infectious disease epidemics with a focus on the COVID-19 outbreak. Inform Med Unlocked. 2021;24:100579. doi:10.1016/j.imu.2021.100579
9. Ashwini KB, Savitha R, Harish A. Application of augmented reality technology for home healthcare product visualization. ECS Transas. 2022;107(1):10921. doi:10.1149/10701.10921ecst
10. Brooks AL. VR/Technologies for Rehabilitation. In: Brooks AL, Brahman S, Kapralos B, Nakajima A, Tyerman J, Jain LC, eds. Recent Advances in Technologies for Inclusive Well-Being Virtual Patients, Gamification and Simulation. Intelligent Systems Reference Library. Springer; 2021:241-252. doi:10.1007/978-3-030-59608-8_13
11. Koulouris D, Menychtas A, Maglogiannis I. An IoT-enabled platform for the assessment of physical and mental activities utilizing augmented reality exergaming. Sensors (Basel). 2022;22(9):3181. Published 2022 Apr 21. doi:10.3390/s22093181
12. Deiss YR, Korkut S, Inglese T. Increase therapy understanding and medication adherence for patients with inflammatory skin diseases through augmented reality. Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022. 2022:21-40. doi:10.1007/978-3-031-06018-2_2
13. Bertino E, Gao W, Steffan B, et al, eds. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer; 2022:21-40.
14. Ruhaiyem NIR, Mazlan NA. Image Modeling Through Augmented Reality for Skin Allergies Recognition. Lecture Notes on Data Engineering and Communications Technologies. 2021:72-79. doi: 10.1007/978-3-030-70713-2_8
15. Park BJ, Perkons NR, Profka E, et al. Three-dimensional augmented reality visualization informs locoregional therapy in a translational model of hepatocellular carcinoma. J Vasc Interv Radiol. 2020;31(10):1612-1618.e1. doi:10.1016/j.jvir.2020.01.028
16. Leo J, Zhou Z, Yang H, et al, eds. Interactive cardiovascular surgical planning via augmented reality. 5th Asian CHI Symposium 2021; 2021. doi:10.1145/3429360.3468195
17. Zuo Y, Jiang T, Dou J, et al. A novel evaluation model for a mixed-reality surgical navigation system: where Microsoft Hololens meets the operating room. Surg Innov. 2020;27(2):193-202. doi:10.1177/1553350619893236
18. Ghaednia H, Fourman MS, Lans A, et al. Augmented and virtual reality in spine surgery, current applications and future potentials. Spine J. 2021;21(10):1617-1625. doi:10.1016/j.spinee.2021.03.018
19. Liu Y, Lee MG, Kim JS. Spine surgery assisted by augmented reality: where have we been?. Yonsei Med J. 2022;63(4):305-316. doi:10.3349/ymj.2022.63.4.305
20. Kimmel S, Cobus V, Heuten W, eds. opticARe—augmented reality mobile patient monitoring in intensive care units. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST; 2021. doi:10.1145/3489849.3489852
21. Voštinár P, Horváthová D, Mitter M, Bako M. The look at the various uses of VR. Open Computer Sci. 2021;11(1):241-250. doi:10.1515/comp-2020-0123
22. Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: a meta-analysis of randomized controlled studies. BMC Med Educ. 2020;20(1):127. Published 2020 Apr 25. doi:10.1186/s12909-020-1994-z
23. Ricci S, Calandrino A, Borgonovo G, Chirico M, Casadio M. Viewpoint: virtual and augmented reality in basic and advanced life support training. JMIR Serious Games. 2022;10(1):e28595. Published 2022 Mar 23. doi:10.2196/28595
24. Ricci S, Mobilio GA, Calandrino A, et al. RiNeo MR: A mixed-reality tool for newborn life support training. Annu Int Conf IEEE Eng Med Biol Soc. 2021;2021:5043-5046. doi:10.1109/EMBC46164.2021.9629612
25. Dhar P, Rocks T, Samarasinghe RM, Stephenson G, Smith C. Augmented reality in medical education: students’ experiences and learning outcomes. Med Educ Online. 2021;26(1):1953953. doi:10.1080/10872981.2021.1953953
26. Pears M, Konstantinidis S. The future of immersive technology in global surgery education [published online ahead of print, 2021 Jul 1]. Indian J Surg. 2021;84(suppl 1):1-5. doi:10.1007/s12262-021-02998-6
27. Liang CJ, Start C, Boley H, Kamat VR, Menassa CC, Aebersold M. Enhancing stroke assessment simulation experience in clinical training using augmented reality. Virt Real. 2021;25(3):575-584. doi:10.1007/s10055-020-00475-1
28. Lacey G, Gozdzielewska L, McAloney-Kocaman K, Ruttle J, Cronin S, Price L. Psychomotor learning theory informing the design and evaluation of an interactive augmented reality hand hygiene training app for healthcare workers. Educ Inf Technol. 2022;27(3):3813-3832. doi:10.1007/s10639-021-10752-4
29. Ryan GV, Callaghan S, Rafferty A, Higgins MF, Mangina E, McAuliffe F. Learning outcomes of immersive technologies in health care student education: systematic review of the literature. J Med Internet Res. 2022;24(2):e30082. Published 2022 Feb 1. doi:10.2196/30082
30. Yu FU, Yan HU, Sundstedt V. A Systematic literature review of virtual, augmented, and mixed reality game applications in healthcare. ACM Trans Comput Healthcare. 2022;3(2);1-27. doi:10.1145/3472303
31. Weeks JK, Amiel JM. Enhancing neuroanatomy education with augmented reality. Med Educ. 2019;53(5):516-517. doi:10.1111/medu.13843
32. Williams MA, McVeigh J, Handa AI, Lee R. Augmented reality in surgical training: a systematic review. Postgrad Med J. 2020;96(1139):537-542. doi:10.1136/postgradmedj-2020-137600
33. Triepels CPR, Smeets CFA, Notten KJB, et al. Does three-dimensional anatomy improve student understanding? Clin Anat. 2020;33(1):25-33. doi:10.1002/ca.23405
34. Pietruski P, Majak M, S´wia¸tek-Najwer E, et al. Supporting fibula free flap harvest with augmented reality: A proof-of-concept study. Laryngoscope. 2020;130(5):1173-1179. doi:10.1002/lary.28090
35. Perkins SL, Krajancich B, Yang CJ, Hargreaves BA, Daniel BL, Berry MF. A patient-specific mixed-reality visualization tool for thoracic surgical planning. Ann Thorac Surg. 2020;110(1):290-295. doi:10.1016/j.athoracsur.2020.01.060
36. Müller F, Roner S, Liebmann F, Spirig JM, Fürnstahl P, Farshad M. Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging. Spine J. 2020;20(4):621-628. doi:10.1016/j.spinee.2019.10.012
37. Kaplan AD, Cruit J, Endsley M, Beers SM, Sawyer BD, Hancock PA. The effects of virtual reality, augmented reality, and mixed reality as training enhancement methods: a meta-analysis. Hum Factors. 2021;63(4):706-726. doi:10.1177/0018720820904229
38. Jud L, Fotouhi J, Andronic O, et al. Applicability of augmented reality in orthopedic surgery - a systematic review. BMC Musculoskelet Disord. 2020;21(1):103. Published 2020 Feb 15. doi:10.1186/s12891-020-3110-2
39. Ara J, Karim FB, Alsubaie MSA, et al. Comprehensive analysis of augmented reality technology in modern healthcare system. Int J Adv Comput Sci Appl. 2021;12(6):845-854. doi:10.14569/IJACSA.2021.0120698
40. Webster A, Gardner J. Aligning technology and institutional readiness: the adoption of innovation. Technol Anal Strateg Manag. 2019;31(10):1229-1241. doi:10.1080/09537325.2019.1601694
41. Hastall MR, Dockweiler C, Mühlhaus J. achieving end user acceptance: building blocks for an evidence-based user-centered framework for health technology development and assessment. In: Antona, M, Stephanidis C, eds. Universal Access in Human–Computer Interaction. Human and Technological Environments. UAHCI 2017. Lecture Notes in Computer Science, vol 10279. Springer, Cham; 2017. doi:10.1007/978-3-319-58700-4_2
42. Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc. 2015;22(6):1179-1182. doi:10.1093/jamia/ocv050
43. Khairat S, Coleman C, Ottmar P, Jayachander DI, Bice T, Carson SS. Association of Electronic Health Record Use With Physician Fatigue and Efficiency. JAMA Netw Open. 2020;3(6):e207385. Published 2020 Jun 1. doi:10.1001/jamanetworkopen.2020.7385
44. Melnick ER, Dyrbye LN, Sinsky CA, et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc. 2020;95(3):476-487. doi:10.1016/j.mayocp.2019.09.024
45. Lee YJ. Comparison of job satisfaction between nonprofit and public employees. Nonprofit Volunt Sect Q. 2016;45(2):295-313. doi:10.1177/0899764015584061
46. Brimhall KC. Inclusion is important... but how do I include? Examining the effects of leader engagement on inclusion, innovation, job satisfaction, and perceived quality of care in a diverse nonprofit health care organization. Nonprofit Volunt Sect Q. 2019;48(4):716-737. doi:10.1177/0899764019829834
47. Moreira MR, Gherman M, Sousa PS. Does innovation influence the performance of healthcare organizations?. Innovation (North Syd). 2017;19(3):335-352. doi:10.1080/14479338.2017.1293489
48. US Department of Veterans Affairs. VA Palo Alto Healthcare System. Updated December 29, 2020. Accessed January 27, 2023. https://www.paloalto.va.gov/about/index.asp
49. Siegel SM, Kaemmerer WF. Measuring the perceived support for innovation in organizations. J Appl Psychol. 1978;63(5):553-562. doi:10.1037/0021-9010.63.5.553
50. Anderson NR, West MA. Measuring climate for work group innovation: development and validation of the team climate inventory. J Organ Behav. 1998;19(3):235-258. doi:10.1002/(SICI)1099-1379(199805)19:3<235::AID-JOB837>3.0.CO;2-C
51. Aarons GA. Measuring provider attitudes toward evidence-based practice: consideration of organizational context and individual differences. Child Adolesc Psychiatr Clin N Am. 2005;14(2):255-viii. doi:10.1016/j.chc.2004.04.008
52. Van der Heijden H. User acceptance of hedonic information systems. MIS Q. 2004;28(4):695-704. doi:10.2307/25148660
53. Venkatesh V, Speier C, Morris MG. User acceptance enablers in individual decision making about technology: Toward an integrated model. Decis Sci. 2002;33(2):297-316. doi:10.1111/j.1540-5915.2002.tb01646.x
54. Puri A, Kim B, Nguyen O, Stolee P, Tung J, Lee J. User acceptance of wrist-worn activity trackers among community-dwelling older adults: mixed method study. JMIR Mhealth Uhealth. 2017;5(11):e173. Published 2017 Nov 15. doi:10.2196/mhealth.8211
55. Huang YC, Backman SJ, Backman KF, Moore D. Exploring user acceptance of 3D virtual worlds in travel and tourism marketing. Tourism Management. 2013;36:490-501. doi:10.1016/j.tourman.2012.09.009
56. Rasimah CM, Ahmad A, Zaman HB. Evaluation of user acceptance of mixed reality technology. AJET. 2011;27(8). doi:10.14742/ajet.899
57. Choi J, Boyle DK. RN workgroup job satisfaction and patient falls in acute care hospital units. J Nurs Adm. 2013;43(11):586-591. doi:10.1097/01.NNA.0000434509.66749.7c58. Tzeng HM, Ketefian S. The relationship between nurses’ job satisfaction and inpatient satisfaction: an exploratory study in a Taiwan teaching hospital. J Nurs Care Qual. 2002;16(2):39-49. doi:10.1097/00001786-200201000-00005
59. Williams ES, Skinner AC. Outcomes of physician job satisfaction: a narrative review, implications, and directions for future research. Health Care Manage Rev. 2003;28(2):119-139. doi:10.1097/00004010-200304000-00004
60. Waldman JD, Kelly F, Arora S, Smith HL. The shocking cost of turnover in health care. Health Care Manage Rev. 2004;29(1):2-7. doi:10.1097/00004010-200401000-00002
61. Hayes LJ, O’Brien-Pallas L, Duffield C, et al. Nurse turnover: a literature review - an update. Int J Nurs Stud. 2012;49(7):887-905. doi:10.1016/j.ijnurstu.2011.10.001
62. US Department of Veterans Affairs. FY 2021/FY 2019 Annual performance plan and report. February 2020. Accessed January 27, 2023. https://www.va.gov/oei/docs/VA2019-2021appr.pdf
1. World Economic Forum. Health and healthcare in the fourth industrial revolution: Global Future Council on the future of health and healthcare 2016-2018. April 2019. Accessed January 27, 2023. https://www3.weforum.org/docs/WEF__Shaping_the_Future_of_Health_Council_Report.pdf
2. Iveroth E, Fryk P, Rapp B. Information technology strategy and alignment issues in health care organizations. Health Care Manage Rev. 2013;38(3):188-200. doi:10.1097/HMR.0b013e31826119d7
3. Thakur R, Hsu SH, Fontenot G. Innovation in healthcare: issues and future trends. J Bus Res. 2012;65(4):562-569. doi:10.1016/j.jbusres.2011.02.022
4. Thimbleby H. Technology and the future of healthcare. J Public Health Res. 2013;2(3):e28. Published 2013 Dec 1. doi:10.4081/jphr.2013.e28
5. Viglialoro RM, Condino S, Turini G, Carbone M, Ferrari V, Gesi M. augmented reality, mixed reality, and hybrid approach in healthcare simulation: a systematic review. Applied Sciences. 2021;11(5):2338. doi:10.3390/app11052338
6. Rawlins CR, Veigulis Z, Hebert C, Curtin C, Osborne T. Effect of immersive virtual reality on pain and anxiety at a Veterans Affairs health care facility. Front Virt Real. 2021;(2):136. doi:10.3389/frvir.2021.719681
7. Chawdhary G, Shoman N. Emerging artificial intelligence applications in otological imaging. Curr Opin Otolaryngol Head Neck Surg. 2021;29(5):357-364. doi:10.1097/MOO.0000000000000754
8. Asadzadeh A, Samad-Soltani T, Rezaei-Hachesu P. Applications of virtual and augmented reality in infectious disease epidemics with a focus on the COVID-19 outbreak. Inform Med Unlocked. 2021;24:100579. doi:10.1016/j.imu.2021.100579
9. Ashwini KB, Savitha R, Harish A. Application of augmented reality technology for home healthcare product visualization. ECS Transas. 2022;107(1):10921. doi:10.1149/10701.10921ecst
10. Brooks AL. VR/Technologies for Rehabilitation. In: Brooks AL, Brahman S, Kapralos B, Nakajima A, Tyerman J, Jain LC, eds. Recent Advances in Technologies for Inclusive Well-Being Virtual Patients, Gamification and Simulation. Intelligent Systems Reference Library. Springer; 2021:241-252. doi:10.1007/978-3-030-59608-8_13
11. Koulouris D, Menychtas A, Maglogiannis I. An IoT-enabled platform for the assessment of physical and mental activities utilizing augmented reality exergaming. Sensors (Basel). 2022;22(9):3181. Published 2022 Apr 21. doi:10.3390/s22093181
12. Deiss YR, Korkut S, Inglese T. Increase therapy understanding and medication adherence for patients with inflammatory skin diseases through augmented reality. Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022. 2022:21-40. doi:10.1007/978-3-031-06018-2_2
13. Bertino E, Gao W, Steffan B, et al, eds. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer; 2022:21-40.
14. Ruhaiyem NIR, Mazlan NA. Image Modeling Through Augmented Reality for Skin Allergies Recognition. Lecture Notes on Data Engineering and Communications Technologies. 2021:72-79. doi: 10.1007/978-3-030-70713-2_8
15. Park BJ, Perkons NR, Profka E, et al. Three-dimensional augmented reality visualization informs locoregional therapy in a translational model of hepatocellular carcinoma. J Vasc Interv Radiol. 2020;31(10):1612-1618.e1. doi:10.1016/j.jvir.2020.01.028
16. Leo J, Zhou Z, Yang H, et al, eds. Interactive cardiovascular surgical planning via augmented reality. 5th Asian CHI Symposium 2021; 2021. doi:10.1145/3429360.3468195
17. Zuo Y, Jiang T, Dou J, et al. A novel evaluation model for a mixed-reality surgical navigation system: where Microsoft Hololens meets the operating room. Surg Innov. 2020;27(2):193-202. doi:10.1177/1553350619893236
18. Ghaednia H, Fourman MS, Lans A, et al. Augmented and virtual reality in spine surgery, current applications and future potentials. Spine J. 2021;21(10):1617-1625. doi:10.1016/j.spinee.2021.03.018
19. Liu Y, Lee MG, Kim JS. Spine surgery assisted by augmented reality: where have we been?. Yonsei Med J. 2022;63(4):305-316. doi:10.3349/ymj.2022.63.4.305
20. Kimmel S, Cobus V, Heuten W, eds. opticARe—augmented reality mobile patient monitoring in intensive care units. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST; 2021. doi:10.1145/3489849.3489852
21. Voštinár P, Horváthová D, Mitter M, Bako M. The look at the various uses of VR. Open Computer Sci. 2021;11(1):241-250. doi:10.1515/comp-2020-0123
22. Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: a meta-analysis of randomized controlled studies. BMC Med Educ. 2020;20(1):127. Published 2020 Apr 25. doi:10.1186/s12909-020-1994-z
23. Ricci S, Calandrino A, Borgonovo G, Chirico M, Casadio M. Viewpoint: virtual and augmented reality in basic and advanced life support training. JMIR Serious Games. 2022;10(1):e28595. Published 2022 Mar 23. doi:10.2196/28595
24. Ricci S, Mobilio GA, Calandrino A, et al. RiNeo MR: A mixed-reality tool for newborn life support training. Annu Int Conf IEEE Eng Med Biol Soc. 2021;2021:5043-5046. doi:10.1109/EMBC46164.2021.9629612
25. Dhar P, Rocks T, Samarasinghe RM, Stephenson G, Smith C. Augmented reality in medical education: students’ experiences and learning outcomes. Med Educ Online. 2021;26(1):1953953. doi:10.1080/10872981.2021.1953953
26. Pears M, Konstantinidis S. The future of immersive technology in global surgery education [published online ahead of print, 2021 Jul 1]. Indian J Surg. 2021;84(suppl 1):1-5. doi:10.1007/s12262-021-02998-6
27. Liang CJ, Start C, Boley H, Kamat VR, Menassa CC, Aebersold M. Enhancing stroke assessment simulation experience in clinical training using augmented reality. Virt Real. 2021;25(3):575-584. doi:10.1007/s10055-020-00475-1
28. Lacey G, Gozdzielewska L, McAloney-Kocaman K, Ruttle J, Cronin S, Price L. Psychomotor learning theory informing the design and evaluation of an interactive augmented reality hand hygiene training app for healthcare workers. Educ Inf Technol. 2022;27(3):3813-3832. doi:10.1007/s10639-021-10752-4
29. Ryan GV, Callaghan S, Rafferty A, Higgins MF, Mangina E, McAuliffe F. Learning outcomes of immersive technologies in health care student education: systematic review of the literature. J Med Internet Res. 2022;24(2):e30082. Published 2022 Feb 1. doi:10.2196/30082
30. Yu FU, Yan HU, Sundstedt V. A Systematic literature review of virtual, augmented, and mixed reality game applications in healthcare. ACM Trans Comput Healthcare. 2022;3(2);1-27. doi:10.1145/3472303
31. Weeks JK, Amiel JM. Enhancing neuroanatomy education with augmented reality. Med Educ. 2019;53(5):516-517. doi:10.1111/medu.13843
32. Williams MA, McVeigh J, Handa AI, Lee R. Augmented reality in surgical training: a systematic review. Postgrad Med J. 2020;96(1139):537-542. doi:10.1136/postgradmedj-2020-137600
33. Triepels CPR, Smeets CFA, Notten KJB, et al. Does three-dimensional anatomy improve student understanding? Clin Anat. 2020;33(1):25-33. doi:10.1002/ca.23405
34. Pietruski P, Majak M, S´wia¸tek-Najwer E, et al. Supporting fibula free flap harvest with augmented reality: A proof-of-concept study. Laryngoscope. 2020;130(5):1173-1179. doi:10.1002/lary.28090
35. Perkins SL, Krajancich B, Yang CJ, Hargreaves BA, Daniel BL, Berry MF. A patient-specific mixed-reality visualization tool for thoracic surgical planning. Ann Thorac Surg. 2020;110(1):290-295. doi:10.1016/j.athoracsur.2020.01.060
36. Müller F, Roner S, Liebmann F, Spirig JM, Fürnstahl P, Farshad M. Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging. Spine J. 2020;20(4):621-628. doi:10.1016/j.spinee.2019.10.012
37. Kaplan AD, Cruit J, Endsley M, Beers SM, Sawyer BD, Hancock PA. The effects of virtual reality, augmented reality, and mixed reality as training enhancement methods: a meta-analysis. Hum Factors. 2021;63(4):706-726. doi:10.1177/0018720820904229
38. Jud L, Fotouhi J, Andronic O, et al. Applicability of augmented reality in orthopedic surgery - a systematic review. BMC Musculoskelet Disord. 2020;21(1):103. Published 2020 Feb 15. doi:10.1186/s12891-020-3110-2
39. Ara J, Karim FB, Alsubaie MSA, et al. Comprehensive analysis of augmented reality technology in modern healthcare system. Int J Adv Comput Sci Appl. 2021;12(6):845-854. doi:10.14569/IJACSA.2021.0120698
40. Webster A, Gardner J. Aligning technology and institutional readiness: the adoption of innovation. Technol Anal Strateg Manag. 2019;31(10):1229-1241. doi:10.1080/09537325.2019.1601694
41. Hastall MR, Dockweiler C, Mühlhaus J. achieving end user acceptance: building blocks for an evidence-based user-centered framework for health technology development and assessment. In: Antona, M, Stephanidis C, eds. Universal Access in Human–Computer Interaction. Human and Technological Environments. UAHCI 2017. Lecture Notes in Computer Science, vol 10279. Springer, Cham; 2017. doi:10.1007/978-3-319-58700-4_2
42. Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc. 2015;22(6):1179-1182. doi:10.1093/jamia/ocv050
43. Khairat S, Coleman C, Ottmar P, Jayachander DI, Bice T, Carson SS. Association of Electronic Health Record Use With Physician Fatigue and Efficiency. JAMA Netw Open. 2020;3(6):e207385. Published 2020 Jun 1. doi:10.1001/jamanetworkopen.2020.7385
44. Melnick ER, Dyrbye LN, Sinsky CA, et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc. 2020;95(3):476-487. doi:10.1016/j.mayocp.2019.09.024
45. Lee YJ. Comparison of job satisfaction between nonprofit and public employees. Nonprofit Volunt Sect Q. 2016;45(2):295-313. doi:10.1177/0899764015584061
46. Brimhall KC. Inclusion is important... but how do I include? Examining the effects of leader engagement on inclusion, innovation, job satisfaction, and perceived quality of care in a diverse nonprofit health care organization. Nonprofit Volunt Sect Q. 2019;48(4):716-737. doi:10.1177/0899764019829834
47. Moreira MR, Gherman M, Sousa PS. Does innovation influence the performance of healthcare organizations?. Innovation (North Syd). 2017;19(3):335-352. doi:10.1080/14479338.2017.1293489
48. US Department of Veterans Affairs. VA Palo Alto Healthcare System. Updated December 29, 2020. Accessed January 27, 2023. https://www.paloalto.va.gov/about/index.asp
49. Siegel SM, Kaemmerer WF. Measuring the perceived support for innovation in organizations. J Appl Psychol. 1978;63(5):553-562. doi:10.1037/0021-9010.63.5.553
50. Anderson NR, West MA. Measuring climate for work group innovation: development and validation of the team climate inventory. J Organ Behav. 1998;19(3):235-258. doi:10.1002/(SICI)1099-1379(199805)19:3<235::AID-JOB837>3.0.CO;2-C
51. Aarons GA. Measuring provider attitudes toward evidence-based practice: consideration of organizational context and individual differences. Child Adolesc Psychiatr Clin N Am. 2005;14(2):255-viii. doi:10.1016/j.chc.2004.04.008
52. Van der Heijden H. User acceptance of hedonic information systems. MIS Q. 2004;28(4):695-704. doi:10.2307/25148660
53. Venkatesh V, Speier C, Morris MG. User acceptance enablers in individual decision making about technology: Toward an integrated model. Decis Sci. 2002;33(2):297-316. doi:10.1111/j.1540-5915.2002.tb01646.x
54. Puri A, Kim B, Nguyen O, Stolee P, Tung J, Lee J. User acceptance of wrist-worn activity trackers among community-dwelling older adults: mixed method study. JMIR Mhealth Uhealth. 2017;5(11):e173. Published 2017 Nov 15. doi:10.2196/mhealth.8211
55. Huang YC, Backman SJ, Backman KF, Moore D. Exploring user acceptance of 3D virtual worlds in travel and tourism marketing. Tourism Management. 2013;36:490-501. doi:10.1016/j.tourman.2012.09.009
56. Rasimah CM, Ahmad A, Zaman HB. Evaluation of user acceptance of mixed reality technology. AJET. 2011;27(8). doi:10.14742/ajet.899
57. Choi J, Boyle DK. RN workgroup job satisfaction and patient falls in acute care hospital units. J Nurs Adm. 2013;43(11):586-591. doi:10.1097/01.NNA.0000434509.66749.7c58. Tzeng HM, Ketefian S. The relationship between nurses’ job satisfaction and inpatient satisfaction: an exploratory study in a Taiwan teaching hospital. J Nurs Care Qual. 2002;16(2):39-49. doi:10.1097/00001786-200201000-00005
59. Williams ES, Skinner AC. Outcomes of physician job satisfaction: a narrative review, implications, and directions for future research. Health Care Manage Rev. 2003;28(2):119-139. doi:10.1097/00004010-200304000-00004
60. Waldman JD, Kelly F, Arora S, Smith HL. The shocking cost of turnover in health care. Health Care Manage Rev. 2004;29(1):2-7. doi:10.1097/00004010-200401000-00002
61. Hayes LJ, O’Brien-Pallas L, Duffield C, et al. Nurse turnover: a literature review - an update. Int J Nurs Stud. 2012;49(7):887-905. doi:10.1016/j.ijnurstu.2011.10.001
62. US Department of Veterans Affairs. FY 2021/FY 2019 Annual performance plan and report. February 2020. Accessed January 27, 2023. https://www.va.gov/oei/docs/VA2019-2021appr.pdf
