User login
Vocational rehabilitation (VR) interventions are offered through Compensated Work Therapy (CWT) as part of clinical care in the Veterans Health Administration (VHA) to improve employment and quality of life outcomes for veterans with life-altering disabilities.1–5 CWT vocational services range from assessment, vocational counseling, and treatment plan development to job placement, coaching, and follow-along support.1 However, many veterans receive care in community-based clinics that are not staffed with a VR specialist (VRS) to provide these services.6–8 Telehealth may increase patient access to VR, especially for rural veterans and those with travel barriers, but it is not known whether veterans and VRS would find this to be a satisfactory service delivery method.8,9 This paper examines veteran and VRS provider perspectives on VR provided by telehealth (VRtele) as part of a VHA clinical demonstration project. To our knowledge, this is the first report of using real-time, clinic-based VRtele.
Methods
The Rural Veterans Supported Employment Telerehabilitation Initiative (RVSETI) was conducted as a field-initiated demonstration project at 2 US Department of Veterans Affairs (VA) medical centers (VAMCs) in Florida between 2014 and 2016: James A. Haley Veterans’ Hospital & Clinics (Tampa) and Malcom Randall VAMC (Gainesville). This retrospective evaluation of its first year did not require institutional review board approval as it was determined to be a quality improvement project by the local research service.
The patient population for the project was veterans with disabilities who were referred by clinical consults to the CWT service, a recovery-oriented vocational program. During the project years, veterans were offered the option of receiving VR services, such as supported employment, community-based employment services, or vocational assistance, through VRtele rather than traditional face-to-face meetings. The specific interventions delivered included patient orientation, interview assessment, treatment plan development, referral activities, vocational counseling, assessment of workplace for accommodation needs, vocational case management, and other employment supports. VR staff participating in the project included 2 VR supervisors, 1 supported employment mentor trainer, and 5 VRSs.
Each clinic was set up for VRtele, and codes were added to the electronic health record (EHR) to ensure proper documentation. Participating VRSs completed teleconferencing training, including a skills assessment using the equipment for real-time, high-quality video streaming over an encrypted network to provide services in a patient’s home or other remote locations. VRS staff provided veterans with instructions on using a VA-provided tablet or their own device and assisted them with establishing connectivity with the network. Video equipment included speakers, camera, and headphones connected to the desktop computer or laptop of the VRS. A patient’s first VRtele
Demographic data, primary diagnosis, VR usage data, and zip codes of participating veterans were extracted from the EHR. Veterans completed a 2-part satisfaction survey administered 90 days after enrollment and at discharge. Part 1 was composed of 15 items, most with a 5-point Likert scale (higher ratings indicated greater satisfaction), on various aspects of the VRtele experience, such as audio and video quality and wait times.10 Part 2 addressed VR services and the VRS and consisted of 8 Likert scale items with the option to add a comment for each and 2 open-ended items that asked the participant to list what they liked best and least about VRtele.
Semistructured, in-person 30- to 60-minute interviews were conducted with VRSs at the initiation of VRtele
After ≥ 2 months of VRtele use
Analyses
Descriptive statistics were used for EHR data and satisfaction surveys. For qualitative analysis, each transcript was read in full by 2 researchers to get an overview of the data, and a rapid analysis approach was used to identify central themes focused on how technology was used and the experiences of the participants.11,12 Relevant text for each topic was tabulated, and a summary table was created that highlighted overlapping ideas discussed by the interviewees as well as differences.
Results
Of the 22 veterans who participated in the project, 11 completed satisfaction surveys and 4 participated in qualitative interviews. The rural and nonrural groups did not differ demographically or by diagnosis, which was predominantly mental health related. Only 1 veteran in each group owned a tablet; the majority of both groups required VA-issued devices: 80% (n = 8) rural and 91.7% (n = 11) nonrural. The number of VRtele sessions for the groups also was similar, 53 for rural and 60 for nonrural, as was the mean (SD) number of sessions per veteran: 5.3 (SD, 3.2) rural and 5.0 (SD, 2.5) urban. Overall, 63 miles per session were saved, mostly for rural veterans, and the number of mean (SD) miles saved per veteran was greater for rural than nonrural veterans: 379.2 (243.0) and 256.1 (275.9), respectively. One veteran who moved to a different state during the program continued VRtele at the new location. In a qualitative sampling of 5 VRtele sessions, all the VRSs used office desktop computers.
Level of satisfaction with aspects of VRtele related to the technology rated was consistently > 4 on the Likert scale. The lowest mean (SD) ratings were 4.2 (1.0) for audio quality and 4.4 (0.5) for video quality, and the highest rating was given for equipment operation explanation and privacy was respected, 4.9 (0.3) for both. All questions related to satisfaction with services were also rated high: The mean (SD) lowest ratings were 4.3 (1.0) given to both vocational needs 4.3 (1.0) and tasks effectively helped achieve goals 4.3 (0.7). The highest mean (SD) ratings were 4.6 (0.5) given to VR program service explained and 4.7 (0.5) for appointment timeliness.
Qualitative Results
At first, some VRSs thought the teleconferencing system might be difficult or awkward to use, but they found it easier to set up than expected and seamless to use. VRS staff reported being surprised at how well it worked despite some issues that occurred with loading the software. Once loaded, however, the connection worked well, one VRS noting that following step-by-step instructions solved the problem. Some VRSs indicated they did not invite all the veterans on their caseload to participate in VRtele due to concerns with the patient’s familiarity with technology, but one VRS stated, “I haven’t had anybody that failed to do a [session] that I couldn’t get them up and running within a few minutes.”
When working in the community, VRSs reported using laptops for VRtele but found that these devices were unreliable due to lack of internet access and were slow to start; several VRSs thought tablets would have been more helpful. Some veterans reported technical glitches, lack of comfort with technology, or a problem with sound due to a tablet’s protective case blocking the speakers. To solve the sound issue, a veteran used headphones. This veteran also explained that the log-on process required a new password every time, so he would keep a pen and paper ready to write it down. Because signing in and setting up takes a little time, this veteran and his VRS agreed to start connecting 5 minutes before their meeting time to allow for that set- up time.
Initially, some VRSs expressed concern that transitioning to VRtele would affect the quality of interactions with the veterans. However, VRSs also identified strengths of VRtele, including flexibility, saved time, and increased interaction. One VRS discussed a veteran’s adaptation by saying, “I think he feels even more involved in his plan [and] enjoys the increased interaction.” Veterans reported enjoying using tablets and identified the main strength of VRtele as being able to talk face-to-face with the VRS. Echoing the VRSs, veterans reported teleconferencing saved time by avoiding travel and enabled spontaneous meetings. One of the veterans summed up the benefits of using VRtele: “I’d rather just connect. It’s going to take us 40 to 50 minutes [to meet in person] when we can just connect right here and it takes 15 to 20. We don’t have to go through the driving.… So this right here, doing it ahead of time and having the appointment, it’s a lot easier.”
In their interviews, VRSs talked about enjoying VRtele. A VRS explained: “It makes it a lot easier. It makes me feel less guilty. This way [veterans] don’t have to use their gas money, use their time. I know [the veteran] had something else he needed to do today.” Thus, both veterans and VRSs were satisfied with their VRtele experiences.
Discussion
This first report on the perspective of providers and veterans using VRtele suggests that it is a viable option for service delivery and that is highly satisfactory for serving veterans with disabilities, many of whom live in rural areas or have travel barriers. These findings are consistent with data on telerehabilitation for veterans with cognitive, physical, and mental disabilities.13-22 Further, the data support the notion of using VRtele to facilitate long-term VR follow-up for persons with disabilities, as illustrated by successful continuation of vocational services after a veteran moved out of state.23
Similar to other reports, our experience highlighted 2 factors that affect successful VRtele: (1) Troubleshooting technology barriers for both VR providers and clients; and (2) supportive leadership to facilitate implementation
Changes to technology and increased usage of VA Video Connect may indicate that the barriers identified from the earlier process described here have been diminished or eliminated. More evaluation is needed to assess whether system upgrades have increased ease of use and access for veterans with disabilities.
Conclusions
Encouragingly, this clinical demonstration project showed that both providers and clients recognize the benefits of VRtele. Patient satisfaction and decreased travel costs were clear advantages to using VRtele for this small group of veterans who had barriers to care due to travel or disability barriers. As this program evaluation was limited by a small sample, absence of a comparison group, and lack of outcome data (eg, employment rates, hours, wages, retention), future research is needed on implementation and outcomes of VRtele
Acknowledgments
The authors thank Lynn Dirk, MAMC, for substantial editorial assistance. This material was based on work supported by Rural Veterans Supported Employment TeleRehabilitation Initiative (RVSETI), funded by the VA Office of Rural Health (Project # N08-FY14Q3-S2-P01222) and by support of the VA Health Services Research and Development Service. This work was presented in part at the 114th Annual Meeting of the American Anthropological Association at Denver, Colorado, November 21, 2015; a field-based Health Services Research and Development Service meeting, US Department of Veterans Affairs at Washington, DC, September 12, 2016; and the 2016 Annual Conference of the American Congress for Rehabilitation Medicine at Chicago, Illinois, October-November 2016.
1. Abraham KM, Yosef M, Resnick SG, Zivin K. Competitive employment outcomes among veterans in VHA therapeutic and supported employment services programs. Psychiatr Serv. 2017;68(9):938-946. doi:10.1176/appi.ps.201600412
2. Davis LL, Kyriakides TC, Suris AM, et al. Effect of evidence-based supported employment vs transitional work on achieving steady work among veterans with posttraumatic stress disorder: a randomized clinical trial. JAMA Psychiatry. 2018;75(4):316. doi:10.1001/jamapsychiatry.2017.4472
3. Ottomanelli L, Goetz LL, Suris A, et al. Effectiveness of supported employment for veterans with spinal cord injuries: results from a randomized multisite study. Arch Phys Med Rehabil. 2012;93(5):740-747. doi:10.1016/j.apmr.2012.01.002
4. Ottomanelli L, Goetz LL, Barnett SD, et al. Individual placement and support in spinal cord injury: a longitudinal observational study of employment outcomes. Arch Phys Med Rehabil. 2017;98(8):1567-1575. doi:10.1016/j.apmr.2016.12.010
5. Cotner BA, Ottomanelli L, O’Connor DR, Njoh EN, Barnett SD, Miech EJ. Quality of life outcomes for veterans with spinal cord injury receiving individual placement and support (IPS). Top Spinal Cord Inj Rehabil. 2018;24(4):325-335. doi:10.1310/sci17-00046
6. Metzel DS, Giordano A. Locations of employment services and people with disabilities: a geographical analysis of accessibility. J Disabil Policy Stud. 2007;18(2):88-97. doi:10.1177/10442073070180020501
7. Landon T, Connor A, McKnight-Lizotte M, Peña J. Rehabilitation counseling in rural settings: a phenomenological study on barriers and supports. J Rehabil. 2019;85(2):47-57.
8. Riemer-Reiss M. Vocational rehabilitation counseling at a distance: Challenges, strategies and ethics to consider. J Rehabil. 2000;66(1):11-17.
9. Schmeler MR, Schein RM, McCue M, Betz K. Telerehabilitation clinical and vocational applications for assistive technology: research, opportunities, and challenges. Int J Telerehabilitation. 2009;1(1):59-72.
10. Levy CE, Silverman E, Jia H, Geiss M, Omura D. Effects of physical therapy delivery via home video telerehabilitation on functional and health-related quality of life outcomes. J Rehabil Res Dev. 2015;52(3):361-370. doi:10.1682/JRRD.2014.10.0239
11. McMullen CK, Ash JS, Sittig DF, et al. Rapid assessment of clinical information systems in the healthcare setting: an efficient method for time-pressed evaluation. Methods Inf Med. 2011;50(4):299-307. doi:10.3414/ME10-01-0042
12. Averill JB. Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qual Health Res. 2002;12(6):855-866.
13. Egede LE, Acierno R, Knapp RG, et al. Psychotherapy for depression in older veterans via telemedicine: a randomised, open-label, non-inferiority trial. Lancet Psychiatry. 2015;2(8):693-701. doi:10.1016/S2215-0366(15)00122-4
14. Fortney JC, Pyne JM, Edlund MJ, et al. A randomized trial of telemedicine-based collaborative care for depression. J Gen Intern Med. 2007;22(8):1086-1093. doi:10.1007/s11606-007-0201-9
15. Fortney JC, Pyne JM, Kimbrell TA, et al. Telemedicine-based collaborative care for posttraumatic stress disorder: a randomized clinical trial. JAMA Psychiatry. 2015;72(1):58. doi:10.1001/jamapsychiatry.2014.1575
16. Grubbs KM, Fortney JC, Dean T, Williams JS, Godleski L. A comparison of mental health diagnoses treated via interactive video and face to face in the Veterans Healthcare Administration. Telemed E-Health. 2015;21(7):564-566. doi:10.1089/tmj.2014.0152
17. Agostini M, Moja L, Banzi R, et al. Telerehabilitation and recovery of motor function: a systematic review and meta-analysis. J Telemed Telecare. 2015;21(4):202-213. doi:10.1177/1357633X15572201
18. Bergquist TF, Thompson K, Gehl C, Munoz Pineda J. Satisfaction ratings after receiving internet-based cognitive rehabilitation in persons with memory impairments after severe acquired brain injury. Telemed E-Health. 2010;16(4):417-423. doi:10.1089/tmj.2009.0118
19. Brennan DM, Georgeadis AC, Baron CR, Barker LM. The effect of videoconference-based telerehabilitation on story retelling performance by brain-injured subjects and its implications for remote speech-language therapy. Telemed J E Health. 2004;10(2):147-154. doi:10.1089/tmj.2004.10.147
20. Dallolio L, Menarini M, China S, et al. Functional and clinical outcomes of telemedicine in patients with spinal cord injury. Arch Phys Med Rehabil. 2008;89(12):2332-2341. doi:10.1016/j.apmr.2008.06.012
21. Houlihan BV, Jette A, Friedman RH, et al. A pilot study of a telehealth intervention for persons with spinal cord dysfunction. Spinal Cord. 2013;51(9):715-720.doi:10.1038/sc.2013.45
22. Smith MW, Hill ML, Hopkins KL, Kiratli BJ, Cronkite RC. A modeled analysis of telehealth methods for treating pressure ulcers after spinal cord injury. Int J Telemed Appl. 2012;2012:1-10. doi:10.1155/2012/729492
23. Balcazar FE, Keys CB, Davis M, Lardon C, Jones C. Strengths and challenges of intervention research in vocational rehabilitation: an illustration of agency-university collaboration. J Rehabil. 2005;71(2):40-48.
24. Martinez RN, Hogan TP, Balbale S, et al. Sociotechnical perspective on implementing clinical video telehealth for veterans with spinal cord injuries and disorders. Telemed J E Health. 2017;23(7):567-576. doi:10.1089/tmj.2016.0200
25. Martinez RN, Hogan TP, Lones K, et al. Evaluation and treatment of mild traumatic brain injury through the implementation of clinical video telehealth: provider perspectives from the Veterans Health Administration. PM R. 2017;9(3):231-240. doi:10.1016/j.pmrj.2016.07.002
26. Smith AC, Thomas E, Snoswell CL, et al. Telehealth for global emergencies: implications for coronavirus disease 2019 (COVID-19). J Telemed Telecare. 2020;26(5):309-313. doi:10.1177/1357633X20916567
27. Cowper-Ripley DC, Jia H, Wang X, et al. Trends in VA telerehabilitation patients and encounters over time and by rurality. Fed Pract. 2019; 36(3):122-128.
28. US Department of Veterans Affairs. Veterans VA Video Connect. Published May 22, 2020. Accessed May 29, 2020. https://mobile.va.gov/app/va-video-connect#AppDescription.
29. US Department of Veterans Affairs. VA telehealth at home. Accessed May 29, 2020. https://telehealth.va.gov/type/home
Vocational rehabilitation (VR) interventions are offered through Compensated Work Therapy (CWT) as part of clinical care in the Veterans Health Administration (VHA) to improve employment and quality of life outcomes for veterans with life-altering disabilities.1–5 CWT vocational services range from assessment, vocational counseling, and treatment plan development to job placement, coaching, and follow-along support.1 However, many veterans receive care in community-based clinics that are not staffed with a VR specialist (VRS) to provide these services.6–8 Telehealth may increase patient access to VR, especially for rural veterans and those with travel barriers, but it is not known whether veterans and VRS would find this to be a satisfactory service delivery method.8,9 This paper examines veteran and VRS provider perspectives on VR provided by telehealth (VRtele) as part of a VHA clinical demonstration project. To our knowledge, this is the first report of using real-time, clinic-based VRtele.
Methods
The Rural Veterans Supported Employment Telerehabilitation Initiative (RVSETI) was conducted as a field-initiated demonstration project at 2 US Department of Veterans Affairs (VA) medical centers (VAMCs) in Florida between 2014 and 2016: James A. Haley Veterans’ Hospital & Clinics (Tampa) and Malcom Randall VAMC (Gainesville). This retrospective evaluation of its first year did not require institutional review board approval as it was determined to be a quality improvement project by the local research service.
The patient population for the project was veterans with disabilities who were referred by clinical consults to the CWT service, a recovery-oriented vocational program. During the project years, veterans were offered the option of receiving VR services, such as supported employment, community-based employment services, or vocational assistance, through VRtele rather than traditional face-to-face meetings. The specific interventions delivered included patient orientation, interview assessment, treatment plan development, referral activities, vocational counseling, assessment of workplace for accommodation needs, vocational case management, and other employment supports. VR staff participating in the project included 2 VR supervisors, 1 supported employment mentor trainer, and 5 VRSs.
Each clinic was set up for VRtele, and codes were added to the electronic health record (EHR) to ensure proper documentation. Participating VRSs completed teleconferencing training, including a skills assessment using the equipment for real-time, high-quality video streaming over an encrypted network to provide services in a patient’s home or other remote locations. VRS staff provided veterans with instructions on using a VA-provided tablet or their own device and assisted them with establishing connectivity with the network. Video equipment included speakers, camera, and headphones connected to the desktop computer or laptop of the VRS. A patient’s first VRtele
Demographic data, primary diagnosis, VR usage data, and zip codes of participating veterans were extracted from the EHR. Veterans completed a 2-part satisfaction survey administered 90 days after enrollment and at discharge. Part 1 was composed of 15 items, most with a 5-point Likert scale (higher ratings indicated greater satisfaction), on various aspects of the VRtele experience, such as audio and video quality and wait times.10 Part 2 addressed VR services and the VRS and consisted of 8 Likert scale items with the option to add a comment for each and 2 open-ended items that asked the participant to list what they liked best and least about VRtele.
Semistructured, in-person 30- to 60-minute interviews were conducted with VRSs at the initiation of VRtele
After ≥ 2 months of VRtele use
Analyses
Descriptive statistics were used for EHR data and satisfaction surveys. For qualitative analysis, each transcript was read in full by 2 researchers to get an overview of the data, and a rapid analysis approach was used to identify central themes focused on how technology was used and the experiences of the participants.11,12 Relevant text for each topic was tabulated, and a summary table was created that highlighted overlapping ideas discussed by the interviewees as well as differences.
Results
Of the 22 veterans who participated in the project, 11 completed satisfaction surveys and 4 participated in qualitative interviews. The rural and nonrural groups did not differ demographically or by diagnosis, which was predominantly mental health related. Only 1 veteran in each group owned a tablet; the majority of both groups required VA-issued devices: 80% (n = 8) rural and 91.7% (n = 11) nonrural. The number of VRtele sessions for the groups also was similar, 53 for rural and 60 for nonrural, as was the mean (SD) number of sessions per veteran: 5.3 (SD, 3.2) rural and 5.0 (SD, 2.5) urban. Overall, 63 miles per session were saved, mostly for rural veterans, and the number of mean (SD) miles saved per veteran was greater for rural than nonrural veterans: 379.2 (243.0) and 256.1 (275.9), respectively. One veteran who moved to a different state during the program continued VRtele at the new location. In a qualitative sampling of 5 VRtele sessions, all the VRSs used office desktop computers.
Level of satisfaction with aspects of VRtele related to the technology rated was consistently > 4 on the Likert scale. The lowest mean (SD) ratings were 4.2 (1.0) for audio quality and 4.4 (0.5) for video quality, and the highest rating was given for equipment operation explanation and privacy was respected, 4.9 (0.3) for both. All questions related to satisfaction with services were also rated high: The mean (SD) lowest ratings were 4.3 (1.0) given to both vocational needs 4.3 (1.0) and tasks effectively helped achieve goals 4.3 (0.7). The highest mean (SD) ratings were 4.6 (0.5) given to VR program service explained and 4.7 (0.5) for appointment timeliness.
Qualitative Results
At first, some VRSs thought the teleconferencing system might be difficult or awkward to use, but they found it easier to set up than expected and seamless to use. VRS staff reported being surprised at how well it worked despite some issues that occurred with loading the software. Once loaded, however, the connection worked well, one VRS noting that following step-by-step instructions solved the problem. Some VRSs indicated they did not invite all the veterans on their caseload to participate in VRtele due to concerns with the patient’s familiarity with technology, but one VRS stated, “I haven’t had anybody that failed to do a [session] that I couldn’t get them up and running within a few minutes.”
When working in the community, VRSs reported using laptops for VRtele but found that these devices were unreliable due to lack of internet access and were slow to start; several VRSs thought tablets would have been more helpful. Some veterans reported technical glitches, lack of comfort with technology, or a problem with sound due to a tablet’s protective case blocking the speakers. To solve the sound issue, a veteran used headphones. This veteran also explained that the log-on process required a new password every time, so he would keep a pen and paper ready to write it down. Because signing in and setting up takes a little time, this veteran and his VRS agreed to start connecting 5 minutes before their meeting time to allow for that set- up time.
Initially, some VRSs expressed concern that transitioning to VRtele would affect the quality of interactions with the veterans. However, VRSs also identified strengths of VRtele, including flexibility, saved time, and increased interaction. One VRS discussed a veteran’s adaptation by saying, “I think he feels even more involved in his plan [and] enjoys the increased interaction.” Veterans reported enjoying using tablets and identified the main strength of VRtele as being able to talk face-to-face with the VRS. Echoing the VRSs, veterans reported teleconferencing saved time by avoiding travel and enabled spontaneous meetings. One of the veterans summed up the benefits of using VRtele: “I’d rather just connect. It’s going to take us 40 to 50 minutes [to meet in person] when we can just connect right here and it takes 15 to 20. We don’t have to go through the driving.… So this right here, doing it ahead of time and having the appointment, it’s a lot easier.”
In their interviews, VRSs talked about enjoying VRtele. A VRS explained: “It makes it a lot easier. It makes me feel less guilty. This way [veterans] don’t have to use their gas money, use their time. I know [the veteran] had something else he needed to do today.” Thus, both veterans and VRSs were satisfied with their VRtele experiences.
Discussion
This first report on the perspective of providers and veterans using VRtele suggests that it is a viable option for service delivery and that is highly satisfactory for serving veterans with disabilities, many of whom live in rural areas or have travel barriers. These findings are consistent with data on telerehabilitation for veterans with cognitive, physical, and mental disabilities.13-22 Further, the data support the notion of using VRtele to facilitate long-term VR follow-up for persons with disabilities, as illustrated by successful continuation of vocational services after a veteran moved out of state.23
Similar to other reports, our experience highlighted 2 factors that affect successful VRtele: (1) Troubleshooting technology barriers for both VR providers and clients; and (2) supportive leadership to facilitate implementation
Changes to technology and increased usage of VA Video Connect may indicate that the barriers identified from the earlier process described here have been diminished or eliminated. More evaluation is needed to assess whether system upgrades have increased ease of use and access for veterans with disabilities.
Conclusions
Encouragingly, this clinical demonstration project showed that both providers and clients recognize the benefits of VRtele. Patient satisfaction and decreased travel costs were clear advantages to using VRtele for this small group of veterans who had barriers to care due to travel or disability barriers. As this program evaluation was limited by a small sample, absence of a comparison group, and lack of outcome data (eg, employment rates, hours, wages, retention), future research is needed on implementation and outcomes of VRtele
Acknowledgments
The authors thank Lynn Dirk, MAMC, for substantial editorial assistance. This material was based on work supported by Rural Veterans Supported Employment TeleRehabilitation Initiative (RVSETI), funded by the VA Office of Rural Health (Project # N08-FY14Q3-S2-P01222) and by support of the VA Health Services Research and Development Service. This work was presented in part at the 114th Annual Meeting of the American Anthropological Association at Denver, Colorado, November 21, 2015; a field-based Health Services Research and Development Service meeting, US Department of Veterans Affairs at Washington, DC, September 12, 2016; and the 2016 Annual Conference of the American Congress for Rehabilitation Medicine at Chicago, Illinois, October-November 2016.
Vocational rehabilitation (VR) interventions are offered through Compensated Work Therapy (CWT) as part of clinical care in the Veterans Health Administration (VHA) to improve employment and quality of life outcomes for veterans with life-altering disabilities.1–5 CWT vocational services range from assessment, vocational counseling, and treatment plan development to job placement, coaching, and follow-along support.1 However, many veterans receive care in community-based clinics that are not staffed with a VR specialist (VRS) to provide these services.6–8 Telehealth may increase patient access to VR, especially for rural veterans and those with travel barriers, but it is not known whether veterans and VRS would find this to be a satisfactory service delivery method.8,9 This paper examines veteran and VRS provider perspectives on VR provided by telehealth (VRtele) as part of a VHA clinical demonstration project. To our knowledge, this is the first report of using real-time, clinic-based VRtele.
Methods
The Rural Veterans Supported Employment Telerehabilitation Initiative (RVSETI) was conducted as a field-initiated demonstration project at 2 US Department of Veterans Affairs (VA) medical centers (VAMCs) in Florida between 2014 and 2016: James A. Haley Veterans’ Hospital & Clinics (Tampa) and Malcom Randall VAMC (Gainesville). This retrospective evaluation of its first year did not require institutional review board approval as it was determined to be a quality improvement project by the local research service.
The patient population for the project was veterans with disabilities who were referred by clinical consults to the CWT service, a recovery-oriented vocational program. During the project years, veterans were offered the option of receiving VR services, such as supported employment, community-based employment services, or vocational assistance, through VRtele rather than traditional face-to-face meetings. The specific interventions delivered included patient orientation, interview assessment, treatment plan development, referral activities, vocational counseling, assessment of workplace for accommodation needs, vocational case management, and other employment supports. VR staff participating in the project included 2 VR supervisors, 1 supported employment mentor trainer, and 5 VRSs.
Each clinic was set up for VRtele, and codes were added to the electronic health record (EHR) to ensure proper documentation. Participating VRSs completed teleconferencing training, including a skills assessment using the equipment for real-time, high-quality video streaming over an encrypted network to provide services in a patient’s home or other remote locations. VRS staff provided veterans with instructions on using a VA-provided tablet or their own device and assisted them with establishing connectivity with the network. Video equipment included speakers, camera, and headphones connected to the desktop computer or laptop of the VRS. A patient’s first VRtele
Demographic data, primary diagnosis, VR usage data, and zip codes of participating veterans were extracted from the EHR. Veterans completed a 2-part satisfaction survey administered 90 days after enrollment and at discharge. Part 1 was composed of 15 items, most with a 5-point Likert scale (higher ratings indicated greater satisfaction), on various aspects of the VRtele experience, such as audio and video quality and wait times.10 Part 2 addressed VR services and the VRS and consisted of 8 Likert scale items with the option to add a comment for each and 2 open-ended items that asked the participant to list what they liked best and least about VRtele.
Semistructured, in-person 30- to 60-minute interviews were conducted with VRSs at the initiation of VRtele
After ≥ 2 months of VRtele use
Analyses
Descriptive statistics were used for EHR data and satisfaction surveys. For qualitative analysis, each transcript was read in full by 2 researchers to get an overview of the data, and a rapid analysis approach was used to identify central themes focused on how technology was used and the experiences of the participants.11,12 Relevant text for each topic was tabulated, and a summary table was created that highlighted overlapping ideas discussed by the interviewees as well as differences.
Results
Of the 22 veterans who participated in the project, 11 completed satisfaction surveys and 4 participated in qualitative interviews. The rural and nonrural groups did not differ demographically or by diagnosis, which was predominantly mental health related. Only 1 veteran in each group owned a tablet; the majority of both groups required VA-issued devices: 80% (n = 8) rural and 91.7% (n = 11) nonrural. The number of VRtele sessions for the groups also was similar, 53 for rural and 60 for nonrural, as was the mean (SD) number of sessions per veteran: 5.3 (SD, 3.2) rural and 5.0 (SD, 2.5) urban. Overall, 63 miles per session were saved, mostly for rural veterans, and the number of mean (SD) miles saved per veteran was greater for rural than nonrural veterans: 379.2 (243.0) and 256.1 (275.9), respectively. One veteran who moved to a different state during the program continued VRtele at the new location. In a qualitative sampling of 5 VRtele sessions, all the VRSs used office desktop computers.
Level of satisfaction with aspects of VRtele related to the technology rated was consistently > 4 on the Likert scale. The lowest mean (SD) ratings were 4.2 (1.0) for audio quality and 4.4 (0.5) for video quality, and the highest rating was given for equipment operation explanation and privacy was respected, 4.9 (0.3) for both. All questions related to satisfaction with services were also rated high: The mean (SD) lowest ratings were 4.3 (1.0) given to both vocational needs 4.3 (1.0) and tasks effectively helped achieve goals 4.3 (0.7). The highest mean (SD) ratings were 4.6 (0.5) given to VR program service explained and 4.7 (0.5) for appointment timeliness.
Qualitative Results
At first, some VRSs thought the teleconferencing system might be difficult or awkward to use, but they found it easier to set up than expected and seamless to use. VRS staff reported being surprised at how well it worked despite some issues that occurred with loading the software. Once loaded, however, the connection worked well, one VRS noting that following step-by-step instructions solved the problem. Some VRSs indicated they did not invite all the veterans on their caseload to participate in VRtele due to concerns with the patient’s familiarity with technology, but one VRS stated, “I haven’t had anybody that failed to do a [session] that I couldn’t get them up and running within a few minutes.”
When working in the community, VRSs reported using laptops for VRtele but found that these devices were unreliable due to lack of internet access and were slow to start; several VRSs thought tablets would have been more helpful. Some veterans reported technical glitches, lack of comfort with technology, or a problem with sound due to a tablet’s protective case blocking the speakers. To solve the sound issue, a veteran used headphones. This veteran also explained that the log-on process required a new password every time, so he would keep a pen and paper ready to write it down. Because signing in and setting up takes a little time, this veteran and his VRS agreed to start connecting 5 minutes before their meeting time to allow for that set- up time.
Initially, some VRSs expressed concern that transitioning to VRtele would affect the quality of interactions with the veterans. However, VRSs also identified strengths of VRtele, including flexibility, saved time, and increased interaction. One VRS discussed a veteran’s adaptation by saying, “I think he feels even more involved in his plan [and] enjoys the increased interaction.” Veterans reported enjoying using tablets and identified the main strength of VRtele as being able to talk face-to-face with the VRS. Echoing the VRSs, veterans reported teleconferencing saved time by avoiding travel and enabled spontaneous meetings. One of the veterans summed up the benefits of using VRtele: “I’d rather just connect. It’s going to take us 40 to 50 minutes [to meet in person] when we can just connect right here and it takes 15 to 20. We don’t have to go through the driving.… So this right here, doing it ahead of time and having the appointment, it’s a lot easier.”
In their interviews, VRSs talked about enjoying VRtele. A VRS explained: “It makes it a lot easier. It makes me feel less guilty. This way [veterans] don’t have to use their gas money, use their time. I know [the veteran] had something else he needed to do today.” Thus, both veterans and VRSs were satisfied with their VRtele experiences.
Discussion
This first report on the perspective of providers and veterans using VRtele suggests that it is a viable option for service delivery and that is highly satisfactory for serving veterans with disabilities, many of whom live in rural areas or have travel barriers. These findings are consistent with data on telerehabilitation for veterans with cognitive, physical, and mental disabilities.13-22 Further, the data support the notion of using VRtele to facilitate long-term VR follow-up for persons with disabilities, as illustrated by successful continuation of vocational services after a veteran moved out of state.23
Similar to other reports, our experience highlighted 2 factors that affect successful VRtele: (1) Troubleshooting technology barriers for both VR providers and clients; and (2) supportive leadership to facilitate implementation
Changes to technology and increased usage of VA Video Connect may indicate that the barriers identified from the earlier process described here have been diminished or eliminated. More evaluation is needed to assess whether system upgrades have increased ease of use and access for veterans with disabilities.
Conclusions
Encouragingly, this clinical demonstration project showed that both providers and clients recognize the benefits of VRtele. Patient satisfaction and decreased travel costs were clear advantages to using VRtele for this small group of veterans who had barriers to care due to travel or disability barriers. As this program evaluation was limited by a small sample, absence of a comparison group, and lack of outcome data (eg, employment rates, hours, wages, retention), future research is needed on implementation and outcomes of VRtele
Acknowledgments
The authors thank Lynn Dirk, MAMC, for substantial editorial assistance. This material was based on work supported by Rural Veterans Supported Employment TeleRehabilitation Initiative (RVSETI), funded by the VA Office of Rural Health (Project # N08-FY14Q3-S2-P01222) and by support of the VA Health Services Research and Development Service. This work was presented in part at the 114th Annual Meeting of the American Anthropological Association at Denver, Colorado, November 21, 2015; a field-based Health Services Research and Development Service meeting, US Department of Veterans Affairs at Washington, DC, September 12, 2016; and the 2016 Annual Conference of the American Congress for Rehabilitation Medicine at Chicago, Illinois, October-November 2016.
1. Abraham KM, Yosef M, Resnick SG, Zivin K. Competitive employment outcomes among veterans in VHA therapeutic and supported employment services programs. Psychiatr Serv. 2017;68(9):938-946. doi:10.1176/appi.ps.201600412
2. Davis LL, Kyriakides TC, Suris AM, et al. Effect of evidence-based supported employment vs transitional work on achieving steady work among veterans with posttraumatic stress disorder: a randomized clinical trial. JAMA Psychiatry. 2018;75(4):316. doi:10.1001/jamapsychiatry.2017.4472
3. Ottomanelli L, Goetz LL, Suris A, et al. Effectiveness of supported employment for veterans with spinal cord injuries: results from a randomized multisite study. Arch Phys Med Rehabil. 2012;93(5):740-747. doi:10.1016/j.apmr.2012.01.002
4. Ottomanelli L, Goetz LL, Barnett SD, et al. Individual placement and support in spinal cord injury: a longitudinal observational study of employment outcomes. Arch Phys Med Rehabil. 2017;98(8):1567-1575. doi:10.1016/j.apmr.2016.12.010
5. Cotner BA, Ottomanelli L, O’Connor DR, Njoh EN, Barnett SD, Miech EJ. Quality of life outcomes for veterans with spinal cord injury receiving individual placement and support (IPS). Top Spinal Cord Inj Rehabil. 2018;24(4):325-335. doi:10.1310/sci17-00046
6. Metzel DS, Giordano A. Locations of employment services and people with disabilities: a geographical analysis of accessibility. J Disabil Policy Stud. 2007;18(2):88-97. doi:10.1177/10442073070180020501
7. Landon T, Connor A, McKnight-Lizotte M, Peña J. Rehabilitation counseling in rural settings: a phenomenological study on barriers and supports. J Rehabil. 2019;85(2):47-57.
8. Riemer-Reiss M. Vocational rehabilitation counseling at a distance: Challenges, strategies and ethics to consider. J Rehabil. 2000;66(1):11-17.
9. Schmeler MR, Schein RM, McCue M, Betz K. Telerehabilitation clinical and vocational applications for assistive technology: research, opportunities, and challenges. Int J Telerehabilitation. 2009;1(1):59-72.
10. Levy CE, Silverman E, Jia H, Geiss M, Omura D. Effects of physical therapy delivery via home video telerehabilitation on functional and health-related quality of life outcomes. J Rehabil Res Dev. 2015;52(3):361-370. doi:10.1682/JRRD.2014.10.0239
11. McMullen CK, Ash JS, Sittig DF, et al. Rapid assessment of clinical information systems in the healthcare setting: an efficient method for time-pressed evaluation. Methods Inf Med. 2011;50(4):299-307. doi:10.3414/ME10-01-0042
12. Averill JB. Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qual Health Res. 2002;12(6):855-866.
13. Egede LE, Acierno R, Knapp RG, et al. Psychotherapy for depression in older veterans via telemedicine: a randomised, open-label, non-inferiority trial. Lancet Psychiatry. 2015;2(8):693-701. doi:10.1016/S2215-0366(15)00122-4
14. Fortney JC, Pyne JM, Edlund MJ, et al. A randomized trial of telemedicine-based collaborative care for depression. J Gen Intern Med. 2007;22(8):1086-1093. doi:10.1007/s11606-007-0201-9
15. Fortney JC, Pyne JM, Kimbrell TA, et al. Telemedicine-based collaborative care for posttraumatic stress disorder: a randomized clinical trial. JAMA Psychiatry. 2015;72(1):58. doi:10.1001/jamapsychiatry.2014.1575
16. Grubbs KM, Fortney JC, Dean T, Williams JS, Godleski L. A comparison of mental health diagnoses treated via interactive video and face to face in the Veterans Healthcare Administration. Telemed E-Health. 2015;21(7):564-566. doi:10.1089/tmj.2014.0152
17. Agostini M, Moja L, Banzi R, et al. Telerehabilitation and recovery of motor function: a systematic review and meta-analysis. J Telemed Telecare. 2015;21(4):202-213. doi:10.1177/1357633X15572201
18. Bergquist TF, Thompson K, Gehl C, Munoz Pineda J. Satisfaction ratings after receiving internet-based cognitive rehabilitation in persons with memory impairments after severe acquired brain injury. Telemed E-Health. 2010;16(4):417-423. doi:10.1089/tmj.2009.0118
19. Brennan DM, Georgeadis AC, Baron CR, Barker LM. The effect of videoconference-based telerehabilitation on story retelling performance by brain-injured subjects and its implications for remote speech-language therapy. Telemed J E Health. 2004;10(2):147-154. doi:10.1089/tmj.2004.10.147
20. Dallolio L, Menarini M, China S, et al. Functional and clinical outcomes of telemedicine in patients with spinal cord injury. Arch Phys Med Rehabil. 2008;89(12):2332-2341. doi:10.1016/j.apmr.2008.06.012
21. Houlihan BV, Jette A, Friedman RH, et al. A pilot study of a telehealth intervention for persons with spinal cord dysfunction. Spinal Cord. 2013;51(9):715-720.doi:10.1038/sc.2013.45
22. Smith MW, Hill ML, Hopkins KL, Kiratli BJ, Cronkite RC. A modeled analysis of telehealth methods for treating pressure ulcers after spinal cord injury. Int J Telemed Appl. 2012;2012:1-10. doi:10.1155/2012/729492
23. Balcazar FE, Keys CB, Davis M, Lardon C, Jones C. Strengths and challenges of intervention research in vocational rehabilitation: an illustration of agency-university collaboration. J Rehabil. 2005;71(2):40-48.
24. Martinez RN, Hogan TP, Balbale S, et al. Sociotechnical perspective on implementing clinical video telehealth for veterans with spinal cord injuries and disorders. Telemed J E Health. 2017;23(7):567-576. doi:10.1089/tmj.2016.0200
25. Martinez RN, Hogan TP, Lones K, et al. Evaluation and treatment of mild traumatic brain injury through the implementation of clinical video telehealth: provider perspectives from the Veterans Health Administration. PM R. 2017;9(3):231-240. doi:10.1016/j.pmrj.2016.07.002
26. Smith AC, Thomas E, Snoswell CL, et al. Telehealth for global emergencies: implications for coronavirus disease 2019 (COVID-19). J Telemed Telecare. 2020;26(5):309-313. doi:10.1177/1357633X20916567
27. Cowper-Ripley DC, Jia H, Wang X, et al. Trends in VA telerehabilitation patients and encounters over time and by rurality. Fed Pract. 2019; 36(3):122-128.
28. US Department of Veterans Affairs. Veterans VA Video Connect. Published May 22, 2020. Accessed May 29, 2020. https://mobile.va.gov/app/va-video-connect#AppDescription.
29. US Department of Veterans Affairs. VA telehealth at home. Accessed May 29, 2020. https://telehealth.va.gov/type/home
1. Abraham KM, Yosef M, Resnick SG, Zivin K. Competitive employment outcomes among veterans in VHA therapeutic and supported employment services programs. Psychiatr Serv. 2017;68(9):938-946. doi:10.1176/appi.ps.201600412
2. Davis LL, Kyriakides TC, Suris AM, et al. Effect of evidence-based supported employment vs transitional work on achieving steady work among veterans with posttraumatic stress disorder: a randomized clinical trial. JAMA Psychiatry. 2018;75(4):316. doi:10.1001/jamapsychiatry.2017.4472
3. Ottomanelli L, Goetz LL, Suris A, et al. Effectiveness of supported employment for veterans with spinal cord injuries: results from a randomized multisite study. Arch Phys Med Rehabil. 2012;93(5):740-747. doi:10.1016/j.apmr.2012.01.002
4. Ottomanelli L, Goetz LL, Barnett SD, et al. Individual placement and support in spinal cord injury: a longitudinal observational study of employment outcomes. Arch Phys Med Rehabil. 2017;98(8):1567-1575. doi:10.1016/j.apmr.2016.12.010
5. Cotner BA, Ottomanelli L, O’Connor DR, Njoh EN, Barnett SD, Miech EJ. Quality of life outcomes for veterans with spinal cord injury receiving individual placement and support (IPS). Top Spinal Cord Inj Rehabil. 2018;24(4):325-335. doi:10.1310/sci17-00046
6. Metzel DS, Giordano A. Locations of employment services and people with disabilities: a geographical analysis of accessibility. J Disabil Policy Stud. 2007;18(2):88-97. doi:10.1177/10442073070180020501
7. Landon T, Connor A, McKnight-Lizotte M, Peña J. Rehabilitation counseling in rural settings: a phenomenological study on barriers and supports. J Rehabil. 2019;85(2):47-57.
8. Riemer-Reiss M. Vocational rehabilitation counseling at a distance: Challenges, strategies and ethics to consider. J Rehabil. 2000;66(1):11-17.
9. Schmeler MR, Schein RM, McCue M, Betz K. Telerehabilitation clinical and vocational applications for assistive technology: research, opportunities, and challenges. Int J Telerehabilitation. 2009;1(1):59-72.
10. Levy CE, Silverman E, Jia H, Geiss M, Omura D. Effects of physical therapy delivery via home video telerehabilitation on functional and health-related quality of life outcomes. J Rehabil Res Dev. 2015;52(3):361-370. doi:10.1682/JRRD.2014.10.0239
11. McMullen CK, Ash JS, Sittig DF, et al. Rapid assessment of clinical information systems in the healthcare setting: an efficient method for time-pressed evaluation. Methods Inf Med. 2011;50(4):299-307. doi:10.3414/ME10-01-0042
12. Averill JB. Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qual Health Res. 2002;12(6):855-866.
13. Egede LE, Acierno R, Knapp RG, et al. Psychotherapy for depression in older veterans via telemedicine: a randomised, open-label, non-inferiority trial. Lancet Psychiatry. 2015;2(8):693-701. doi:10.1016/S2215-0366(15)00122-4
14. Fortney JC, Pyne JM, Edlund MJ, et al. A randomized trial of telemedicine-based collaborative care for depression. J Gen Intern Med. 2007;22(8):1086-1093. doi:10.1007/s11606-007-0201-9
15. Fortney JC, Pyne JM, Kimbrell TA, et al. Telemedicine-based collaborative care for posttraumatic stress disorder: a randomized clinical trial. JAMA Psychiatry. 2015;72(1):58. doi:10.1001/jamapsychiatry.2014.1575
16. Grubbs KM, Fortney JC, Dean T, Williams JS, Godleski L. A comparison of mental health diagnoses treated via interactive video and face to face in the Veterans Healthcare Administration. Telemed E-Health. 2015;21(7):564-566. doi:10.1089/tmj.2014.0152
17. Agostini M, Moja L, Banzi R, et al. Telerehabilitation and recovery of motor function: a systematic review and meta-analysis. J Telemed Telecare. 2015;21(4):202-213. doi:10.1177/1357633X15572201
18. Bergquist TF, Thompson K, Gehl C, Munoz Pineda J. Satisfaction ratings after receiving internet-based cognitive rehabilitation in persons with memory impairments after severe acquired brain injury. Telemed E-Health. 2010;16(4):417-423. doi:10.1089/tmj.2009.0118
19. Brennan DM, Georgeadis AC, Baron CR, Barker LM. The effect of videoconference-based telerehabilitation on story retelling performance by brain-injured subjects and its implications for remote speech-language therapy. Telemed J E Health. 2004;10(2):147-154. doi:10.1089/tmj.2004.10.147
20. Dallolio L, Menarini M, China S, et al. Functional and clinical outcomes of telemedicine in patients with spinal cord injury. Arch Phys Med Rehabil. 2008;89(12):2332-2341. doi:10.1016/j.apmr.2008.06.012
21. Houlihan BV, Jette A, Friedman RH, et al. A pilot study of a telehealth intervention for persons with spinal cord dysfunction. Spinal Cord. 2013;51(9):715-720.doi:10.1038/sc.2013.45
22. Smith MW, Hill ML, Hopkins KL, Kiratli BJ, Cronkite RC. A modeled analysis of telehealth methods for treating pressure ulcers after spinal cord injury. Int J Telemed Appl. 2012;2012:1-10. doi:10.1155/2012/729492
23. Balcazar FE, Keys CB, Davis M, Lardon C, Jones C. Strengths and challenges of intervention research in vocational rehabilitation: an illustration of agency-university collaboration. J Rehabil. 2005;71(2):40-48.
24. Martinez RN, Hogan TP, Balbale S, et al. Sociotechnical perspective on implementing clinical video telehealth for veterans with spinal cord injuries and disorders. Telemed J E Health. 2017;23(7):567-576. doi:10.1089/tmj.2016.0200
25. Martinez RN, Hogan TP, Lones K, et al. Evaluation and treatment of mild traumatic brain injury through the implementation of clinical video telehealth: provider perspectives from the Veterans Health Administration. PM R. 2017;9(3):231-240. doi:10.1016/j.pmrj.2016.07.002
26. Smith AC, Thomas E, Snoswell CL, et al. Telehealth for global emergencies: implications for coronavirus disease 2019 (COVID-19). J Telemed Telecare. 2020;26(5):309-313. doi:10.1177/1357633X20916567
27. Cowper-Ripley DC, Jia H, Wang X, et al. Trends in VA telerehabilitation patients and encounters over time and by rurality. Fed Pract. 2019; 36(3):122-128.
28. US Department of Veterans Affairs. Veterans VA Video Connect. Published May 22, 2020. Accessed May 29, 2020. https://mobile.va.gov/app/va-video-connect#AppDescription.
29. US Department of Veterans Affairs. VA telehealth at home. Accessed May 29, 2020. https://telehealth.va.gov/type/home