User login
Evaluating the Impact of a Simulated Hypersensitivity Reaction Case Study for New Fellows and Chemotherapy Nurses in an Outpatient Infusion Clinic
Background
All chemotherapeutic agents have potential to cause infusion reactions. Our primary objective was to develop a project to assist in appropriate training of nursing staff and incoming fellows for clinic efficiency and patient safety.
Methods
A multi-disciplinary team, including physicians, nurses, and a pharmacist met and following a pre-assessment, a pareto chart was created to determine where to focus our efforts. The results revealed the following areas of concern from most important to least important: utilization of an infusion reaction “kit,” team discussion with staff, infusion reaction simulation, a competency checklist for reactions and “other.” Other responses included: reaction orders in the chart, hands on scenarios, and continued reinforcements. The team resolved to conduct an infusion reaction simulation program to provide an environment to meet many needs of the team, new and experienced. Set in the outpatient infusion center, the program included: a patient/actor, a facilitator, infusion nursing staff, and physicians/fellows. Physicians were invited to participate in the training, but infusion staff were unaware of the program to provide another real life aspect to the simulation; however, both were blinded to the scenario. The pharmacist facilitated the event where the patient actor proceeded to start with a minor infusion reaction that progressed to full anaphylaxis.
Results
Using a Likert scale, a post simulation assessment included 6 questions: 90% of participants felt strongly the exercise increased awareness of the infusion reaction e-kit, 80% felt strongly the exercise was meaningful to their practice, 90% strongly agreed or agreed the scenario simulated a real life situation, also 90% strongly agreed or agreed the program helped them think critically. Finally, 100% of participants strongly agreed or agreed they felt confident in their ability to intervene in the event of a hypersensitivity reaction. Our objectives were achieved: identify the signs and symptoms of a hypersensitivity reaction, utilize the proper intervention in the event of a hypersensitivity reaction. Other outcomes include an updated chemotherapy order consult complete with standing reaction orders in the medical record.
Conclusion
Ultimately, our interdisciplinary simulation concluded with increased awareness, improved confidence, and strengthened collaboration, communication and accountability among our infusion staff and oncology providers
Background
All chemotherapeutic agents have potential to cause infusion reactions. Our primary objective was to develop a project to assist in appropriate training of nursing staff and incoming fellows for clinic efficiency and patient safety.
Methods
A multi-disciplinary team, including physicians, nurses, and a pharmacist met and following a pre-assessment, a pareto chart was created to determine where to focus our efforts. The results revealed the following areas of concern from most important to least important: utilization of an infusion reaction “kit,” team discussion with staff, infusion reaction simulation, a competency checklist for reactions and “other.” Other responses included: reaction orders in the chart, hands on scenarios, and continued reinforcements. The team resolved to conduct an infusion reaction simulation program to provide an environment to meet many needs of the team, new and experienced. Set in the outpatient infusion center, the program included: a patient/actor, a facilitator, infusion nursing staff, and physicians/fellows. Physicians were invited to participate in the training, but infusion staff were unaware of the program to provide another real life aspect to the simulation; however, both were blinded to the scenario. The pharmacist facilitated the event where the patient actor proceeded to start with a minor infusion reaction that progressed to full anaphylaxis.
Results
Using a Likert scale, a post simulation assessment included 6 questions: 90% of participants felt strongly the exercise increased awareness of the infusion reaction e-kit, 80% felt strongly the exercise was meaningful to their practice, 90% strongly agreed or agreed the scenario simulated a real life situation, also 90% strongly agreed or agreed the program helped them think critically. Finally, 100% of participants strongly agreed or agreed they felt confident in their ability to intervene in the event of a hypersensitivity reaction. Our objectives were achieved: identify the signs and symptoms of a hypersensitivity reaction, utilize the proper intervention in the event of a hypersensitivity reaction. Other outcomes include an updated chemotherapy order consult complete with standing reaction orders in the medical record.
Conclusion
Ultimately, our interdisciplinary simulation concluded with increased awareness, improved confidence, and strengthened collaboration, communication and accountability among our infusion staff and oncology providers
Background
All chemotherapeutic agents have potential to cause infusion reactions. Our primary objective was to develop a project to assist in appropriate training of nursing staff and incoming fellows for clinic efficiency and patient safety.
Methods
A multi-disciplinary team, including physicians, nurses, and a pharmacist met and following a pre-assessment, a pareto chart was created to determine where to focus our efforts. The results revealed the following areas of concern from most important to least important: utilization of an infusion reaction “kit,” team discussion with staff, infusion reaction simulation, a competency checklist for reactions and “other.” Other responses included: reaction orders in the chart, hands on scenarios, and continued reinforcements. The team resolved to conduct an infusion reaction simulation program to provide an environment to meet many needs of the team, new and experienced. Set in the outpatient infusion center, the program included: a patient/actor, a facilitator, infusion nursing staff, and physicians/fellows. Physicians were invited to participate in the training, but infusion staff were unaware of the program to provide another real life aspect to the simulation; however, both were blinded to the scenario. The pharmacist facilitated the event where the patient actor proceeded to start with a minor infusion reaction that progressed to full anaphylaxis.
Results
Using a Likert scale, a post simulation assessment included 6 questions: 90% of participants felt strongly the exercise increased awareness of the infusion reaction e-kit, 80% felt strongly the exercise was meaningful to their practice, 90% strongly agreed or agreed the scenario simulated a real life situation, also 90% strongly agreed or agreed the program helped them think critically. Finally, 100% of participants strongly agreed or agreed they felt confident in their ability to intervene in the event of a hypersensitivity reaction. Our objectives were achieved: identify the signs and symptoms of a hypersensitivity reaction, utilize the proper intervention in the event of a hypersensitivity reaction. Other outcomes include an updated chemotherapy order consult complete with standing reaction orders in the medical record.
Conclusion
Ultimately, our interdisciplinary simulation concluded with increased awareness, improved confidence, and strengthened collaboration, communication and accountability among our infusion staff and oncology providers
Guideline Concordance with Durvalumab in Unresectable Stage III Non-Small Cell Lung Cancer: A Single Center Veterans Hospital Experience
The US Food and Drug Administration (FDA) approved the use of durvalumab for patients with unresectable stage III non-small cell lung cancer (NSCLC) whose disease has not progressed following concurrent platinum-based chemotherapy and radiation therapy (CRT).1 After 2 randomized phase 3 studies in 2017 and 2018 showed significant progression-free and overall survival respectively,2,3 durvalumab became a category 1 recommendation for the above indication per National Comprehensive Cancer Network (NCCN) guidelines.4 Adherence to guidelines have been shown to improve patient survival across several cancer types.5-7 However, guideline adherence rates have been variable across health institutions. Therefore, further study is warranted to evaluate nonadherent practices with the goal of improving the quality of cancer care delivery.8,9
Stage III NSCLC is associated with poor survival rates.10 Concurrent CRT remains the standard of care in patients with good performance status based on clinical trial populations.4 Lung cancer remains a disease of the elderly, with a median age at diagnosis of 70 years.11 Discrepancies in the treatment of lung cancer in older adults can vary widely due to a lack of evidence surrounding the treatment in those who have comorbidities and poor performance status, widening the gap between clinical trial and real-world populations.11
A recent review by Passaro and colleagues revealed that at least 11 pivotal randomized controlled trials have shown the activity of immune checkpoint inhibitors (ICI) in locally advanced and metastatic lung cancer. However, these studies have mostly excluded patients with a performance status of the Eastern Cooperative Oncology Group (ECOG) level ≥ 2.11
Durvalumab is one of many new therapies to enter clinical practice to demonstrate survival benefit, but its use among veterans with stage III NSCLC in adherence with National Comprehensive Cancer Network (NCCN) guidelines was not robust at the Birmingham Veterans Affairs Medical Center (VAMC) in Alabama. Therefore, we decided to study the level of adherence and to identify barriers to conformity to the category 1 NCCN recommendations.
Methods
The Birmingham VAMC Outpatient Oncology Clinic billing data identified all individuals diagnosed with lung cancer treated between October 2017 and August 2019. Patients who did not have NSCLC that was stage III and unresectable were excluded from our study. Patients who did not receive a majority of their treatment at US Department of Veterans Affairs (VA) facilities were excluded as well. Each patient’s demographic, functional level, and tumor characteristics during the treatment planning phase and follow-up visits were obtained. Two investigators who evaluated health care provider documentation using the VA Computerized Patient Record System (CPRS) conducted chart reviews.
The primary outcomes were the proportion of patients who received concurrent CRT and the proportion who received durvalumab consolidation. Our chart review also categorized reasons for nonreceipt of concurrent CRT and subsequent durvalumab. Documented reasons for guideline discordancy were generated empirically and broadly. We noted if documentation was unclear and included reasons for why a veteran was not a candidate for CRT, the presence of toxicities associated with CRT, and a patient’s refusal for therapy despite medical advice. Descriptive data were analyzed for all clinical or demographic characteristics and outcomes.
This was considered an internal quality improvement initiative. As such, Birmingham VAMC did not require institutional review board approval for the study. The facility is accredited by the American College of Surgeons Commission on Cancer.
Results
A total of 41 veterans with stage III NSCLC were identified to have established care in the Birmingham VAMC Oncology Clinic between October 2017 and August 2019. Of these, 7 received the majority of their treatment from community-based non-VA facilities and 14 were not candidates for CRT and were excluded from this study.
The mean (SD) age of study participants was 70.0 (8.4) years (range, 57 to 92 years). Most of the study veterans (33; 97.1%) were male and 20 (58.8%) were African American (Table). Eighteen (53%) of study participants had clinical stage IIIa NSCLC; 19 (56%) showed a squamous subtype of NSCLC. A majority (53%) of the veterans studied were evaluated to be functionally fit with an ECOG status of 0 to 1, although documentation of ECOG status was lacking in 5 (14.7%) patients in the initial treatment planning visit records. It was unclear if performance status had been reevaluated and changes noted over the course of concurrent CRT.
CRT Patients
The relative distribution of veterans who underwent CRT for stage III NSCLC plus the reasons they did not receive guideline-based treatment with durvalumab is shown in the Figure. Fourteen patients (41%) were inappropriate candidates for CRT; the most common reason for this was their poor performance status upon initial evaluation and 3 patients (8.8%) in the study had extensive disease or were upstaged upon follow-up clinic visit.
Twenty (59%) veterans in the study initiated CRT. However, only 16 (47.1%) completed CRT. Those who dropped out of CRT did so because of toxicities that included various cytopenia, gastrointestinal toxicities due to radiation and/or chemotherapy, or failure to thrive.
Durvalumab Treatment
After initiation of CRT, 9 (26.5%) patients did not go on to receive durvalumab. Three patients (8.8%) suffered toxicities during CRT. One study patient was found to have a severe respiratory infection requiring intensive care unit admission. Another study patient was found to have a new sternal lesion on follow-up positron emission tomography. One declined because of a history of severe antineutrophil cytoplasmic antibodies vasculitis, which made durvalumab use unsafe. Three patients (8.8%) declined treatment with CRT or durvalumab because of personal preference. Documentation was unclear as to why durvalumab was prescribed to one patient who had completed CRT.
Discussion
NCCN guidelines on the use of durvalumab in NSCLC are based on the phase 3 PACIFIC placebo-controlled randomized clinical trial. This trial, which included only patients with documented performance status of ECOG 0 or 1, reported that grade 3 or 4 events occurred in 30.5% of patients randomized to consolidative durvalumab. Treatment was discontinued in 15.4% of patients due to adverse events.3
Our study examined consolidation therapy with durvalumab in patients with unresectable stage III NSCLC with an ECOG performance status of 0 to 1 who had not progressed after 2 or more cycles of definitive concurrent CRT.4 Patients with previous exposure to immunotherapy, a history of immunodeficiency, active infection, unresolved toxicity from CRT, autoimmune disease, and patients who received sequential CRT were excluded.2 Surprisingly, the adherence rate to guidelines was close to 100% with appropriate documentation and justification of CRT initiation and durvalumab use. Five (14.7%) of veterans with unresectable stage III NSCLC did not have clear documentation of ECOG status on initial visit and only 1 veteran who completed CRT did not have clear documentation as to why durvalumab was not provided. Unfortunately, 23 (68.6%) veterans in the study were unable to receive durvalumab, a potentially disease-modifying drug; nearly one-third (10) of veterans were deemed poor candidates for concurrent CRT despite the fact that 52.9% (18) of veterans in the study had a documented ECOG of 0 or 1 on initial evaluation.
Clinical Trials vs Real World
The heterogeneity between anticipated study populations, those who were able to receive durvalumab in the PACIFIC trial, compared with our observed real-world veteran population, likely stems from the lack of information about how comorbidity and fitness can affect the choice of therapeutic intervention in patients with lung cancer.12 In addition, older adults who participated in randomized controlled trials (RCTs) are not representative of the average older adult who presents to medical oncology clinics, making the application of guideline concordant care difficult.13
Similar real-world observations parallel to our analyses have confirmed, complemented and/or refuted findings of RCTs, and have helped impact the treatment of multiple acute and chronic conditions including influenza, cardiovascular disease, and diabetes.14
A component of socioeconomic barriers and access to supportive care played roles in the decisions of certain patients who chose not to undergo concurrent CRT despite medical advice. These 2 obstacles also affected the decision making for some in the study when considering the use of durvalumab (administered by a 60-minute IV infusion every 2 weeks for 1 year) per recommended guidelines.1 These hurdles need further study in the context of their effect on quality of life and the difficulties generated by various social determinants of health.
Limitations
Study limitations included the biased and confounding factors previously described about retrospective and nonrandomized observational studies that are controlled for during RCTs.15 Electronic health record data may have been incorrectly collected resulting in missing or wrong data points that affect the validity of our conclusion. Recall bias with regard to documentation by health care providers describing reasons why CRT or durvalumab were not initiated or the patient’s ability to recall previous treatments and report ECOG status or toxicities also may have impacted our findings. Comorbidities and poor performance status, frequently occurring among veterans, negatively impact cancer treatment decisions and may result in a detection bias. For example, tobacco use, cardiovascular disease, including heart failure, and chronic obstructive pulmonary disease, are notoriously higher in the US veteran population when compared with civilian cohorts.16-18 Also, veterans with poorly controlled depression and posttraumatic stress disorder resulting in functional impairment are a factor.19 Steps were taken to address some of these biases by performing repeat checks of tabulated data and employing 2 independent reviewers to evaluate all relevant clinical documentation, compare results, and reach a consensus.
Conlcusions
This retrospective analysis of adherence to category 1 NCCN guidelines for durvalumab use among patients at the Birmingham VAMC Oncology Clinic reinforced our practice and identified minor deficiencies in documentation that would impact future clinical visits. More importantly, it depicted the massive disparity in treatment candidacy among Birmingham veterans compared with clinical trial populations. Efforts will be made to address factors impacting a veteran’s candidacy for CRT and explore other variables such as socioeconomic barriers to treatment. Multiple complementary tools to assess patients’ frailty, such as the Charlson Comorbidity Index (CCI), are now being used for a variety of disorders including cancers. More robust data and standardization are needed to validate the use of these assessments in predicting response to immune checkpoint inhibitors.
Immune checkpoint inhibitors are currently being evaluated in stage III NSCLC studies and may be implemented as routine practice in the future.12 It is important to distinguish fit from frail veterans with lung cancer for treatment selection. We would like to see the expansion of the eligibility criteria for clinical trials to include patients with a performance status of ECOG 2 in order for results to be truly generalizable to the real-world population. Our hope is that such work will improve not only the quality of lung cancer care, but also the quality of care across multiple tumor types.
1. US Food and Drug Administration. FDA approves durvalumab after chemoradiation for unresectable stage II. Published February 20, 2018. Accessed October 9, 2020. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-durvalumab-after-chemoradiation-unresectable-stage-iii-nsclc
2. Antonia SJ, Villegas A, Daniel D, et al. Durvalumab after chemoradiotherapy in stage III non-small-cell lung cancer. N Engl J Med. 2017;377(20):1919-1929. doi:10.1056/NEJMoa1709937
3. Antonia SJ, Villegas A, Daniel D, et al. Overall survival with durvalumab after chemoradiotherapy in stage III NSCLC. N Engl J Med. 2018;379(24):2342-2350. doi:10.1056/NEJMoa1809697
4. Ettinger DS, Wood DE, Aisner DL et al. NCCN clinical practice guidelines in oncology: non-small cell lung cancer. Version8.2020. Updated September 15, 2020. Accessed October 9, 2020. https://www.nccn.org/professionals/physician_gls/pdf/nscl.pdf
5. Bristow RE, Chang J, Ziogas A, Campos B, Chavez LR, Anton-Culver H. Impact of National Cancer Institute Comprehensive Cancer Centers on ovarian cancer treatment and survival. J Am Coll Surg. 2015;220(5):940-950. doi:10.1016/j.jamcollsurg.2015.01.056
6. Boland GM, Chang GJ, Haynes AB, et al. Association between adherence to National Comprehensive Cancer Network treatment guidelines and improved survival in patients with colon cancer. Cancer. 2013;119(8):1593-1601. doi:10.1002/cncr.27935
7. Schwentner L, Wöckel A, König J, et al. Adherence to treatment guidelines and survival in triple-negative breast cancer: a retrospective multi-center cohort study with 9,156 patients. BMC Cancer. 2013;13:487. Published 2013 Oct 21. doi:10.1186/1471-2407-13-487
8. Jazieh A, Alkaiyat MO, Ali Y, Hashim MA, Abdelhafiz N, Al Olayan A. Improving adherence to lung cancer guidelines: a quality improvement project that uses chart review, audit and feedback approach. BMJ Open Qual. 2019;8(3):e000436. Published 2019 Aug 26. doi:10.1136/bmjoq-2018-000436
9. Shaverdian N, Offin MD, Rimner A, et al. Utilization and factors precluding the initiation of consolidative durvalumab in unresectable stage III non-small cell lung cancer. Radiother Oncol. 2020;144:101-104. doi:10.1016/j.radonc.2019.11.015
10. National Cancer Institute. SEER cancer statistics review, 1975-2015, Table 15.1 cancer of the lung and bronchus. Accessed October 19, 2020 https://seer.cancer.gov/archive/csr/1975_2015/results_merged/sect_15_lung_bronchus.pdf. Updated September 10, 2018
11. Passaro A, Spitaleri G, Gyawali B, de Marinis F. Immunotherapy in non-small-cell lung cancer patients with performance status 2: clinical decision making with scant evidence. J Clin Oncol. 2019;37(22):1863-1867. doi:10.1200/JCO.18.02118
12. Driessen EJM, Janssen-Heijnen MLG, Maas HA, Dingemans AC, van Loon JGM. Study protocol of the NVALT25-ELDAPT trial: selecting the optimal treatment for older patients with stage III non-small-cell lung cancer. Clin Lung Cancer. 2018;19(6):e849-e852. doi:10.1016/j.cllc.2018.07.003
13. Schulkes KJ, Nguyen C, van den Bos F, van Elden LJ, Hamaker ME. Selection of Patients in Ongoing Clinical Trials on Lung Cancer. Lung. 2016;194(6):967-974. doi:10.1007/s00408-016-9943-7
14. Blonde L, Khunti K, Harris SB, Meizinger C, Skolnik NS. Interpretation and impact of real-world clinical data for the practicing clinician. Adv Ther. 2018;35(11):1763-1774. doi:10.1007/s12325-018-0805-y
15. Garrison LP Jr, Neumann PJ, Erickson P, Marshall D, Mullins CD. Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report. Value Health. 2007;10(5):326-335. doi:10.1111/j.1524-4733.2007.00186.x
16. Assari S. Veterans and risk of heart disease in the United States: a cohort with 20 years of follow up. Int J Prev Med. 2014;5(6):703-709.
17. Shahoumian TA, Phillips BR, Backus LI. Cigarette smoking, reduction and quit attempts: prevalence among veterans with coronary heart disease. Prev Chronic Dis. 2016;13:E41. Published 2016 Mar 24. doi:10.5888/pcd13.150282
18. Murphy DE, Chaudhry Z, Almoosa KF, Panos RJ. High prevalence of chronic obstructive pulmonary disease among veterans in the urban midwest. Mil Med. 2011;176(5):552-560. doi:10.7205/milmed-d-10-00377
19. Kozel FA, Didehbani N, DeLaRosa B, et al. Factors impacting functional status in veterans of recent conflicts with PTSD. J Neuropsychiatry Clin Neurosci. 2016;28(2):112-117. doi:10.1176/appi.neuropsych.15070183
The US Food and Drug Administration (FDA) approved the use of durvalumab for patients with unresectable stage III non-small cell lung cancer (NSCLC) whose disease has not progressed following concurrent platinum-based chemotherapy and radiation therapy (CRT).1 After 2 randomized phase 3 studies in 2017 and 2018 showed significant progression-free and overall survival respectively,2,3 durvalumab became a category 1 recommendation for the above indication per National Comprehensive Cancer Network (NCCN) guidelines.4 Adherence to guidelines have been shown to improve patient survival across several cancer types.5-7 However, guideline adherence rates have been variable across health institutions. Therefore, further study is warranted to evaluate nonadherent practices with the goal of improving the quality of cancer care delivery.8,9
Stage III NSCLC is associated with poor survival rates.10 Concurrent CRT remains the standard of care in patients with good performance status based on clinical trial populations.4 Lung cancer remains a disease of the elderly, with a median age at diagnosis of 70 years.11 Discrepancies in the treatment of lung cancer in older adults can vary widely due to a lack of evidence surrounding the treatment in those who have comorbidities and poor performance status, widening the gap between clinical trial and real-world populations.11
A recent review by Passaro and colleagues revealed that at least 11 pivotal randomized controlled trials have shown the activity of immune checkpoint inhibitors (ICI) in locally advanced and metastatic lung cancer. However, these studies have mostly excluded patients with a performance status of the Eastern Cooperative Oncology Group (ECOG) level ≥ 2.11
Durvalumab is one of many new therapies to enter clinical practice to demonstrate survival benefit, but its use among veterans with stage III NSCLC in adherence with National Comprehensive Cancer Network (NCCN) guidelines was not robust at the Birmingham Veterans Affairs Medical Center (VAMC) in Alabama. Therefore, we decided to study the level of adherence and to identify barriers to conformity to the category 1 NCCN recommendations.
Methods
The Birmingham VAMC Outpatient Oncology Clinic billing data identified all individuals diagnosed with lung cancer treated between October 2017 and August 2019. Patients who did not have NSCLC that was stage III and unresectable were excluded from our study. Patients who did not receive a majority of their treatment at US Department of Veterans Affairs (VA) facilities were excluded as well. Each patient’s demographic, functional level, and tumor characteristics during the treatment planning phase and follow-up visits were obtained. Two investigators who evaluated health care provider documentation using the VA Computerized Patient Record System (CPRS) conducted chart reviews.
The primary outcomes were the proportion of patients who received concurrent CRT and the proportion who received durvalumab consolidation. Our chart review also categorized reasons for nonreceipt of concurrent CRT and subsequent durvalumab. Documented reasons for guideline discordancy were generated empirically and broadly. We noted if documentation was unclear and included reasons for why a veteran was not a candidate for CRT, the presence of toxicities associated with CRT, and a patient’s refusal for therapy despite medical advice. Descriptive data were analyzed for all clinical or demographic characteristics and outcomes.
This was considered an internal quality improvement initiative. As such, Birmingham VAMC did not require institutional review board approval for the study. The facility is accredited by the American College of Surgeons Commission on Cancer.
Results
A total of 41 veterans with stage III NSCLC were identified to have established care in the Birmingham VAMC Oncology Clinic between October 2017 and August 2019. Of these, 7 received the majority of their treatment from community-based non-VA facilities and 14 were not candidates for CRT and were excluded from this study.
The mean (SD) age of study participants was 70.0 (8.4) years (range, 57 to 92 years). Most of the study veterans (33; 97.1%) were male and 20 (58.8%) were African American (Table). Eighteen (53%) of study participants had clinical stage IIIa NSCLC; 19 (56%) showed a squamous subtype of NSCLC. A majority (53%) of the veterans studied were evaluated to be functionally fit with an ECOG status of 0 to 1, although documentation of ECOG status was lacking in 5 (14.7%) patients in the initial treatment planning visit records. It was unclear if performance status had been reevaluated and changes noted over the course of concurrent CRT.
CRT Patients
The relative distribution of veterans who underwent CRT for stage III NSCLC plus the reasons they did not receive guideline-based treatment with durvalumab is shown in the Figure. Fourteen patients (41%) were inappropriate candidates for CRT; the most common reason for this was their poor performance status upon initial evaluation and 3 patients (8.8%) in the study had extensive disease or were upstaged upon follow-up clinic visit.
Twenty (59%) veterans in the study initiated CRT. However, only 16 (47.1%) completed CRT. Those who dropped out of CRT did so because of toxicities that included various cytopenia, gastrointestinal toxicities due to radiation and/or chemotherapy, or failure to thrive.
Durvalumab Treatment
After initiation of CRT, 9 (26.5%) patients did not go on to receive durvalumab. Three patients (8.8%) suffered toxicities during CRT. One study patient was found to have a severe respiratory infection requiring intensive care unit admission. Another study patient was found to have a new sternal lesion on follow-up positron emission tomography. One declined because of a history of severe antineutrophil cytoplasmic antibodies vasculitis, which made durvalumab use unsafe. Three patients (8.8%) declined treatment with CRT or durvalumab because of personal preference. Documentation was unclear as to why durvalumab was prescribed to one patient who had completed CRT.
Discussion
NCCN guidelines on the use of durvalumab in NSCLC are based on the phase 3 PACIFIC placebo-controlled randomized clinical trial. This trial, which included only patients with documented performance status of ECOG 0 or 1, reported that grade 3 or 4 events occurred in 30.5% of patients randomized to consolidative durvalumab. Treatment was discontinued in 15.4% of patients due to adverse events.3
Our study examined consolidation therapy with durvalumab in patients with unresectable stage III NSCLC with an ECOG performance status of 0 to 1 who had not progressed after 2 or more cycles of definitive concurrent CRT.4 Patients with previous exposure to immunotherapy, a history of immunodeficiency, active infection, unresolved toxicity from CRT, autoimmune disease, and patients who received sequential CRT were excluded.2 Surprisingly, the adherence rate to guidelines was close to 100% with appropriate documentation and justification of CRT initiation and durvalumab use. Five (14.7%) of veterans with unresectable stage III NSCLC did not have clear documentation of ECOG status on initial visit and only 1 veteran who completed CRT did not have clear documentation as to why durvalumab was not provided. Unfortunately, 23 (68.6%) veterans in the study were unable to receive durvalumab, a potentially disease-modifying drug; nearly one-third (10) of veterans were deemed poor candidates for concurrent CRT despite the fact that 52.9% (18) of veterans in the study had a documented ECOG of 0 or 1 on initial evaluation.
Clinical Trials vs Real World
The heterogeneity between anticipated study populations, those who were able to receive durvalumab in the PACIFIC trial, compared with our observed real-world veteran population, likely stems from the lack of information about how comorbidity and fitness can affect the choice of therapeutic intervention in patients with lung cancer.12 In addition, older adults who participated in randomized controlled trials (RCTs) are not representative of the average older adult who presents to medical oncology clinics, making the application of guideline concordant care difficult.13
Similar real-world observations parallel to our analyses have confirmed, complemented and/or refuted findings of RCTs, and have helped impact the treatment of multiple acute and chronic conditions including influenza, cardiovascular disease, and diabetes.14
A component of socioeconomic barriers and access to supportive care played roles in the decisions of certain patients who chose not to undergo concurrent CRT despite medical advice. These 2 obstacles also affected the decision making for some in the study when considering the use of durvalumab (administered by a 60-minute IV infusion every 2 weeks for 1 year) per recommended guidelines.1 These hurdles need further study in the context of their effect on quality of life and the difficulties generated by various social determinants of health.
Limitations
Study limitations included the biased and confounding factors previously described about retrospective and nonrandomized observational studies that are controlled for during RCTs.15 Electronic health record data may have been incorrectly collected resulting in missing or wrong data points that affect the validity of our conclusion. Recall bias with regard to documentation by health care providers describing reasons why CRT or durvalumab were not initiated or the patient’s ability to recall previous treatments and report ECOG status or toxicities also may have impacted our findings. Comorbidities and poor performance status, frequently occurring among veterans, negatively impact cancer treatment decisions and may result in a detection bias. For example, tobacco use, cardiovascular disease, including heart failure, and chronic obstructive pulmonary disease, are notoriously higher in the US veteran population when compared with civilian cohorts.16-18 Also, veterans with poorly controlled depression and posttraumatic stress disorder resulting in functional impairment are a factor.19 Steps were taken to address some of these biases by performing repeat checks of tabulated data and employing 2 independent reviewers to evaluate all relevant clinical documentation, compare results, and reach a consensus.
Conlcusions
This retrospective analysis of adherence to category 1 NCCN guidelines for durvalumab use among patients at the Birmingham VAMC Oncology Clinic reinforced our practice and identified minor deficiencies in documentation that would impact future clinical visits. More importantly, it depicted the massive disparity in treatment candidacy among Birmingham veterans compared with clinical trial populations. Efforts will be made to address factors impacting a veteran’s candidacy for CRT and explore other variables such as socioeconomic barriers to treatment. Multiple complementary tools to assess patients’ frailty, such as the Charlson Comorbidity Index (CCI), are now being used for a variety of disorders including cancers. More robust data and standardization are needed to validate the use of these assessments in predicting response to immune checkpoint inhibitors.
Immune checkpoint inhibitors are currently being evaluated in stage III NSCLC studies and may be implemented as routine practice in the future.12 It is important to distinguish fit from frail veterans with lung cancer for treatment selection. We would like to see the expansion of the eligibility criteria for clinical trials to include patients with a performance status of ECOG 2 in order for results to be truly generalizable to the real-world population. Our hope is that such work will improve not only the quality of lung cancer care, but also the quality of care across multiple tumor types.
The US Food and Drug Administration (FDA) approved the use of durvalumab for patients with unresectable stage III non-small cell lung cancer (NSCLC) whose disease has not progressed following concurrent platinum-based chemotherapy and radiation therapy (CRT).1 After 2 randomized phase 3 studies in 2017 and 2018 showed significant progression-free and overall survival respectively,2,3 durvalumab became a category 1 recommendation for the above indication per National Comprehensive Cancer Network (NCCN) guidelines.4 Adherence to guidelines have been shown to improve patient survival across several cancer types.5-7 However, guideline adherence rates have been variable across health institutions. Therefore, further study is warranted to evaluate nonadherent practices with the goal of improving the quality of cancer care delivery.8,9
Stage III NSCLC is associated with poor survival rates.10 Concurrent CRT remains the standard of care in patients with good performance status based on clinical trial populations.4 Lung cancer remains a disease of the elderly, with a median age at diagnosis of 70 years.11 Discrepancies in the treatment of lung cancer in older adults can vary widely due to a lack of evidence surrounding the treatment in those who have comorbidities and poor performance status, widening the gap between clinical trial and real-world populations.11
A recent review by Passaro and colleagues revealed that at least 11 pivotal randomized controlled trials have shown the activity of immune checkpoint inhibitors (ICI) in locally advanced and metastatic lung cancer. However, these studies have mostly excluded patients with a performance status of the Eastern Cooperative Oncology Group (ECOG) level ≥ 2.11
Durvalumab is one of many new therapies to enter clinical practice to demonstrate survival benefit, but its use among veterans with stage III NSCLC in adherence with National Comprehensive Cancer Network (NCCN) guidelines was not robust at the Birmingham Veterans Affairs Medical Center (VAMC) in Alabama. Therefore, we decided to study the level of adherence and to identify barriers to conformity to the category 1 NCCN recommendations.
Methods
The Birmingham VAMC Outpatient Oncology Clinic billing data identified all individuals diagnosed with lung cancer treated between October 2017 and August 2019. Patients who did not have NSCLC that was stage III and unresectable were excluded from our study. Patients who did not receive a majority of their treatment at US Department of Veterans Affairs (VA) facilities were excluded as well. Each patient’s demographic, functional level, and tumor characteristics during the treatment planning phase and follow-up visits were obtained. Two investigators who evaluated health care provider documentation using the VA Computerized Patient Record System (CPRS) conducted chart reviews.
The primary outcomes were the proportion of patients who received concurrent CRT and the proportion who received durvalumab consolidation. Our chart review also categorized reasons for nonreceipt of concurrent CRT and subsequent durvalumab. Documented reasons for guideline discordancy were generated empirically and broadly. We noted if documentation was unclear and included reasons for why a veteran was not a candidate for CRT, the presence of toxicities associated with CRT, and a patient’s refusal for therapy despite medical advice. Descriptive data were analyzed for all clinical or demographic characteristics and outcomes.
This was considered an internal quality improvement initiative. As such, Birmingham VAMC did not require institutional review board approval for the study. The facility is accredited by the American College of Surgeons Commission on Cancer.
Results
A total of 41 veterans with stage III NSCLC were identified to have established care in the Birmingham VAMC Oncology Clinic between October 2017 and August 2019. Of these, 7 received the majority of their treatment from community-based non-VA facilities and 14 were not candidates for CRT and were excluded from this study.
The mean (SD) age of study participants was 70.0 (8.4) years (range, 57 to 92 years). Most of the study veterans (33; 97.1%) were male and 20 (58.8%) were African American (Table). Eighteen (53%) of study participants had clinical stage IIIa NSCLC; 19 (56%) showed a squamous subtype of NSCLC. A majority (53%) of the veterans studied were evaluated to be functionally fit with an ECOG status of 0 to 1, although documentation of ECOG status was lacking in 5 (14.7%) patients in the initial treatment planning visit records. It was unclear if performance status had been reevaluated and changes noted over the course of concurrent CRT.
CRT Patients
The relative distribution of veterans who underwent CRT for stage III NSCLC plus the reasons they did not receive guideline-based treatment with durvalumab is shown in the Figure. Fourteen patients (41%) were inappropriate candidates for CRT; the most common reason for this was their poor performance status upon initial evaluation and 3 patients (8.8%) in the study had extensive disease or were upstaged upon follow-up clinic visit.
Twenty (59%) veterans in the study initiated CRT. However, only 16 (47.1%) completed CRT. Those who dropped out of CRT did so because of toxicities that included various cytopenia, gastrointestinal toxicities due to radiation and/or chemotherapy, or failure to thrive.
Durvalumab Treatment
After initiation of CRT, 9 (26.5%) patients did not go on to receive durvalumab. Three patients (8.8%) suffered toxicities during CRT. One study patient was found to have a severe respiratory infection requiring intensive care unit admission. Another study patient was found to have a new sternal lesion on follow-up positron emission tomography. One declined because of a history of severe antineutrophil cytoplasmic antibodies vasculitis, which made durvalumab use unsafe. Three patients (8.8%) declined treatment with CRT or durvalumab because of personal preference. Documentation was unclear as to why durvalumab was prescribed to one patient who had completed CRT.
Discussion
NCCN guidelines on the use of durvalumab in NSCLC are based on the phase 3 PACIFIC placebo-controlled randomized clinical trial. This trial, which included only patients with documented performance status of ECOG 0 or 1, reported that grade 3 or 4 events occurred in 30.5% of patients randomized to consolidative durvalumab. Treatment was discontinued in 15.4% of patients due to adverse events.3
Our study examined consolidation therapy with durvalumab in patients with unresectable stage III NSCLC with an ECOG performance status of 0 to 1 who had not progressed after 2 or more cycles of definitive concurrent CRT.4 Patients with previous exposure to immunotherapy, a history of immunodeficiency, active infection, unresolved toxicity from CRT, autoimmune disease, and patients who received sequential CRT were excluded.2 Surprisingly, the adherence rate to guidelines was close to 100% with appropriate documentation and justification of CRT initiation and durvalumab use. Five (14.7%) of veterans with unresectable stage III NSCLC did not have clear documentation of ECOG status on initial visit and only 1 veteran who completed CRT did not have clear documentation as to why durvalumab was not provided. Unfortunately, 23 (68.6%) veterans in the study were unable to receive durvalumab, a potentially disease-modifying drug; nearly one-third (10) of veterans were deemed poor candidates for concurrent CRT despite the fact that 52.9% (18) of veterans in the study had a documented ECOG of 0 or 1 on initial evaluation.
Clinical Trials vs Real World
The heterogeneity between anticipated study populations, those who were able to receive durvalumab in the PACIFIC trial, compared with our observed real-world veteran population, likely stems from the lack of information about how comorbidity and fitness can affect the choice of therapeutic intervention in patients with lung cancer.12 In addition, older adults who participated in randomized controlled trials (RCTs) are not representative of the average older adult who presents to medical oncology clinics, making the application of guideline concordant care difficult.13
Similar real-world observations parallel to our analyses have confirmed, complemented and/or refuted findings of RCTs, and have helped impact the treatment of multiple acute and chronic conditions including influenza, cardiovascular disease, and diabetes.14
A component of socioeconomic barriers and access to supportive care played roles in the decisions of certain patients who chose not to undergo concurrent CRT despite medical advice. These 2 obstacles also affected the decision making for some in the study when considering the use of durvalumab (administered by a 60-minute IV infusion every 2 weeks for 1 year) per recommended guidelines.1 These hurdles need further study in the context of their effect on quality of life and the difficulties generated by various social determinants of health.
Limitations
Study limitations included the biased and confounding factors previously described about retrospective and nonrandomized observational studies that are controlled for during RCTs.15 Electronic health record data may have been incorrectly collected resulting in missing or wrong data points that affect the validity of our conclusion. Recall bias with regard to documentation by health care providers describing reasons why CRT or durvalumab were not initiated or the patient’s ability to recall previous treatments and report ECOG status or toxicities also may have impacted our findings. Comorbidities and poor performance status, frequently occurring among veterans, negatively impact cancer treatment decisions and may result in a detection bias. For example, tobacco use, cardiovascular disease, including heart failure, and chronic obstructive pulmonary disease, are notoriously higher in the US veteran population when compared with civilian cohorts.16-18 Also, veterans with poorly controlled depression and posttraumatic stress disorder resulting in functional impairment are a factor.19 Steps were taken to address some of these biases by performing repeat checks of tabulated data and employing 2 independent reviewers to evaluate all relevant clinical documentation, compare results, and reach a consensus.
Conlcusions
This retrospective analysis of adherence to category 1 NCCN guidelines for durvalumab use among patients at the Birmingham VAMC Oncology Clinic reinforced our practice and identified minor deficiencies in documentation that would impact future clinical visits. More importantly, it depicted the massive disparity in treatment candidacy among Birmingham veterans compared with clinical trial populations. Efforts will be made to address factors impacting a veteran’s candidacy for CRT and explore other variables such as socioeconomic barriers to treatment. Multiple complementary tools to assess patients’ frailty, such as the Charlson Comorbidity Index (CCI), are now being used for a variety of disorders including cancers. More robust data and standardization are needed to validate the use of these assessments in predicting response to immune checkpoint inhibitors.
Immune checkpoint inhibitors are currently being evaluated in stage III NSCLC studies and may be implemented as routine practice in the future.12 It is important to distinguish fit from frail veterans with lung cancer for treatment selection. We would like to see the expansion of the eligibility criteria for clinical trials to include patients with a performance status of ECOG 2 in order for results to be truly generalizable to the real-world population. Our hope is that such work will improve not only the quality of lung cancer care, but also the quality of care across multiple tumor types.
1. US Food and Drug Administration. FDA approves durvalumab after chemoradiation for unresectable stage II. Published February 20, 2018. Accessed October 9, 2020. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-durvalumab-after-chemoradiation-unresectable-stage-iii-nsclc
2. Antonia SJ, Villegas A, Daniel D, et al. Durvalumab after chemoradiotherapy in stage III non-small-cell lung cancer. N Engl J Med. 2017;377(20):1919-1929. doi:10.1056/NEJMoa1709937
3. Antonia SJ, Villegas A, Daniel D, et al. Overall survival with durvalumab after chemoradiotherapy in stage III NSCLC. N Engl J Med. 2018;379(24):2342-2350. doi:10.1056/NEJMoa1809697
4. Ettinger DS, Wood DE, Aisner DL et al. NCCN clinical practice guidelines in oncology: non-small cell lung cancer. Version8.2020. Updated September 15, 2020. Accessed October 9, 2020. https://www.nccn.org/professionals/physician_gls/pdf/nscl.pdf
5. Bristow RE, Chang J, Ziogas A, Campos B, Chavez LR, Anton-Culver H. Impact of National Cancer Institute Comprehensive Cancer Centers on ovarian cancer treatment and survival. J Am Coll Surg. 2015;220(5):940-950. doi:10.1016/j.jamcollsurg.2015.01.056
6. Boland GM, Chang GJ, Haynes AB, et al. Association between adherence to National Comprehensive Cancer Network treatment guidelines and improved survival in patients with colon cancer. Cancer. 2013;119(8):1593-1601. doi:10.1002/cncr.27935
7. Schwentner L, Wöckel A, König J, et al. Adherence to treatment guidelines and survival in triple-negative breast cancer: a retrospective multi-center cohort study with 9,156 patients. BMC Cancer. 2013;13:487. Published 2013 Oct 21. doi:10.1186/1471-2407-13-487
8. Jazieh A, Alkaiyat MO, Ali Y, Hashim MA, Abdelhafiz N, Al Olayan A. Improving adherence to lung cancer guidelines: a quality improvement project that uses chart review, audit and feedback approach. BMJ Open Qual. 2019;8(3):e000436. Published 2019 Aug 26. doi:10.1136/bmjoq-2018-000436
9. Shaverdian N, Offin MD, Rimner A, et al. Utilization and factors precluding the initiation of consolidative durvalumab in unresectable stage III non-small cell lung cancer. Radiother Oncol. 2020;144:101-104. doi:10.1016/j.radonc.2019.11.015
10. National Cancer Institute. SEER cancer statistics review, 1975-2015, Table 15.1 cancer of the lung and bronchus. Accessed October 19, 2020 https://seer.cancer.gov/archive/csr/1975_2015/results_merged/sect_15_lung_bronchus.pdf. Updated September 10, 2018
11. Passaro A, Spitaleri G, Gyawali B, de Marinis F. Immunotherapy in non-small-cell lung cancer patients with performance status 2: clinical decision making with scant evidence. J Clin Oncol. 2019;37(22):1863-1867. doi:10.1200/JCO.18.02118
12. Driessen EJM, Janssen-Heijnen MLG, Maas HA, Dingemans AC, van Loon JGM. Study protocol of the NVALT25-ELDAPT trial: selecting the optimal treatment for older patients with stage III non-small-cell lung cancer. Clin Lung Cancer. 2018;19(6):e849-e852. doi:10.1016/j.cllc.2018.07.003
13. Schulkes KJ, Nguyen C, van den Bos F, van Elden LJ, Hamaker ME. Selection of Patients in Ongoing Clinical Trials on Lung Cancer. Lung. 2016;194(6):967-974. doi:10.1007/s00408-016-9943-7
14. Blonde L, Khunti K, Harris SB, Meizinger C, Skolnik NS. Interpretation and impact of real-world clinical data for the practicing clinician. Adv Ther. 2018;35(11):1763-1774. doi:10.1007/s12325-018-0805-y
15. Garrison LP Jr, Neumann PJ, Erickson P, Marshall D, Mullins CD. Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report. Value Health. 2007;10(5):326-335. doi:10.1111/j.1524-4733.2007.00186.x
16. Assari S. Veterans and risk of heart disease in the United States: a cohort with 20 years of follow up. Int J Prev Med. 2014;5(6):703-709.
17. Shahoumian TA, Phillips BR, Backus LI. Cigarette smoking, reduction and quit attempts: prevalence among veterans with coronary heart disease. Prev Chronic Dis. 2016;13:E41. Published 2016 Mar 24. doi:10.5888/pcd13.150282
18. Murphy DE, Chaudhry Z, Almoosa KF, Panos RJ. High prevalence of chronic obstructive pulmonary disease among veterans in the urban midwest. Mil Med. 2011;176(5):552-560. doi:10.7205/milmed-d-10-00377
19. Kozel FA, Didehbani N, DeLaRosa B, et al. Factors impacting functional status in veterans of recent conflicts with PTSD. J Neuropsychiatry Clin Neurosci. 2016;28(2):112-117. doi:10.1176/appi.neuropsych.15070183
1. US Food and Drug Administration. FDA approves durvalumab after chemoradiation for unresectable stage II. Published February 20, 2018. Accessed October 9, 2020. https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-durvalumab-after-chemoradiation-unresectable-stage-iii-nsclc
2. Antonia SJ, Villegas A, Daniel D, et al. Durvalumab after chemoradiotherapy in stage III non-small-cell lung cancer. N Engl J Med. 2017;377(20):1919-1929. doi:10.1056/NEJMoa1709937
3. Antonia SJ, Villegas A, Daniel D, et al. Overall survival with durvalumab after chemoradiotherapy in stage III NSCLC. N Engl J Med. 2018;379(24):2342-2350. doi:10.1056/NEJMoa1809697
4. Ettinger DS, Wood DE, Aisner DL et al. NCCN clinical practice guidelines in oncology: non-small cell lung cancer. Version8.2020. Updated September 15, 2020. Accessed October 9, 2020. https://www.nccn.org/professionals/physician_gls/pdf/nscl.pdf
5. Bristow RE, Chang J, Ziogas A, Campos B, Chavez LR, Anton-Culver H. Impact of National Cancer Institute Comprehensive Cancer Centers on ovarian cancer treatment and survival. J Am Coll Surg. 2015;220(5):940-950. doi:10.1016/j.jamcollsurg.2015.01.056
6. Boland GM, Chang GJ, Haynes AB, et al. Association between adherence to National Comprehensive Cancer Network treatment guidelines and improved survival in patients with colon cancer. Cancer. 2013;119(8):1593-1601. doi:10.1002/cncr.27935
7. Schwentner L, Wöckel A, König J, et al. Adherence to treatment guidelines and survival in triple-negative breast cancer: a retrospective multi-center cohort study with 9,156 patients. BMC Cancer. 2013;13:487. Published 2013 Oct 21. doi:10.1186/1471-2407-13-487
8. Jazieh A, Alkaiyat MO, Ali Y, Hashim MA, Abdelhafiz N, Al Olayan A. Improving adherence to lung cancer guidelines: a quality improvement project that uses chart review, audit and feedback approach. BMJ Open Qual. 2019;8(3):e000436. Published 2019 Aug 26. doi:10.1136/bmjoq-2018-000436
9. Shaverdian N, Offin MD, Rimner A, et al. Utilization and factors precluding the initiation of consolidative durvalumab in unresectable stage III non-small cell lung cancer. Radiother Oncol. 2020;144:101-104. doi:10.1016/j.radonc.2019.11.015
10. National Cancer Institute. SEER cancer statistics review, 1975-2015, Table 15.1 cancer of the lung and bronchus. Accessed October 19, 2020 https://seer.cancer.gov/archive/csr/1975_2015/results_merged/sect_15_lung_bronchus.pdf. Updated September 10, 2018
11. Passaro A, Spitaleri G, Gyawali B, de Marinis F. Immunotherapy in non-small-cell lung cancer patients with performance status 2: clinical decision making with scant evidence. J Clin Oncol. 2019;37(22):1863-1867. doi:10.1200/JCO.18.02118
12. Driessen EJM, Janssen-Heijnen MLG, Maas HA, Dingemans AC, van Loon JGM. Study protocol of the NVALT25-ELDAPT trial: selecting the optimal treatment for older patients with stage III non-small-cell lung cancer. Clin Lung Cancer. 2018;19(6):e849-e852. doi:10.1016/j.cllc.2018.07.003
13. Schulkes KJ, Nguyen C, van den Bos F, van Elden LJ, Hamaker ME. Selection of Patients in Ongoing Clinical Trials on Lung Cancer. Lung. 2016;194(6):967-974. doi:10.1007/s00408-016-9943-7
14. Blonde L, Khunti K, Harris SB, Meizinger C, Skolnik NS. Interpretation and impact of real-world clinical data for the practicing clinician. Adv Ther. 2018;35(11):1763-1774. doi:10.1007/s12325-018-0805-y
15. Garrison LP Jr, Neumann PJ, Erickson P, Marshall D, Mullins CD. Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report. Value Health. 2007;10(5):326-335. doi:10.1111/j.1524-4733.2007.00186.x
16. Assari S. Veterans and risk of heart disease in the United States: a cohort with 20 years of follow up. Int J Prev Med. 2014;5(6):703-709.
17. Shahoumian TA, Phillips BR, Backus LI. Cigarette smoking, reduction and quit attempts: prevalence among veterans with coronary heart disease. Prev Chronic Dis. 2016;13:E41. Published 2016 Mar 24. doi:10.5888/pcd13.150282
18. Murphy DE, Chaudhry Z, Almoosa KF, Panos RJ. High prevalence of chronic obstructive pulmonary disease among veterans in the urban midwest. Mil Med. 2011;176(5):552-560. doi:10.7205/milmed-d-10-00377
19. Kozel FA, Didehbani N, DeLaRosa B, et al. Factors impacting functional status in veterans of recent conflicts with PTSD. J Neuropsychiatry Clin Neurosci. 2016;28(2):112-117. doi:10.1176/appi.neuropsych.15070183