Oncologists agree with AI treatment decisions about half the time

When it comes to treatment recommendations for high-risk breast cancer, oncologists agree with a leading artificial intelligence platform about half the time, according to investigators.

In the first study of its kind, involving 10 Chinese oncologists, recommendation concordance with the Watson for Oncology treatment advisory tool (WfO) was generally lower for hormone receptor–positive and metastatic cancers than hormone receptor–negative and nonmetastatic cases, reported Fengrui Xu, MD, of the Academy of Military Medical Sciences in Beijing, and colleagues. Refinement could enable broad use of Watson, not to dictate treatment decisions, but instead to propose alternate treatment approaches and offer point-of-care access to relevant evidence.

“[WfO] is an example of a quantitative oncology clinical decision support that leverages the clinical expertise of oncologists at Memorial Sloan Kettering Cancer Center [MSKCC],” the investigators wrote in JCO Clinical Cancer Informatics. The platform uses machine-learning software to interpret patient scenarios in light from MSKCC training cases, MSKCC treatment guidelines, and more than 300 medical textbooks and journals.

To compare WfO with real-world decision makers, the investigators recruited three chief physicians, four attending physicians, and three fellows to provide treatment recommendations for 1,977 patients with complex breast cancer who were treated at 10 hospitals in China. Participating physicians shared the workload; each evaluated an average of 198 different cases.

On average, oncologists and WfO made the same treatment recommendations 56% of the time. Out of the different types of physicians, fellows were most likely to agree with WfO, based on a 68% concordance rate, compared with 54% for chief physicians and 49% for attending physicians. Including all physicians, concordance was lowest for hormone receptor–positive/HER2-positive disease (48%) and highest for triple-negative cases (71%). Adjuvant and metastatic therapies were also evaluated, with high concordance for adjuvant endocrine (78%) and targeted therapy (100%), compared with moderate concordance for first- (52%) and second-line metastatic therapy (50%). The investigators described concordance results as generally “modest;” however, they noted that such levels are promising.

“This degree of concordance is encouraging because therapeutic decisions in these cases are often difficult as a result of the current limits of medical knowledge for treating complex breast cancers and the presence of local contextual factors that affect physician treatment choices,” the investigators wrote. “It is important to note that nonconcordance does not imply that one treatment is correct for a given patient and another is not, nor does it necessarily diminish the potential value of a decision support system that provides access to supporting evidence and insight into its reasoning process.”

The study was funded by Zefei Jiang. The investigators reported affiliations with IBM Watson Health, Pharmaceutical Manufacturer Institution, Merck, and others.

SOURCE: Xu F et al. JCO Clin Cancer Inform. 2019 Aug 16. doi: 10.1200/CCI.18.00159.

When it comes to treatment recommendations for high-risk breast cancer, oncologists agree with a leading artificial intelligence platform about half the time, according to investigators.

In the first study of its kind, involving 10 Chinese oncologists, recommendation concordance with the Watson for Oncology treatment advisory tool (WfO) was generally lower for hormone receptor–positive and metastatic cancers than hormone receptor–negative and nonmetastatic cases, reported Fengrui Xu, MD, of the Academy of Military Medical Sciences in Beijing, and colleagues. Refinement could enable broad use of Watson, not to dictate treatment decisions, but instead to propose alternate treatment approaches and offer point-of-care access to relevant evidence.

“[WfO] is an example of a quantitative oncology clinical decision support that leverages the clinical expertise of oncologists at Memorial Sloan Kettering Cancer Center [MSKCC],” the investigators wrote in JCO Clinical Cancer Informatics. The platform uses machine-learning software to interpret patient scenarios in light from MSKCC training cases, MSKCC treatment guidelines, and more than 300 medical textbooks and journals.

To compare WfO with real-world decision makers, the investigators recruited three chief physicians, four attending physicians, and three fellows to provide treatment recommendations for 1,977 patients with complex breast cancer who were treated at 10 hospitals in China. Participating physicians shared the workload; each evaluated an average of 198 different cases.

On average, oncologists and WfO made the same treatment recommendations 56% of the time. Out of the different types of physicians, fellows were most likely to agree with WfO, based on a 68% concordance rate, compared with 54% for chief physicians and 49% for attending physicians. Including all physicians, concordance was lowest for hormone receptor–positive/HER2-positive disease (48%) and highest for triple-negative cases (71%). Adjuvant and metastatic therapies were also evaluated, with high concordance for adjuvant endocrine (78%) and targeted therapy (100%), compared with moderate concordance for first- (52%) and second-line metastatic therapy (50%). The investigators described concordance results as generally “modest;” however, they noted that such levels are promising.

“This degree of concordance is encouraging because therapeutic decisions in these cases are often difficult as a result of the current limits of medical knowledge for treating complex breast cancers and the presence of local contextual factors that affect physician treatment choices,” the investigators wrote. “It is important to note that nonconcordance does not imply that one treatment is correct for a given patient and another is not, nor does it necessarily diminish the potential value of a decision support system that provides access to supporting evidence and insight into its reasoning process.”

The study was funded by Zefei Jiang. The investigators reported affiliations with IBM Watson Health, Pharmaceutical Manufacturer Institution, Merck, and others.

SOURCE: Xu F et al. JCO Clin Cancer Inform. 2019 Aug 16. doi: 10.1200/CCI.18.00159.

When it comes to treatment recommendations for high-risk breast cancer, oncologists agree with a leading artificial intelligence platform about half the time, according to investigators.

In the first study of its kind, involving 10 Chinese oncologists, recommendation concordance with the Watson for Oncology treatment advisory tool (WfO) was generally lower for hormone receptor–positive and metastatic cancers than hormone receptor–negative and nonmetastatic cases, reported Fengrui Xu, MD, of the Academy of Military Medical Sciences in Beijing, and colleagues. Refinement could enable broad use of Watson, not to dictate treatment decisions, but instead to propose alternate treatment approaches and offer point-of-care access to relevant evidence.

“[WfO] is an example of a quantitative oncology clinical decision support that leverages the clinical expertise of oncologists at Memorial Sloan Kettering Cancer Center [MSKCC],” the investigators wrote in JCO Clinical Cancer Informatics. The platform uses machine-learning software to interpret patient scenarios in light from MSKCC training cases, MSKCC treatment guidelines, and more than 300 medical textbooks and journals.

To compare WfO with real-world decision makers, the investigators recruited three chief physicians, four attending physicians, and three fellows to provide treatment recommendations for 1,977 patients with complex breast cancer who were treated at 10 hospitals in China. Participating physicians shared the workload; each evaluated an average of 198 different cases.

On average, oncologists and WfO made the same treatment recommendations 56% of the time. Out of the different types of physicians, fellows were most likely to agree with WfO, based on a 68% concordance rate, compared with 54% for chief physicians and 49% for attending physicians. Including all physicians, concordance was lowest for hormone receptor–positive/HER2-positive disease (48%) and highest for triple-negative cases (71%). Adjuvant and metastatic therapies were also evaluated, with high concordance for adjuvant endocrine (78%) and targeted therapy (100%), compared with moderate concordance for first- (52%) and second-line metastatic therapy (50%). The investigators described concordance results as generally “modest;” however, they noted that such levels are promising.

“This degree of concordance is encouraging because therapeutic decisions in these cases are often difficult as a result of the current limits of medical knowledge for treating complex breast cancers and the presence of local contextual factors that affect physician treatment choices,” the investigators wrote. “It is important to note that nonconcordance does not imply that one treatment is correct for a given patient and another is not, nor does it necessarily diminish the potential value of a decision support system that provides access to supporting evidence and insight into its reasoning process.”

The study was funded by Zefei Jiang. The investigators reported affiliations with IBM Watson Health, Pharmaceutical Manufacturer Institution, Merck, and others.

SOURCE: Xu F et al. JCO Clin Cancer Inform. 2019 Aug 16. doi: 10.1200/CCI.18.00159.