Expert argues for change in the way biomarkers are evaluated

SEATTLE – It’s time to consider an alternative strategy for evaluating new biomarkers that focuses on the predictive information they add, Michael W. Kattan, Ph.D., said at a joint meeting by Global Biomarkers Consortium and World Cutaneous Malignancies Congress.

“In my view, we get too fixated on P values or hazard ratios and odds ratios. Instead, we need to step back and think more about what the goal of any new marker is, and often, it’s to improve our ability to predict a patient outcome,” said Dr. Kattan, professor of medicine, epidemiology, and biostatistics at Case Western Reserve University, Cleveland, and chair of quantitative health sciences at the Cleveland Clinic. “If that’s the case, why not worry more about something like incremental predictive accuracy or incremental predictive ability associated with that new marker, and make our decisions and our modeling steps toward that?”

The long-used, conventional three-step approach to evaluating a new biomarker – assessing its correlation with an established biomarker, its association with an outcome in univariate analysis, and finally its performance in a multivariate analysis (J Natl Cancer Inst. 2003;95:634-5) – has considerable issues, according to Dr. Kattan.

In particular, the multivariate analysis is problematic. “My P value is testing whether my hazard ratio is 1, it’s not per se an improvement in predictive accuracy, which is what I’m going to argue that the new marker should do,” he said. But more concerning is the fact that the hazard ratio is affected by factors the investigators control, such as whether the new biomarker is coded as a continuous or categorical variable, which established biomarkers are included, and any data transformations done.

“At the end of the day, things are getting a little bit subjective because I have a bunch of knobs under my control as the keeper of the data. I can turn all of these knobs, and unfortunately, I don’t have excellent arguments to defend how I would do that, and they may very well affect the [hazard ratio] that has everyone’s attention,” Dr. Kattan elaborated.

Thus, an alternative approach is needed, one that tests the new biomarker as part of a model and addresses the central question of whether it improves predictive accuracy, he maintained. “It’s [comparing] a model of markers that lacks the new marker versus a model of markers that contains the new marker. So it’s a model versus model comparison, it’s not simply looking at the marker in isolation, which is where we get in trouble with the typical way.”

Furthermore, aiming for the most accurate model removes much of the subjectivity of the conventional approach, he added. “Remember, I said there were knobs I could turn that might change the hazard ratio and I didn’t have a good defense for how I would turn these knobs. … Now I do, now I have an explicit goal: I want to have a prediction model that predicts patient outcome as accurately as I can. So whatever I’m doing with my knobs and stuff, that ought to be delivering a more accurate prediction model.”

Dr. Kattan outlined a four-step alternative approach to evaluating new biomarkers. The first step entails calculating the improvement in the concordance index, similar to an area under the receiver operating characteristic (ROC) curve, with the new biomarker. Ideally, that number will increase in a model that contains the marker, indicating an improvement in predictive accuracy.

In the second step, which assesses model calibration, established and new biomarkers are entered into a multivariate model predicting the outcome of interest (Clin Cancer Res. 2004;10:822-4). If the concordance index drops by a clinically significant degree when the new biomarker is omitted, indicating a loss of predictive accuracy, it advances.

The third step is to construct scatterplots comparing results obtained with prediction models of the outcome, say, 10-year progression-free survival, that do and do not contain the new marker, say, surgeon experience with prostatectomy (Cancer. 2009;115:1005-10). If the improvement in accuracy here is clinically significant, the marker again advances.

In the fourth and final step, decision curve analysis, the net benefit is plotted as a function of the threshold for clinical action (Epidemiology. 2010;21:128-38). “This gets at, should I be making clinical decisions based on the prediction model, or should I just treat everyone or treat no one. It’s a way of looking at what the net benefit is of the prediction model across the spectrum of predictions,” Dr. Kattan explained. “So you would first decide what’s my threshold for action … where’s it going to change what I do, and then read upwards [in the plot] and see what the net benefit is.”

Dr. Kattan disclosed that he receives consulting fees from Bayer, Exosome, GlaxoSmithKline, HistoSonics, and Merck.

SEATTLE – It’s time to consider an alternative strategy for evaluating new biomarkers that focuses on the predictive information they add, Michael W. Kattan, Ph.D., said at a joint meeting by Global Biomarkers Consortium and World Cutaneous Malignancies Congress.

“In my view, we get too fixated on P values or hazard ratios and odds ratios. Instead, we need to step back and think more about what the goal of any new marker is, and often, it’s to improve our ability to predict a patient outcome,” said Dr. Kattan, professor of medicine, epidemiology, and biostatistics at Case Western Reserve University, Cleveland, and chair of quantitative health sciences at the Cleveland Clinic. “If that’s the case, why not worry more about something like incremental predictive accuracy or incremental predictive ability associated with that new marker, and make our decisions and our modeling steps toward that?”

The long-used, conventional three-step approach to evaluating a new biomarker – assessing its correlation with an established biomarker, its association with an outcome in univariate analysis, and finally its performance in a multivariate analysis (J Natl Cancer Inst. 2003;95:634-5) – has considerable issues, according to Dr. Kattan.

In particular, the multivariate analysis is problematic. “My P value is testing whether my hazard ratio is 1, it’s not per se an improvement in predictive accuracy, which is what I’m going to argue that the new marker should do,” he said. But more concerning is the fact that the hazard ratio is affected by factors the investigators control, such as whether the new biomarker is coded as a continuous or categorical variable, which established biomarkers are included, and any data transformations done.

“At the end of the day, things are getting a little bit subjective because I have a bunch of knobs under my control as the keeper of the data. I can turn all of these knobs, and unfortunately, I don’t have excellent arguments to defend how I would do that, and they may very well affect the [hazard ratio] that has everyone’s attention,” Dr. Kattan elaborated.

Thus, an alternative approach is needed, one that tests the new biomarker as part of a model and addresses the central question of whether it improves predictive accuracy, he maintained. “It’s [comparing] a model of markers that lacks the new marker versus a model of markers that contains the new marker. So it’s a model versus model comparison, it’s not simply looking at the marker in isolation, which is where we get in trouble with the typical way.”

Furthermore, aiming for the most accurate model removes much of the subjectivity of the conventional approach, he added. “Remember, I said there were knobs I could turn that might change the hazard ratio and I didn’t have a good defense for how I would turn these knobs. … Now I do, now I have an explicit goal: I want to have a prediction model that predicts patient outcome as accurately as I can. So whatever I’m doing with my knobs and stuff, that ought to be delivering a more accurate prediction model.”

Dr. Kattan outlined a four-step alternative approach to evaluating new biomarkers. The first step entails calculating the improvement in the concordance index, similar to an area under the receiver operating characteristic (ROC) curve, with the new biomarker. Ideally, that number will increase in a model that contains the marker, indicating an improvement in predictive accuracy.

In the second step, which assesses model calibration, established and new biomarkers are entered into a multivariate model predicting the outcome of interest (Clin Cancer Res. 2004;10:822-4). If the concordance index drops by a clinically significant degree when the new biomarker is omitted, indicating a loss of predictive accuracy, it advances.

The third step is to construct scatterplots comparing results obtained with prediction models of the outcome, say, 10-year progression-free survival, that do and do not contain the new marker, say, surgeon experience with prostatectomy (Cancer. 2009;115:1005-10). If the improvement in accuracy here is clinically significant, the marker again advances.

In the fourth and final step, decision curve analysis, the net benefit is plotted as a function of the threshold for clinical action (Epidemiology. 2010;21:128-38). “This gets at, should I be making clinical decisions based on the prediction model, or should I just treat everyone or treat no one. It’s a way of looking at what the net benefit is of the prediction model across the spectrum of predictions,” Dr. Kattan explained. “So you would first decide what’s my threshold for action … where’s it going to change what I do, and then read upwards [in the plot] and see what the net benefit is.”

Dr. Kattan disclosed that he receives consulting fees from Bayer, Exosome, GlaxoSmithKline, HistoSonics, and Merck.

SEATTLE – It’s time to consider an alternative strategy for evaluating new biomarkers that focuses on the predictive information they add, Michael W. Kattan, Ph.D., said at a joint meeting by Global Biomarkers Consortium and World Cutaneous Malignancies Congress.

“In my view, we get too fixated on P values or hazard ratios and odds ratios. Instead, we need to step back and think more about what the goal of any new marker is, and often, it’s to improve our ability to predict a patient outcome,” said Dr. Kattan, professor of medicine, epidemiology, and biostatistics at Case Western Reserve University, Cleveland, and chair of quantitative health sciences at the Cleveland Clinic. “If that’s the case, why not worry more about something like incremental predictive accuracy or incremental predictive ability associated with that new marker, and make our decisions and our modeling steps toward that?”

The long-used, conventional three-step approach to evaluating a new biomarker – assessing its correlation with an established biomarker, its association with an outcome in univariate analysis, and finally its performance in a multivariate analysis (J Natl Cancer Inst. 2003;95:634-5) – has considerable issues, according to Dr. Kattan.

In particular, the multivariate analysis is problematic. “My P value is testing whether my hazard ratio is 1, it’s not per se an improvement in predictive accuracy, which is what I’m going to argue that the new marker should do,” he said. But more concerning is the fact that the hazard ratio is affected by factors the investigators control, such as whether the new biomarker is coded as a continuous or categorical variable, which established biomarkers are included, and any data transformations done.

“At the end of the day, things are getting a little bit subjective because I have a bunch of knobs under my control as the keeper of the data. I can turn all of these knobs, and unfortunately, I don’t have excellent arguments to defend how I would do that, and they may very well affect the [hazard ratio] that has everyone’s attention,” Dr. Kattan elaborated.

Thus, an alternative approach is needed, one that tests the new biomarker as part of a model and addresses the central question of whether it improves predictive accuracy, he maintained. “It’s [comparing] a model of markers that lacks the new marker versus a model of markers that contains the new marker. So it’s a model versus model comparison, it’s not simply looking at the marker in isolation, which is where we get in trouble with the typical way.”

Furthermore, aiming for the most accurate model removes much of the subjectivity of the conventional approach, he added. “Remember, I said there were knobs I could turn that might change the hazard ratio and I didn’t have a good defense for how I would turn these knobs. … Now I do, now I have an explicit goal: I want to have a prediction model that predicts patient outcome as accurately as I can. So whatever I’m doing with my knobs and stuff, that ought to be delivering a more accurate prediction model.”

Dr. Kattan outlined a four-step alternative approach to evaluating new biomarkers. The first step entails calculating the improvement in the concordance index, similar to an area under the receiver operating characteristic (ROC) curve, with the new biomarker. Ideally, that number will increase in a model that contains the marker, indicating an improvement in predictive accuracy.

In the second step, which assesses model calibration, established and new biomarkers are entered into a multivariate model predicting the outcome of interest (Clin Cancer Res. 2004;10:822-4). If the concordance index drops by a clinically significant degree when the new biomarker is omitted, indicating a loss of predictive accuracy, it advances.

The third step is to construct scatterplots comparing results obtained with prediction models of the outcome, say, 10-year progression-free survival, that do and do not contain the new marker, say, surgeon experience with prostatectomy (Cancer. 2009;115:1005-10). If the improvement in accuracy here is clinically significant, the marker again advances.

In the fourth and final step, decision curve analysis, the net benefit is plotted as a function of the threshold for clinical action (Epidemiology. 2010;21:128-38). “This gets at, should I be making clinical decisions based on the prediction model, or should I just treat everyone or treat no one. It’s a way of looking at what the net benefit is of the prediction model across the spectrum of predictions,” Dr. Kattan explained. “So you would first decide what’s my threshold for action … where’s it going to change what I do, and then read upwards [in the plot] and see what the net benefit is.”

Dr. Kattan disclosed that he receives consulting fees from Bayer, Exosome, GlaxoSmithKline, HistoSonics, and Merck.