Stopping Trials Early: Do Results Get Clouded?

The early stopping of a clinical trial because of clear benefit from a new drug or regimen is usually good news. In the long run, however, researchers may find that they lack clear answers about the ethics and validity—and sometimes the benefit itself—of doing so.

The controversy over early stopping and early release of data stems from the desire to speed patient access to what appears to be a superior treatment, and the question of whether this will end up being done at the cost of scientific validity.

Trials involving sunitinib (Sutent) for pancreatic neuroendocrine tumors and lenalidomide (Revlimid) for multiple myeloma are two recent examples.

In February, researchers announced that a data-monitoring committee stopped the 171-patient sunitinib study after an analysis revealed that progression-free survival, the primary end point, was much longer in the active treatment arm.

In December 2009, a 568-patient study was not stopped, but early release of initial data showed that progression-free survival was longer in patients who received maintenance therapy with the oral drug lenalidomide after stem cell transplant than in patients who got a placebo.

“On the one hand, if the trial really is positive, then that's important information to get out there to the medical community and patients,” said Dr. Richard Schilsky, chairman of Cancer and Leukemia Group B (CALGB), which conducted the multiple myeloma trial. “On the other hand, there's this risk that you put the data out there early, and it doesn't hold up over time.”

When trials are stopped early for benefit, “typically the experimental arm is so much better than the standard arm that it is felt that it is medically and ethically necessary to announce the results before the trial is actually completed,” said Dr. Schilsky, former president of the American Society of Clinical Oncology and chief of hematology and oncology at the University of Chicago Medical Center.

One area of contention is the choice of end point that's used in a study's predefined early-stopping rules. Presumably, the primary end point of the study is used in the early stopping criteria, Dr. Schilsky said. In oncology trials, that often means progression-free survival.

But progression-free survival often is used as a surrogate for overall survival, said Dr. Jennifer Obel, a member of ASCO's cancer communications committee and an oncologist with NorthShore University HealthSystem in Illinois. Progression-free survival is quicker to measure, but whether it's clinically meaningful is unclear. Both the sunitinib and lenalidomide studies were stopped for benefits in disease progression.

“The real, meaningful end point is overall survival,” Dr. Obel said. Usually, patients who live longer do so without progression, but in some instances, drugs have prolonged progression-free survival but not overall survival.

Smaller P Values, Fewer Events

Another argument against early stopping is that it can overestimate the treatment benefit. In a systematic review of trials stopped early for benefit, researchers found that trials that accrued fewer events (that is, with end points that drove early stopping) estimated larger treatment effects (JAMA 2005;294:2203-9).

“The whole problem with this is that the effect size that we're looking for is relatively small,” said Dr. Jeffrey Crawford, chief of medical oncology at Duke University in Durham, N.C. “We're looking for a very small difference in survival in an unselected population. When you look for these small differences, the only way to really know is to have a large enough sample size that you have confidence in the result.”

Dr. Schilsky noted that “the stringency of the statistical analysis plan is important.” Early stopping often requires a much smaller P value to show significance. “It's a much higher threshold for stopping early because there are so many fewer events. You don't want to declare a trial to be positive unless the events are so unbalanced that you think it's extremely unlikely to be due to chance,” he said.

Weighing Drug Availability

Patients in the control arm pose yet another complication. If the experimental arm is better, patients who receive the control drug or regimen often gain access to the experimental drug or regimen. “Does that then potentially confound the final interpretation of the study results, because then you have a potentially significant number of patients who cross over?” Dr. Schilsky asked.

Early stopping because of efficacy is even more complicated when the experimental drug is not yet available. In that event, it's up to the manufacturer to develop a compassionate-use protocol specifying the patient population and how to administer the drug, he noted. Compassionate use allows the drug to be available to patients, even as the company is following the usual regulatory procedure for approval.

The effects of early stopping because of efficacy go far beyond the trial itself, with implications for ongoing and future studies of the same disease. “If you stop early and declare that you have a [new] standard of care, does that impact other ongoing studies that are not using that [new standard] of care arm, or does that impact the planning for the next generation of studies?” Dr. Schilsky asked. “You may find that the accrual to other ongoing trials screeches to a halt because everyone says 'this trial design is no longer relevant in light of this new information.'”

Making the Decision

Ultimately, the decision to stop a trial early is in the hands of the safety data–monitoring committee (DMC). “This sort of data release is almost always undertaken only after a very thorough review by an independent data-monitoring committee,” Dr. Schilsky said. “DMCs are in the best position to make these decisions and they do it only after lots of careful deliberation.”

Ideally, the DMC should weigh in and make a recommendation on what should be done for the patients who are in the control arm of the study, at the very least, he said.

In the case of lenalidomide for multiple myeloma, “the study was markedly positive in favor of the Revlimid maintenance, and led our data- and safety-monitoring board to recommend early release of the results,” Dr. Schilsky noted. Accrual had been completed and all patients were in follow-up, but “the study crossed a statistical boundary for early release of the data before the protocol-specified number of events for the final analysis was available.”

This process involved notifying the Food and Drug Administration, the National Cancer Institute, and the drug manufacturer, as well as developing letters for physicians and patients involved in the study, and sending out a press release. In this case, because lenalidomide is already available, oncologists could start prescribing the drug for this patient group immediately.

Advancing technology and the development of more personalized medicine may help resolve some of the issues concerning the early stopping of trials, Dr. Crawford suggested.

“What will solve this will be biomarker-directed treatment subgroups. … We can look at a much smaller sample of patients because we're looking at a much bigger effect,” he said. “In the long run, that's going to be a much better approach to oncology than continuing to treat thousands of patients with treatment A vs. B.”

Disclosures: Both Dr. Schilsky and Dr. Obel said that they have no significant financial relationships. Dr. Crawford has received research support from and been an adviser, speaker, or consultant for several pharmaceutical companies.

'There's this risk that you put the data out there early, and it doesn't hold up over time.'

Source Dr. Schilsky

The early stopping of a clinical trial because of clear benefit from a new drug or regimen is usually good news. In the long run, however, researchers may find that they lack clear answers about the ethics and validity—and sometimes the benefit itself—of doing so.

The controversy over early stopping and early release of data stems from the desire to speed patient access to what appears to be a superior treatment, and the question of whether this will end up being done at the cost of scientific validity.

Trials involving sunitinib (Sutent) for pancreatic neuroendocrine tumors and lenalidomide (Revlimid) for multiple myeloma are two recent examples.

In February, researchers announced that a data-monitoring committee stopped the 171-patient sunitinib study after an analysis revealed that progression-free survival, the primary end point, was much longer in the active treatment arm.

In December 2009, a 568-patient study was not stopped, but early release of initial data showed that progression-free survival was longer in patients who received maintenance therapy with the oral drug lenalidomide after stem cell transplant than in patients who got a placebo.

“On the one hand, if the trial really is positive, then that's important information to get out there to the medical community and patients,” said Dr. Richard Schilsky, chairman of Cancer and Leukemia Group B (CALGB), which conducted the multiple myeloma trial. “On the other hand, there's this risk that you put the data out there early, and it doesn't hold up over time.”

When trials are stopped early for benefit, “typically the experimental arm is so much better than the standard arm that it is felt that it is medically and ethically necessary to announce the results before the trial is actually completed,” said Dr. Schilsky, former president of the American Society of Clinical Oncology and chief of hematology and oncology at the University of Chicago Medical Center.

One area of contention is the choice of end point that's used in a study's predefined early-stopping rules. Presumably, the primary end point of the study is used in the early stopping criteria, Dr. Schilsky said. In oncology trials, that often means progression-free survival.

But progression-free survival often is used as a surrogate for overall survival, said Dr. Jennifer Obel, a member of ASCO's cancer communications committee and an oncologist with NorthShore University HealthSystem in Illinois. Progression-free survival is quicker to measure, but whether it's clinically meaningful is unclear. Both the sunitinib and lenalidomide studies were stopped for benefits in disease progression.

“The real, meaningful end point is overall survival,” Dr. Obel said. Usually, patients who live longer do so without progression, but in some instances, drugs have prolonged progression-free survival but not overall survival.

Smaller P Values, Fewer Events

Another argument against early stopping is that it can overestimate the treatment benefit. In a systematic review of trials stopped early for benefit, researchers found that trials that accrued fewer events (that is, with end points that drove early stopping) estimated larger treatment effects (JAMA 2005;294:2203-9).

“The whole problem with this is that the effect size that we're looking for is relatively small,” said Dr. Jeffrey Crawford, chief of medical oncology at Duke University in Durham, N.C. “We're looking for a very small difference in survival in an unselected population. When you look for these small differences, the only way to really know is to have a large enough sample size that you have confidence in the result.”

Dr. Schilsky noted that “the stringency of the statistical analysis plan is important.” Early stopping often requires a much smaller P value to show significance. “It's a much higher threshold for stopping early because there are so many fewer events. You don't want to declare a trial to be positive unless the events are so unbalanced that you think it's extremely unlikely to be due to chance,” he said.

Weighing Drug Availability

Patients in the control arm pose yet another complication. If the experimental arm is better, patients who receive the control drug or regimen often gain access to the experimental drug or regimen. “Does that then potentially confound the final interpretation of the study results, because then you have a potentially significant number of patients who cross over?” Dr. Schilsky asked.

Early stopping because of efficacy is even more complicated when the experimental drug is not yet available. In that event, it's up to the manufacturer to develop a compassionate-use protocol specifying the patient population and how to administer the drug, he noted. Compassionate use allows the drug to be available to patients, even as the company is following the usual regulatory procedure for approval.

The effects of early stopping because of efficacy go far beyond the trial itself, with implications for ongoing and future studies of the same disease. “If you stop early and declare that you have a [new] standard of care, does that impact other ongoing studies that are not using that [new standard] of care arm, or does that impact the planning for the next generation of studies?” Dr. Schilsky asked. “You may find that the accrual to other ongoing trials screeches to a halt because everyone says 'this trial design is no longer relevant in light of this new information.'”

Making the Decision

Ultimately, the decision to stop a trial early is in the hands of the safety data–monitoring committee (DMC). “This sort of data release is almost always undertaken only after a very thorough review by an independent data-monitoring committee,” Dr. Schilsky said. “DMCs are in the best position to make these decisions and they do it only after lots of careful deliberation.”

Ideally, the DMC should weigh in and make a recommendation on what should be done for the patients who are in the control arm of the study, at the very least, he said.

In the case of lenalidomide for multiple myeloma, “the study was markedly positive in favor of the Revlimid maintenance, and led our data- and safety-monitoring board to recommend early release of the results,” Dr. Schilsky noted. Accrual had been completed and all patients were in follow-up, but “the study crossed a statistical boundary for early release of the data before the protocol-specified number of events for the final analysis was available.”

This process involved notifying the Food and Drug Administration, the National Cancer Institute, and the drug manufacturer, as well as developing letters for physicians and patients involved in the study, and sending out a press release. In this case, because lenalidomide is already available, oncologists could start prescribing the drug for this patient group immediately.

Advancing technology and the development of more personalized medicine may help resolve some of the issues concerning the early stopping of trials, Dr. Crawford suggested.

“What will solve this will be biomarker-directed treatment subgroups. … We can look at a much smaller sample of patients because we're looking at a much bigger effect,” he said. “In the long run, that's going to be a much better approach to oncology than continuing to treat thousands of patients with treatment A vs. B.”

Disclosures: Both Dr. Schilsky and Dr. Obel said that they have no significant financial relationships. Dr. Crawford has received research support from and been an adviser, speaker, or consultant for several pharmaceutical companies.

'There's this risk that you put the data out there early, and it doesn't hold up over time.'

Source Dr. Schilsky

The early stopping of a clinical trial because of clear benefit from a new drug or regimen is usually good news. In the long run, however, researchers may find that they lack clear answers about the ethics and validity—and sometimes the benefit itself—of doing so.

The controversy over early stopping and early release of data stems from the desire to speed patient access to what appears to be a superior treatment, and the question of whether this will end up being done at the cost of scientific validity.

Trials involving sunitinib (Sutent) for pancreatic neuroendocrine tumors and lenalidomide (Revlimid) for multiple myeloma are two recent examples.

In February, researchers announced that a data-monitoring committee stopped the 171-patient sunitinib study after an analysis revealed that progression-free survival, the primary end point, was much longer in the active treatment arm.

In December 2009, a 568-patient study was not stopped, but early release of initial data showed that progression-free survival was longer in patients who received maintenance therapy with the oral drug lenalidomide after stem cell transplant than in patients who got a placebo.

“On the one hand, if the trial really is positive, then that's important information to get out there to the medical community and patients,” said Dr. Richard Schilsky, chairman of Cancer and Leukemia Group B (CALGB), which conducted the multiple myeloma trial. “On the other hand, there's this risk that you put the data out there early, and it doesn't hold up over time.”

When trials are stopped early for benefit, “typically the experimental arm is so much better than the standard arm that it is felt that it is medically and ethically necessary to announce the results before the trial is actually completed,” said Dr. Schilsky, former president of the American Society of Clinical Oncology and chief of hematology and oncology at the University of Chicago Medical Center.

One area of contention is the choice of end point that's used in a study's predefined early-stopping rules. Presumably, the primary end point of the study is used in the early stopping criteria, Dr. Schilsky said. In oncology trials, that often means progression-free survival.

But progression-free survival often is used as a surrogate for overall survival, said Dr. Jennifer Obel, a member of ASCO's cancer communications committee and an oncologist with NorthShore University HealthSystem in Illinois. Progression-free survival is quicker to measure, but whether it's clinically meaningful is unclear. Both the sunitinib and lenalidomide studies were stopped for benefits in disease progression.

“The real, meaningful end point is overall survival,” Dr. Obel said. Usually, patients who live longer do so without progression, but in some instances, drugs have prolonged progression-free survival but not overall survival.

Smaller P Values, Fewer Events

Another argument against early stopping is that it can overestimate the treatment benefit. In a systematic review of trials stopped early for benefit, researchers found that trials that accrued fewer events (that is, with end points that drove early stopping) estimated larger treatment effects (JAMA 2005;294:2203-9).

“The whole problem with this is that the effect size that we're looking for is relatively small,” said Dr. Jeffrey Crawford, chief of medical oncology at Duke University in Durham, N.C. “We're looking for a very small difference in survival in an unselected population. When you look for these small differences, the only way to really know is to have a large enough sample size that you have confidence in the result.”

Dr. Schilsky noted that “the stringency of the statistical analysis plan is important.” Early stopping often requires a much smaller P value to show significance. “It's a much higher threshold for stopping early because there are so many fewer events. You don't want to declare a trial to be positive unless the events are so unbalanced that you think it's extremely unlikely to be due to chance,” he said.

Weighing Drug Availability

Patients in the control arm pose yet another complication. If the experimental arm is better, patients who receive the control drug or regimen often gain access to the experimental drug or regimen. “Does that then potentially confound the final interpretation of the study results, because then you have a potentially significant number of patients who cross over?” Dr. Schilsky asked.

Early stopping because of efficacy is even more complicated when the experimental drug is not yet available. In that event, it's up to the manufacturer to develop a compassionate-use protocol specifying the patient population and how to administer the drug, he noted. Compassionate use allows the drug to be available to patients, even as the company is following the usual regulatory procedure for approval.

The effects of early stopping because of efficacy go far beyond the trial itself, with implications for ongoing and future studies of the same disease. “If you stop early and declare that you have a [new] standard of care, does that impact other ongoing studies that are not using that [new standard] of care arm, or does that impact the planning for the next generation of studies?” Dr. Schilsky asked. “You may find that the accrual to other ongoing trials screeches to a halt because everyone says 'this trial design is no longer relevant in light of this new information.'”

Making the Decision

Ultimately, the decision to stop a trial early is in the hands of the safety data–monitoring committee (DMC). “This sort of data release is almost always undertaken only after a very thorough review by an independent data-monitoring committee,” Dr. Schilsky said. “DMCs are in the best position to make these decisions and they do it only after lots of careful deliberation.”

Ideally, the DMC should weigh in and make a recommendation on what should be done for the patients who are in the control arm of the study, at the very least, he said.

In the case of lenalidomide for multiple myeloma, “the study was markedly positive in favor of the Revlimid maintenance, and led our data- and safety-monitoring board to recommend early release of the results,” Dr. Schilsky noted. Accrual had been completed and all patients were in follow-up, but “the study crossed a statistical boundary for early release of the data before the protocol-specified number of events for the final analysis was available.”

This process involved notifying the Food and Drug Administration, the National Cancer Institute, and the drug manufacturer, as well as developing letters for physicians and patients involved in the study, and sending out a press release. In this case, because lenalidomide is already available, oncologists could start prescribing the drug for this patient group immediately.

Advancing technology and the development of more personalized medicine may help resolve some of the issues concerning the early stopping of trials, Dr. Crawford suggested.

“What will solve this will be biomarker-directed treatment subgroups. … We can look at a much smaller sample of patients because we're looking at a much bigger effect,” he said. “In the long run, that's going to be a much better approach to oncology than continuing to treat thousands of patients with treatment A vs. B.”

Disclosures: Both Dr. Schilsky and Dr. Obel said that they have no significant financial relationships. Dr. Crawford has received research support from and been an adviser, speaker, or consultant for several pharmaceutical companies.

'There's this risk that you put the data out there early, and it doesn't hold up over time.'

Source Dr. Schilsky