Clinical Trials: Causes for Concern and Opportunities for Improvement

Fifteen years ago the FDA established a registry for clinical trials (www.clinicaltrials.gov) so that scientific investigators and the general public would have a ready source of information about medical research – planned, ongoing, and completed. In 2004 the International Committee of Medical Journal Editors (ICMJE) announced that their journals would publish the results of trials only if they had been registered.

Such a registry is of great potential value to medical scientists who wish to learn whether others are pursuing similar lines of investigation, facilitating their efforts to learn from others’ work even when it has not been published, and potentially enabling collaboration.

In the May 2, 2012, issue of JAMA, Dr. Robert M. Califf and colleagues report their analysis of trials entered in the registry, focusing on cardiovascular, oncology, and mental health research. Their findings reveal some causes for concern but also can be looked upon as opportunities for improvement.

While there was an increase in early registration of trials between the two periods of this analysis (2004-2007 and 2007-2010), fewer than half of trials are registered before enrollment of participants. This directly thwarts one of the major goals of the registry: transparency. Failure to register trials before they are underway suggests that some studies may not be registered if their findings are negative or otherwise unsatisfactory to investigators, sponsors, or sources of funding. A comprehensive registry can overcome publication bias – the tendency for studies with negative or unwelcome findings never to appear in print – only if trials are uniformly entered.

The study authors found a remarkable proportion of trials to be small and conducted at single sites (JAMA 2012;307:1838-47). This raises concerns about statistical power and external validity. A move toward larger, multicenter trials, which should be enhanced by a comprehensive registry, could effectively address both of these concerns.

In the hierarchy of evidence-based medicine, the randomized, double-blind, prospective trial is at the top of the heap, so to speak, with respect to desired characteristics for an original investigation. Yet only about a third of trials were randomized, with a similar proportion double-blinded. While it is very difficult to conduct some studies with randomization and blinding, the failure to do so introduces major sources of potential bias and confounding, and statistical manipulations can never completely adjust for these. Such disappointing numbers raise serious doubts about the quality of scientific investigation.

Sponsorship is a significant concern, as its influence on study design, interpretation of results, and likelihood of publication if hoped-for outcomes are not obtained has been long established. The numbers in this analysis indicate that nearly 4 in 10 registered trials had industry sponsorship. That a major source of potential bias should be present in such a large proportion of investigations is troubling.

At a time when the Obama administration is emphasizing the need for a robust approach to comparative effectiveness research so that doctors and patients can make the best decisions and assure a high quality of health care, it is disconcerting – to say the least – that only 4% of clinical trials had the National Institutes of Health or other federal agencies listed as lead sponsors. Public funding of this research must expand dramatically.

Finally, from a health policy perspective, it is disappointing to note than only about 10% of trials were focused on prevention, and only about 2% in the area of health services. If we are to make our health care system more effective and efficient, this most certainly represents an opportunity to do better by doing more.

Dr. Solomon is core faculty in the emergency medicine residency at Allegheny General Hospital, Pittsburgh, and assistant professor in the department of emergency medicine at Temple University, Philadelphia. He is also Medical Editor of ACEP News, a publication of Elsevier.

Fifteen years ago the FDA established a registry for clinical trials (www.clinicaltrials.gov) so that scientific investigators and the general public would have a ready source of information about medical research – planned, ongoing, and completed. In 2004 the International Committee of Medical Journal Editors (ICMJE) announced that their journals would publish the results of trials only if they had been registered.

Such a registry is of great potential value to medical scientists who wish to learn whether others are pursuing similar lines of investigation, facilitating their efforts to learn from others’ work even when it has not been published, and potentially enabling collaboration.

In the May 2, 2012, issue of JAMA, Dr. Robert M. Califf and colleagues report their analysis of trials entered in the registry, focusing on cardiovascular, oncology, and mental health research. Their findings reveal some causes for concern but also can be looked upon as opportunities for improvement.

While there was an increase in early registration of trials between the two periods of this analysis (2004-2007 and 2007-2010), fewer than half of trials are registered before enrollment of participants. This directly thwarts one of the major goals of the registry: transparency. Failure to register trials before they are underway suggests that some studies may not be registered if their findings are negative or otherwise unsatisfactory to investigators, sponsors, or sources of funding. A comprehensive registry can overcome publication bias – the tendency for studies with negative or unwelcome findings never to appear in print – only if trials are uniformly entered.

The study authors found a remarkable proportion of trials to be small and conducted at single sites (JAMA 2012;307:1838-47). This raises concerns about statistical power and external validity. A move toward larger, multicenter trials, which should be enhanced by a comprehensive registry, could effectively address both of these concerns.

In the hierarchy of evidence-based medicine, the randomized, double-blind, prospective trial is at the top of the heap, so to speak, with respect to desired characteristics for an original investigation. Yet only about a third of trials were randomized, with a similar proportion double-blinded. While it is very difficult to conduct some studies with randomization and blinding, the failure to do so introduces major sources of potential bias and confounding, and statistical manipulations can never completely adjust for these. Such disappointing numbers raise serious doubts about the quality of scientific investigation.

Sponsorship is a significant concern, as its influence on study design, interpretation of results, and likelihood of publication if hoped-for outcomes are not obtained has been long established. The numbers in this analysis indicate that nearly 4 in 10 registered trials had industry sponsorship. That a major source of potential bias should be present in such a large proportion of investigations is troubling.

At a time when the Obama administration is emphasizing the need for a robust approach to comparative effectiveness research so that doctors and patients can make the best decisions and assure a high quality of health care, it is disconcerting – to say the least – that only 4% of clinical trials had the National Institutes of Health or other federal agencies listed as lead sponsors. Public funding of this research must expand dramatically.

Finally, from a health policy perspective, it is disappointing to note than only about 10% of trials were focused on prevention, and only about 2% in the area of health services. If we are to make our health care system more effective and efficient, this most certainly represents an opportunity to do better by doing more.

Dr. Solomon is core faculty in the emergency medicine residency at Allegheny General Hospital, Pittsburgh, and assistant professor in the department of emergency medicine at Temple University, Philadelphia. He is also Medical Editor of ACEP News, a publication of Elsevier.

Fifteen years ago the FDA established a registry for clinical trials (www.clinicaltrials.gov) so that scientific investigators and the general public would have a ready source of information about medical research – planned, ongoing, and completed. In 2004 the International Committee of Medical Journal Editors (ICMJE) announced that their journals would publish the results of trials only if they had been registered.

Such a registry is of great potential value to medical scientists who wish to learn whether others are pursuing similar lines of investigation, facilitating their efforts to learn from others’ work even when it has not been published, and potentially enabling collaboration.

In the May 2, 2012, issue of JAMA, Dr. Robert M. Califf and colleagues report their analysis of trials entered in the registry, focusing on cardiovascular, oncology, and mental health research. Their findings reveal some causes for concern but also can be looked upon as opportunities for improvement.

While there was an increase in early registration of trials between the two periods of this analysis (2004-2007 and 2007-2010), fewer than half of trials are registered before enrollment of participants. This directly thwarts one of the major goals of the registry: transparency. Failure to register trials before they are underway suggests that some studies may not be registered if their findings are negative or otherwise unsatisfactory to investigators, sponsors, or sources of funding. A comprehensive registry can overcome publication bias – the tendency for studies with negative or unwelcome findings never to appear in print – only if trials are uniformly entered.

The study authors found a remarkable proportion of trials to be small and conducted at single sites (JAMA 2012;307:1838-47). This raises concerns about statistical power and external validity. A move toward larger, multicenter trials, which should be enhanced by a comprehensive registry, could effectively address both of these concerns.

In the hierarchy of evidence-based medicine, the randomized, double-blind, prospective trial is at the top of the heap, so to speak, with respect to desired characteristics for an original investigation. Yet only about a third of trials were randomized, with a similar proportion double-blinded. While it is very difficult to conduct some studies with randomization and blinding, the failure to do so introduces major sources of potential bias and confounding, and statistical manipulations can never completely adjust for these. Such disappointing numbers raise serious doubts about the quality of scientific investigation.

Sponsorship is a significant concern, as its influence on study design, interpretation of results, and likelihood of publication if hoped-for outcomes are not obtained has been long established. The numbers in this analysis indicate that nearly 4 in 10 registered trials had industry sponsorship. That a major source of potential bias should be present in such a large proportion of investigations is troubling.

At a time when the Obama administration is emphasizing the need for a robust approach to comparative effectiveness research so that doctors and patients can make the best decisions and assure a high quality of health care, it is disconcerting – to say the least – that only 4% of clinical trials had the National Institutes of Health or other federal agencies listed as lead sponsors. Public funding of this research must expand dramatically.

Finally, from a health policy perspective, it is disappointing to note than only about 10% of trials were focused on prevention, and only about 2% in the area of health services. If we are to make our health care system more effective and efficient, this most certainly represents an opportunity to do better by doing more.

Dr. Solomon is core faculty in the emergency medicine residency at Allegheny General Hospital, Pittsburgh, and assistant professor in the department of emergency medicine at Temple University, Philadelphia. He is also Medical Editor of ACEP News, a publication of Elsevier.