User login
Photo by Aaron Logan
A study of animal-based research published over the last 70 years suggests that scientists could have done more to reduce the risk of bias.
A group of researchers examined a few thousand published studies conducted in animals and found that, most of the time, scientists did not employ 4 measures believed to reduce the risk of bias: randomization, blinded assessment of outcome, a conflict of interest statement, and sample size calculation.
“I don’t believe for a moment that scientists set out to do anything other than excellent research, but what this work shows is that there is considerable room for improvement,” said Malcolm Macleod, PhD, of the University of Edinburgh in the UK.
Dr Macleod and his colleagues conducted this research and reported their findings in PLOS Biology.
The team first looked at a random sample of 146 in vivo studies published between 1941 and 2012. Randomization was reported in 20% of the publications in which it would have been appropriate.
Three percent of the publications reported blinding, 10% reported a conflict of interest statement, and none reported a sample size calculation.
Next, the researchers looked at 2671 studies published between 1992 and 2012 that reported drug efficacy in 8 disease models. Randomization was reported in 25% of publications, blinding in 30%, sample size calculation in 0.7%, and a statement of potential conflict of interest in 12%.
However, there were significant increases in 3 of these measures over time. The use of randomization increased from 14% in 1992 to 42% in 2011 (P<0.001), blinding increased from 16% to 39% (P<0.001), and a statement of possible conflict of interest increased from 2% to 35% (P<0.001).
The reporting of a sample size calculation did not change significantly and actually decreased from 2% to 1%.
Dr Macleod and his colleagues also used this dataset to determine whether a journal’s impact factor (a commonly used but disputed measure of journal “quality”)
played a role in the use of the 4 risk-of-bias measures.
The median journal impact factor was significantly higher (P<0.001) for studies reporting a potential conflict of interest, but it was significantly lower in studies reporting randomization (P=0.001). There was no significant difference for the other 2 measures.
Finally, the researchers examined bias in animal studies from the UK’s top 5 universities (according to the
2008 Research Assessment Exercise).
Of 1028 studies published in 2009 and 2010, 14% reported randomization, 17% reported blinding, 10% reported inclusion or exclusion criteria or both, and 1% reported a sample size calculation. Only 1 publication reported using all 4 of these measures.
Dr Macleod and his colleagues concluded that, although this study had its limitations, the results suggest room for improvement in in vivo research. 
Photo by Aaron Logan
A study of animal-based research published over the last 70 years suggests that scientists could have done more to reduce the risk of bias.
A group of researchers examined a few thousand published studies conducted in animals and found that, most of the time, scientists did not employ 4 measures believed to reduce the risk of bias: randomization, blinded assessment of outcome, a conflict of interest statement, and sample size calculation.
“I don’t believe for a moment that scientists set out to do anything other than excellent research, but what this work shows is that there is considerable room for improvement,” said Malcolm Macleod, PhD, of the University of Edinburgh in the UK.
Dr Macleod and his colleagues conducted this research and reported their findings in PLOS Biology.
The team first looked at a random sample of 146 in vivo studies published between 1941 and 2012. Randomization was reported in 20% of the publications in which it would have been appropriate.
Three percent of the publications reported blinding, 10% reported a conflict of interest statement, and none reported a sample size calculation.
Next, the researchers looked at 2671 studies published between 1992 and 2012 that reported drug efficacy in 8 disease models. Randomization was reported in 25% of publications, blinding in 30%, sample size calculation in 0.7%, and a statement of potential conflict of interest in 12%.
However, there were significant increases in 3 of these measures over time. The use of randomization increased from 14% in 1992 to 42% in 2011 (P<0.001), blinding increased from 16% to 39% (P<0.001), and a statement of possible conflict of interest increased from 2% to 35% (P<0.001).
The reporting of a sample size calculation did not change significantly and actually decreased from 2% to 1%.
Dr Macleod and his colleagues also used this dataset to determine whether a journal’s impact factor (a commonly used but disputed measure of journal “quality”)
played a role in the use of the 4 risk-of-bias measures.
The median journal impact factor was significantly higher (P<0.001) for studies reporting a potential conflict of interest, but it was significantly lower in studies reporting randomization (P=0.001). There was no significant difference for the other 2 measures.
Finally, the researchers examined bias in animal studies from the UK’s top 5 universities (according to the
2008 Research Assessment Exercise).
Of 1028 studies published in 2009 and 2010, 14% reported randomization, 17% reported blinding, 10% reported inclusion or exclusion criteria or both, and 1% reported a sample size calculation. Only 1 publication reported using all 4 of these measures.
Dr Macleod and his colleagues concluded that, although this study had its limitations, the results suggest room for improvement in in vivo research.
Photo by Aaron Logan
A study of animal-based research published over the last 70 years suggests that scientists could have done more to reduce the risk of bias.
A group of researchers examined a few thousand published studies conducted in animals and found that, most of the time, scientists did not employ 4 measures believed to reduce the risk of bias: randomization, blinded assessment of outcome, a conflict of interest statement, and sample size calculation.
“I don’t believe for a moment that scientists set out to do anything other than excellent research, but what this work shows is that there is considerable room for improvement,” said Malcolm Macleod, PhD, of the University of Edinburgh in the UK.
Dr Macleod and his colleagues conducted this research and reported their findings in PLOS Biology.
The team first looked at a random sample of 146 in vivo studies published between 1941 and 2012. Randomization was reported in 20% of the publications in which it would have been appropriate.
Three percent of the publications reported blinding, 10% reported a conflict of interest statement, and none reported a sample size calculation.
Next, the researchers looked at 2671 studies published between 1992 and 2012 that reported drug efficacy in 8 disease models. Randomization was reported in 25% of publications, blinding in 30%, sample size calculation in 0.7%, and a statement of potential conflict of interest in 12%.
However, there were significant increases in 3 of these measures over time. The use of randomization increased from 14% in 1992 to 42% in 2011 (P<0.001), blinding increased from 16% to 39% (P<0.001), and a statement of possible conflict of interest increased from 2% to 35% (P<0.001).
The reporting of a sample size calculation did not change significantly and actually decreased from 2% to 1%.
Dr Macleod and his colleagues also used this dataset to determine whether a journal’s impact factor (a commonly used but disputed measure of journal “quality”)
played a role in the use of the 4 risk-of-bias measures.
The median journal impact factor was significantly higher (P<0.001) for studies reporting a potential conflict of interest, but it was significantly lower in studies reporting randomization (P=0.001). There was no significant difference for the other 2 measures.
Finally, the researchers examined bias in animal studies from the UK’s top 5 universities (according to the
2008 Research Assessment Exercise).
Of 1028 studies published in 2009 and 2010, 14% reported randomization, 17% reported blinding, 10% reported inclusion or exclusion criteria or both, and 1% reported a sample size calculation. Only 1 publication reported using all 4 of these measures.
Dr Macleod and his colleagues concluded that, although this study had its limitations, the results suggest room for improvement in in vivo research.