NIH’s peer-review process is flawed, team says

Researchers in the lab

Photo by Rhoda Baer

The peer-review process the National Institutes of Health (NIH) use to allocate government research funds to US scientists may work no better than distributing those dollars at random, according to a group of researchers.

The group said their findings, published in eLife, suggest that peer review is not necessarily funding the best science, and awarding grants by lottery might actually produce equally good, if not better, results.

“The NIH claims that they are funding the best grants by the best scientists,” said study author Arturo Casadevall, MD, PhD, of the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland.

“While [our] data would argue that the NIH is funding a lot of very good science, they are also leaving a lot of very good science on the table. The government can’t afford to fund every good grant proposal, but the problems with the current system make it worse than awarding grants through a lottery.”

The researchers noted that the NIH rejects the majority of research grant proposals it receives. To decide which proposals to fund, the organization relies on expert panels whose members score each application. Funding decisions are made on the basis of these scores and the amount of available funds.

In recent years, the NIH has only funded those proposals ranked around the top 10%. The 2015 annual research budget for the NIH was $30.1 billion.

For their study, Dr Casadevall and his colleagues reanalyzed data on the 102,740 research project grants funded by the NIH from 1980 through 2008.

Another group of researchers previously collected the data. Their research, published in Science in 2015, suggested that peer review works, as the highest ranked research projects funded by the NIH earned the most citations.

For the current study, Dr Casadevall and his colleagues decided to look only at the top 20% of grants awarded. They found very little difference between the top-ranked projects and those projects ranked in the 20th percentile when it came to citations.

What the peer-review process can do, they determined, is discriminate between very good science and very bad science—that is, those in the top 20% versus those below the 50th percentile.

“We are not criticizing the peer reviewers,” said study author Ferric Fang, MD, of the University of Washington in Seattle.

“We are simply showing that there are limits to the ability of peer review to predict future productivity based on grant applications. This suggests that some of the resources and effort spent on ranking applications might be better spent elsewhere. While the average productivity of grants with better scores was somewhat higher, the differences were extremely small, raising questions as to whether the effort is worthwhile.”

The researchers noted that peer review isn’t cheap. The annual budget of the NIH Center for Scientific Review is $110 million. Individual NIH institutes and centers also spend money on peer review. The team said that money could be used toward more grants.

They also noted that peer review allows for substantial subjectivity. The objection of a single member of the committee can effectively kill a grant proposal, whether that objection is legitimate or not.

“When people’s opinions count a lot, we may be doing worse than choosing at random,” Dr Casadevall said. “A negative word at the table can often swing the debate. And this is how we allocate research funding in this country.”

However, Dr Casadevall and his colleagues do not recommend abandoning the peer-review process completely. They believe a way to improve the system might be to continue using peer review to identify the top proposals but then place those proposals into a lottery, with grants awarded at random.

Researchers in the lab

Photo by Rhoda Baer

The peer-review process the National Institutes of Health (NIH) use to allocate government research funds to US scientists may work no better than distributing those dollars at random, according to a group of researchers.

The group said their findings, published in eLife, suggest that peer review is not necessarily funding the best science, and awarding grants by lottery might actually produce equally good, if not better, results.

“The NIH claims that they are funding the best grants by the best scientists,” said study author Arturo Casadevall, MD, PhD, of the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland.

“While [our] data would argue that the NIH is funding a lot of very good science, they are also leaving a lot of very good science on the table. The government can’t afford to fund every good grant proposal, but the problems with the current system make it worse than awarding grants through a lottery.”

The researchers noted that the NIH rejects the majority of research grant proposals it receives. To decide which proposals to fund, the organization relies on expert panels whose members score each application. Funding decisions are made on the basis of these scores and the amount of available funds.

In recent years, the NIH has only funded those proposals ranked around the top 10%. The 2015 annual research budget for the NIH was $30.1 billion.

For their study, Dr Casadevall and his colleagues reanalyzed data on the 102,740 research project grants funded by the NIH from 1980 through 2008.

Another group of researchers previously collected the data. Their research, published in Science in 2015, suggested that peer review works, as the highest ranked research projects funded by the NIH earned the most citations.

For the current study, Dr Casadevall and his colleagues decided to look only at the top 20% of grants awarded. They found very little difference between the top-ranked projects and those projects ranked in the 20th percentile when it came to citations.

What the peer-review process can do, they determined, is discriminate between very good science and very bad science—that is, those in the top 20% versus those below the 50th percentile.

“We are not criticizing the peer reviewers,” said study author Ferric Fang, MD, of the University of Washington in Seattle.

“We are simply showing that there are limits to the ability of peer review to predict future productivity based on grant applications. This suggests that some of the resources and effort spent on ranking applications might be better spent elsewhere. While the average productivity of grants with better scores was somewhat higher, the differences were extremely small, raising questions as to whether the effort is worthwhile.”

The researchers noted that peer review isn’t cheap. The annual budget of the NIH Center for Scientific Review is $110 million. Individual NIH institutes and centers also spend money on peer review. The team said that money could be used toward more grants.

They also noted that peer review allows for substantial subjectivity. The objection of a single member of the committee can effectively kill a grant proposal, whether that objection is legitimate or not.

“When people’s opinions count a lot, we may be doing worse than choosing at random,” Dr Casadevall said. “A negative word at the table can often swing the debate. And this is how we allocate research funding in this country.”

However, Dr Casadevall and his colleagues do not recommend abandoning the peer-review process completely. They believe a way to improve the system might be to continue using peer review to identify the top proposals but then place those proposals into a lottery, with grants awarded at random.

Researchers in the lab

Photo by Rhoda Baer

The peer-review process the National Institutes of Health (NIH) use to allocate government research funds to US scientists may work no better than distributing those dollars at random, according to a group of researchers.

The group said their findings, published in eLife, suggest that peer review is not necessarily funding the best science, and awarding grants by lottery might actually produce equally good, if not better, results.

“The NIH claims that they are funding the best grants by the best scientists,” said study author Arturo Casadevall, MD, PhD, of the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland.

“While [our] data would argue that the NIH is funding a lot of very good science, they are also leaving a lot of very good science on the table. The government can’t afford to fund every good grant proposal, but the problems with the current system make it worse than awarding grants through a lottery.”

The researchers noted that the NIH rejects the majority of research grant proposals it receives. To decide which proposals to fund, the organization relies on expert panels whose members score each application. Funding decisions are made on the basis of these scores and the amount of available funds.

In recent years, the NIH has only funded those proposals ranked around the top 10%. The 2015 annual research budget for the NIH was $30.1 billion.

For their study, Dr Casadevall and his colleagues reanalyzed data on the 102,740 research project grants funded by the NIH from 1980 through 2008.

Another group of researchers previously collected the data. Their research, published in Science in 2015, suggested that peer review works, as the highest ranked research projects funded by the NIH earned the most citations.

For the current study, Dr Casadevall and his colleagues decided to look only at the top 20% of grants awarded. They found very little difference between the top-ranked projects and those projects ranked in the 20th percentile when it came to citations.

What the peer-review process can do, they determined, is discriminate between very good science and very bad science—that is, those in the top 20% versus those below the 50th percentile.

“We are not criticizing the peer reviewers,” said study author Ferric Fang, MD, of the University of Washington in Seattle.

“We are simply showing that there are limits to the ability of peer review to predict future productivity based on grant applications. This suggests that some of the resources and effort spent on ranking applications might be better spent elsewhere. While the average productivity of grants with better scores was somewhat higher, the differences were extremely small, raising questions as to whether the effort is worthwhile.”

The researchers noted that peer review isn’t cheap. The annual budget of the NIH Center for Scientific Review is $110 million. Individual NIH institutes and centers also spend money on peer review. The team said that money could be used toward more grants.

They also noted that peer review allows for substantial subjectivity. The objection of a single member of the committee can effectively kill a grant proposal, whether that objection is legitimate or not.

“When people’s opinions count a lot, we may be doing worse than choosing at random,” Dr Casadevall said. “A negative word at the table can often swing the debate. And this is how we allocate research funding in this country.”

However, Dr Casadevall and his colleagues do not recommend abandoning the peer-review process completely. They believe a way to improve the system might be to continue using peer review to identify the top proposals but then place those proposals into a lottery, with grants awarded at random.