While the transition to the new shortened grant applications at the National Institutes of Health (NIH) and the corresponding review guidelines hasn't been completely smooth, reviewers who have participated in the first few rounds of funding under the new system generally support the changes.

Image: Wikimedia commons

"I think it's an improvement over the old system," said Karin Rodland, a researcher at the Pacific Northwest National Laboratory (PNNL) in Washington State and an NIH reviewer since 1998, "but I think there is a learning curve and until everyone recalibrates there may be a period of confusion."

Reviewers still assess applications on the same five criteria (significance, investigator, innovation, approach, and environment), but the final judgment of a grant proposal is based on an "overall impact" rating that is related to, but not completely dictated by, those subscores. "I think that's where most of the confusion comes from for most people," said Tony Hazbun of Purdue University in Indiana, who experienced the new system in his first ever study section.

Because there is no algorithm for calculating the overall impact score, "different people will weight different components differently," Rodland said. Under the old system, the subcriteria weren't rated, so there was only one score -- the priority score -- assigned to an application. Furthermore, one of the subcriteria is "significance," which some reviewers found difficult to distinguish from "overall impact," said Steven Wiley, lead biologist for the Environmental Molecular Sciences Laboratory at PNNL and a member of The Scientist's editorial board. But in fact, the NIH intends the two to be quite different, Wiley said. Whereas "significance" refers to the importance of a project if every specific aim were completed successfully, "overall impact" is weighted by the likelihood of success, which will be influenced by the other subcriteria, such as approach, investigator, or environment. "The impact is basically [when you] take the significance and then say, 'Can they actually do it, and are they the right people to do it?'" Wiley explained.

"Everything is kind of tied together and it depends on each case," Hazbun said. "You have to look at the whole application." Thus, even if a grant proposal has a really high significance score, if it is unlikely to be successful, either because the investigator is ill-equipped to complete the study or his or her proposed approach seems unfeasible, the impact would actually get a much lower score.

Wiley noted that the NIH has made an effort to clarify these points, providing several examples of how specific grant applications should be scored, which has helped lift some of the confusion. "I've been on three different study sections using the new review criteria. The first time through was a little confusing. The second time through it was a little better. The third time I think we got it."

Apart from such issues, reviewers say, the changes to the reviewing guidelines have actually increased the validity and utility of the reviews. For example, in the new system scoring is limited to whole numbers (1 through 9), whereas before, a reviewer could give a proposal a priority score anywhere from 1 to 5 in increments of tenths. But such a fine scale was counterproductive, Wiley noted. "[Y]ou cannot possibly discriminate grants on that kind of level," he said, adding that the process was effectively "a crapshoot after you pick the top 25%."

While some may find this adjustment difficult, overall "it forces [reviewers] more to just look at the score [matrix]" and pick a number that is most appropriate, Hazbun said. In addition, Wiley added, this new system eliminates the "priority score games" that reviewers could play, bumping up scores slightly to increase the likelihood of funding. With the new 9 point scale, Wiley said, "there's no room to play games."

Another benefit of the new system is that reviewers are now required to justify their scores by listing strengths and weaknesses for each subcriterion, Wiley said. And if there are no weaknesses to name, that category must get a 1 -- the highest score possible. "You can't just say the environment is weak; you have to say why and be very specific about it," said Hazbun. "What they're trying to do is really tie in evaluative comments that are going to be constructive for the investigator." This specific feedback can help investigators improve unfunded applications in the next round, he said, adding that the process also helps him make his judgments as a reviewer.

Other changes to the review process specifically aim to cut down on the amount of time the process takes. For example, the written evaluations no longer include -- indeed, specifically exclude -- a written summary of the grant proposal. "To me, that was a waste of time and a waste of paper," Rodland said. The template now provided by the NIH gives bullet points where reviewers are to write a couple of sentences summarizing the strengths and weakness of each category, limiting them to just half a page. "[Some] people used to write 3-page reviews," Rodland said. But the new system and template "encourages succinctness," which is a good thing, she added.

Finally, rather than reviewing the grant proposals in random order, the study section starts with the highest scored applications (based on preliminary scores) and works their way down the list. In addition to cutting the total number of grants the study section will review orally -- low-ranked applications with no chance of getting funded won't even be discussed -- it also helps the reviewers to "recalibrate" their scores, Wiley said, by providing an excellent standard against which the others can be judged. "It was a very clever idea," he said. "I found this has been very, very helpful."

Overall, "I was impressed with the process," Hazbun said of his grant reviewing experience. "It seemed to be working fairly well for me."


Related stories:

New NIH forms raise concerns
[8th December 2009]

How to change NIH peer review?
[12th December 2007]

A New Paradigm for NIH Grants
[August 2007]