How to change NIH peer review?

Instead of giving generous funding of numerous and/or expensive projects to the most popular scientists (yes, politics does play a role in funding decisions) or those with the most popular research ideas (not always the best or most innovative research) please fund the greatest number of researchers and greatest number of meritorious independent research ideas. In times of shrinking funding the NIH plays it too conservatively by funding mainly/only scientists with many other current grants and with conventional research ideas. For example, one colleague recently had 20 R01's funded simultaneously (each at 0.05 FTE), each applying the same (politically popular) research technique to a slightly different topic. At the opposite extreme, a Nobel laureate was denied all NIH funding for four years straight, and this was for the very work for which he later received the Nobel prize!

This is a very unfair idea. It leaves new researchers at a clear disadvantage and so will decrease the diversity of research taking place.

In my 51-year-long well-established clinical experience (See Presentation in www.semeioticabiofisica.it), among a lot of other problems in caring for patients ther's one which plays the central role, so that, solving it, physicians will avoid both losing time in day-to-day practice, and spending uselessly money, to better utilize in other medical undertakings. I mean the ability to bedside recognize all Biophysical-Semeiotic Constitutions, including Oncological Terrain, and their related Inherited Real Risk, with the aid of a simple stethoscope (ibidem; Ask Google, and read my comments on this website). For instance, what does it mean in individuals not involved by congenital predisposition to MALIGNANCY (i. e., Oncological Terrain) prescribe numerous and expensive Laboratory and image semeiotic investigations, NOT to speak of biopsies, to exclude cancer? In addition, today's Medicine brings about a lot of "psychological terrorism", one can avoid IF WE WILL it!
Certainly, to realize such as epochal medical revolution, it's unavoidable necessary THE FREEDOM of scientists, medical Authorities, peer-reviews Editors, as well as Intellectual honesty of Reviewers, as I am.

There is an enormous waste of NIH resources when very senior P.I.s have multiple R01 and P01 awards. Here is a useful calculation. I'm open to suggestion on its assumptions.
Numerator = number of publications in a given year. Demoninator = (any of the below variables for the three years prior to the publication year; this makes an assumption that it took about two years to complete the project and nearly a year publish the results of a project)
1)number of post-docs per lab
2)number of grants per lab
3)number of total personnel per lab.

Using any of the above demominators and doing this calculation yields similar results: smaller labs are far more efficient than larger labs on a resources per paper basis. Go ahead and do it yourself. Just don't tell the administrators at your institution, who may favor the inefficiency of large labs because it brings in more overhead!
If a single PI can have only one or two R01 grants at most, this would go a long way toward distributing NIH funds to more investigators and toward making sure the funds are used efficiently. Further, it will stop the endless grant writing that is taking so many people away from creative scientific work.

I am pleased to see that NIH is working on improving the peer review process. Many of the suggested changes are worthwhile but I have some concerns. First, I don't believe that adding funds to reviewer grants as an incentive is the best approach to retaining peer reviewers - the other approaches being considered are preferrable. This approach would require an allocation of funds on an already strained budget and continue to favor established researchers over new investigators. The age at which grants are first awarded is concerning and I think measures to increase the number of awards to new investigators is very much needed. Academic burn out is a well known phenomenon. Favoring established researchers over new investigators would seem to increase the likelihood that younger investigators will become discouraged and stop trying, thereby decreasing the number of investigators long term.

With regard to the suggestion that researchers have the opportunity to talk with reviewers about their proposals - I think some form of dialogue or approach to offering very specific suggestions for improvement would benefit the scientific community. Investigators often learn in the process of writing proposals. Having a dialogue with reviewers would enhance this learning.

Another suggestion has to do with strengthening the match between reviewers and the proposals reviewed in terms of topic and methods. This is particulary problematic for qualitative proposals.

Please, let's do limit the number of grants a PI can have at any one time. I hate to say this as a PI and scientist, but there are many large labs that I've seen (and I've been in four different academic institutions during my career)where the PI is so caught up in grant-writing that all the details of experiments and mentoring are left to post-docs and very junior faculty.

As such, we are losing the wealth of experience that can be transmitted, and creating a class of scientists that know ever-more esoteric details about their tiny portion of the scientific universe, but really don't know why they use a lot of what they use for their experiments. They have lost sight of fundamentals that are necessary for a real understanding of the "big picture", which is what should be controlling scientific vision.

We have created a system where to get a grant, you have to prove that you can do everything you proposed (and have most of the experiments underway) or your chances of getting funded are that of a snowball in hell. The reality is that in a new area of research, you will have to, of necessity, engage in the much-maligned and ridiculed "fishing expedition" approach some of the time. That is because while you can have good predictive ideas, if you are right all of the time, you are probably not asking the important and difficult questions.

To quote Wernher Von Braun, "Research is what I'm doing when I don't know what I'm doing". Let's hope that reform will take place so that each scientist can make their best contributions, as made possible by their unique perspective, background, and training.

"People vs. projects" by anonymous poster [Comment posted 2007-12-12 14:16:54] states "I would have to disagree to focus on people rather than projects. Where would that leave new PIs who are yet to prove their track record?... There are a lot of postdoctoral fellows who are discouraged by the funding situation and are leaving academia..."

My proposal (NIH Needs to Thinks Out of the Box) is not just for scientists with a track record but also for Post-Doctoral Fellows starting their first independent positions, albeit at $50,000/yr rather than $300,000/yr. I would think we would have a lot more enthusiastic, innovative, and happy young scientists (to be precise, 40,000 of them) if they were already guaranteed 5 years of funding.

I heartily agree with Michael Duchen?s comments on ?Chance to reply?. I?d like to add that many reviewers have admitted to me that they are often reviewing at sub-optimal times (late night, when they are least alert), with distractions (not to mention booze!), and if there is anything that they can?t fully grasp you are ?demoted?, often to the bottom of the pile. By adding more reviewers (give them modest, but real incentives) they will also benefit from more time to run their own laboratories and this problem would be lessened.

Please, let?s also move away from ?techniques du jour?, penalizing those who see other ways of arriving at answers. Technology should be used as appropriate; so many times, senior colleagues tell junior researchers ?You won?t get your grant funded unless the reviewers see that you are capable of applying XYZ technique(s) to your area of research?. Yes, let?s harness the power of the latest innovative technologies, but it should not be an unspoken requirement to be favorably considered.

A modest proposal: Shouldn?t we also have a portion of NIH $$ spent on a type of grant similar to ?contrarian investing? (to borrow a phrase from Wall Street)? That would effectively allow much more innovation than what are currently considered ?high risk? funding mechanisms.

A real stumper is what you are supposed to do when reviewers suggest experiments / revisions, you do them, re-submit, and then one or more have rotated off? The new reviewer(s) disagree, suggest their own experiments, and you are stuck wasting precious time and scarce $$, benefiting no one. This is a point I?d like to see commented on by more researchers. There isn?t any one fix, but surely the problem can be minimized somehow ? so as not to disadvantage new researchers or researchers who have changed tracks; this latter category many times has much experience, better perspective and a lot of potential, if they weren?t being judged by the members of the ?club? in each discipline, as is so often the case. Also, individuals from smaller, less-known labs (contrary to popular hype and assertions by the cognoscenti) with a high degree of drive and creativity can excel, if the current climate did not penalize them for their scientific ancestry. Potential is a great thing, if nurtured.

On (Anonymous), >> I would have to disagree to focus on people rather than projects. Where would that leave new PIs who are yet to prove their track record? I think that innovation and applicability to human health should be the main criteria for funding, followed by previous productivity (publications, preliminary data and such).<< I say hear, hear!! Hope the reformers are listening!!

Oh, and cut the overhead to 30%, as previously suggested in another post?..that is enough,(for now); this would really help to stretch our strained system, and lessen the incentive for greedy administrators to demand researchers submit proposals before they are in the most advantageous position to do so. A portion of the overhead returns should also be required to be re-invested (by each funded institution); this mechanism would be audited for compliance, and will generate modest but essential funds whose sole purpose will be to tide investigators over between grants so their labs don?t have to fold and they can remain productive.

A reality check that SHOULD strongly encourage reform is that now the average age of award for a first RO1 is early forties. After all those years of highly specialized training (many times at taxpayer expense), it is in everyone?s interest to make sure that those individuals can make their best contributions to science. After all, we are finite, and so is our time to contribute to the betterment of mankind?s health.

In my opinion, peer-reviewers could solve problem singularly. I mean that, nowadays, when reviewers read papers written according to new, original paradigms and frame-work, surely reject them, although they are described in details with operative illustration.On the contrary, it proved to be usefull, the dialogue on methods,routine for authors BUT ignored by reviewrs. For instance, Who all around the world is able to examine coscientiously and convincenly an article like "Quantistic Biophysical Semeiotics: non-local Realm in biological Systems" posted at URL:LINK ?

The system will always be highly subjective, and therefore cannot be entirely corrected. The ONLY way the system can work is when substantial funding exists from the government. An attempt to fix the system in other ways is foolishness.

The review comments and score should be done individually by the Primary Reviewers, before a group discussion and composite score.

In the selection of the Primary Reviewers, consider the candidate reviewer's history of scores on proposals from the proposal source lab or P.I.

It is all too frequent that reviewers comments reflect misunderstandings. Sometimes that is the fault of the applicant, failing to make points clear, but it is also an inevitable function of reviewer overload - as reviewers we are all overburdened with work and so it is hard to give as much time as needed to these tasks. I believe it is essential to allow applicants the opportunity to respond to reviewer comments before the panel meetings. I suspect that this will avoid many resubmissions and will make the process fairer.

A major problem lies in PI's asking post docs to review papers/grants. Most post docs take this as a chance to demonstrate how clever they are to their bosses. Their agenda is therefore inappropriate - they necessarily and inevitably lack the generosity of spirit needed to identify potentially good science and to support it. I would suggest that PI's are specifically requested not to have papers or grants reviewed by others in their groups except with permission from the agency. If post docs are reviewing, this MUST be noted on the forms and the PI has an obligation to check the comments (and so should sign a declaration to that effect).

14 Dec 2007

Personally, I feel the biggest problem with NIH?s peer review system is the dismal pay line. For example, the current payline at NIAID for an RO1 grant application is ~10% with a 14% payline for a new investigator. I speak from considerable experience as one who has submitted many grants and reviewed many grants in the last 25 years. It is virtually impossible to fund a highly innovative RO1 research proposal today because in most instances those proposals tend to be higher risk. Most of the major discoveries in the last two decades have come from NIH?s RO1 grants. I therefore strongly recommend that a higher percentage of NIH?s total budget be placed in its RO1 grant program.

Despite the above problem, I agree with the general assessment from the scientific community that NIH is not getting the best scientists and clinicians to review its proposals. The debate is how to fix this problem. For me, there is an easy and fair solution. Simply place the NIH Study Section member in the New Investigator category thereby giving him/her a reasonable incentive to join a Study Section. NIH already has used this rewarding system to try to correct a perceived weakness/problem (e.g., how do we get more young investigators?). In addition, NIH has already decided on how much reward is appropriate. Thus, no one could complain about my suggested incentive and its level. However, to get the bonus percentage points (namely 4 percentage points at NIAID for an RO1 grant), I recommend that the reviewer must agree to become a permanent member of a Study Section. The bonus points should not go to an adhoc member unless NIH still is unable to induce America?s top scientists and clinicians to become permanent members of its Study Sections.

Finally, it must be pointed out that as the situation currently stands, one is severely penalized for sitting on a Study Section. The first problem with the peer review system is that the investigator?s own RO1 grants must be evaluated by a Special Emphasis Panel. It is much harder to obtain a fundable score from a Special Emphasis Panel because there are very few weak grants when the latter Panel meets due the fact that nearly every grant being evaluated originates from a strong Study Section member. Because the funding rate for RO1 grants is ~10%, ~90% of all submissions will be turned down. The second problem relates the anger the Study Section member experiences from many in the scientific community. Due to human nature, it is very difficult for a PI to accept that his/her grant is not fundable. Thus, the PI takes his/her frustration out on the perceived Study Section member who evaluated the PI?s grant rather than NIH and its policy of placing less money in its RO1 program. Who wants to be hated especially if your intent is simply to fund the best work. The third problem is the tremendous amount of time it takes to evaluate the large number of grants every time a Study Section meets. The bottom line is that one has to be crazy to become a permanent member of a Study Section right now. Doesn?t NIH realize that the current situation strongly encourages this country?s top scientists and clinicians from not joining its Study Sections?

Sincerely,
Professor Dept. Med.; Div. Rheum., Immunol., & Allergy
Brigham and Women?s Hospital
Smith Building, Rm. 616B
1 Jimmy Fund Way
Boston, MA 02115-6007
Telephone = 617-525-1231
FAX = 617-525-1310
E-mail = rstevens@rics.bwh.harvard.edu

I think that the changes listed would be great. This will make the US funding system focus more on new areas and less on bureaucracy. I really like the idea of making the grants shorter and more focused on the scientist's track record rather than the project's track record. This could shift science to more innovations and bold explorations.
Many of the ideas for change are more like funding schemes in the UK. Although little is spent on research in the UK (1.8% of GDP, compared to 6.6% in the US), it is very efficient in its funding; ranking fifth internationally on relative prominence of cited scientific literature and government-funded patents, where the US ranks 2nd and 7th, respectively (see www.oecd.org/sti/scoreboard).
I would be thrilled to see the changes!

The fundamental problem with peer review at the NIH is that much of the time the word "peer" is misplaced.

There are undoubtedly many NIH referees who are diligent and bright. However the system has been infiltrated and gamed by swarms of professional grantwriters who do low caliber research and who are poor critics.

They have learned the secret handshakes, they feed each other across the committee tables, and they eviscerate innovative work that they do not understand.

The solution is to pay good scientists, by the hour, for their time in reviewing proposals.

This idea has already been run past Zerhouni's No. 2, who thought it would be a horriying and vulgar disruption of the NIH "culture."

Zerhouni's office is where the problem is.

Unfortunately, too many reviewers at the NIH do not always recuse themselves if a grant is from a colleague whose work is competing with theirs. In addition, some reviewers also use the NIH peer review process as a way to keep colleagues they do not like or whose work they do not like from getting grants.

The process needs to be amended in a way that the reviewer is not aware of who the grant is coming from and be able to give it a fairer rating that is not based on anything subjective.

It appears that all the points NIH proposes are still within the old system. Putting an organism on diet does not change its habitual mannerisms. My suggestion is to reflect whether there are better systems which could replace ?peer reviews?, and I am sure there are. One just has to glance over to ?Silicon Valley? where peer review is practised in a completely different manner. Moreover, there is plenty of research money to be had, but the way it?s collected and distributed is most inefficient. As long as ?club mentality? continues to be practised in academia plenty of human intellectual and innovative resources will be lost.
Gelett Burgess rhetorical skills come to mind on this issue:

A donkey with two bales of hay,
So does the fable run,
Rizgidgeted the livelong day,
Deciding on "which one?"

So, with this stupid brain that's stirred
By sluggish fuss and fidget,
Deciding what to name that word
Do I delay - rizgidget.

American English has such a wonderful way with words. Small people are given attributes such as ?vertically challenged?, low I.Q produces ?mentally challenged? individuals, and so I would designate today?s funding practises as ?academically challenged?.
Recommended reading:
Donald. W. Braben (2004) Pioneering Research-A risk worth taking. Wiley and Sons, New Jersey.

Ilse M. Zalaman
Clinical Psychologist
ENT-University Hospital Tbingen
Elfriede-Aulhorn-Str. 5
72070 Tbingen
Germany
E-mail:isa.zalaman@med.uni-tuebingen.de

Giving the lion's share to persons instead of projects is just another word for seniority. (AKA Rank Has it's Privileges) That will only exacerbate the control that certain older scientists (not a few of whom are burnt out and haven't worked in a lab in a decade) have over younger ones. That will, in turn, worsen the problem of seniors taking credit for what the new ones do, casting in stone the least functional part of our scientific system. Note that science is a discipline in which, for most, real discovery productivity peaks between 27 and 35.

There is one thing nobody has suggested that I think is paramount in importance. We need to establish an office of audit and do what the IRS does for taxes to monitor compliance, auditing a statistically valid percentage of publications and grants selected each year for the purpose of quality control. If it once was true that becoming a research scientist was admittance into a circle of trust that was legitimate, this is true no longer. While our system of presuming mutual trust is a pleasant social fiction, it must also be acknowledged that this fiction makes us prey to those who abuse trust.

I know very well that if such auditing were actually done thoroughly, that a number of names that are riding fairly high would not be in science after they were selected.

The question of who should do such audits and what should they look for is valid. However, the DOI already has such. Simply apply those same criteria and go over everything with their standard fine tooth comb.

Really, everyone. Think about this. Would you trust a bank that was not audited? Would you trust the White House to "do the right thing" without the 4th Estate of the press? Would you trust congressional budgets to get spent as advertised without the GAO?

If we created an office of general scientific audit, we could also use the results to help us with our criteria for grant allocation.

Another point I think needs to be made is that with a science audit office, we will be able to make modifications on a sound scientific basis instead of on the basis of academic politics as we are now.

In my opinion Peer Review is just a symptom of a much bigger problem. I believe there would be more monies allocated to NIH by congress if we could demonstrate the power of the Human Genome Project to prevent and cure diseases, as expected in the funding of the HGP and the doubling of the NIH budget. The challenge was to effectively translate genes into phenotypes and visa versa. We do not currently have such a paradigm- pathophysiology worked in the past because we were limited to our superficial knowledge of phenotypes of health and disease. In the post-genomic era we must use the broadest perspective, that of adaptation and maladaptation in order to maximize our use of the genomes for humans, mice, frogs, worms, yeast. Once we make that Kuhnian paradigm shift the utility of genomics will become readily apparent. I think we're missing the 'big picture' and we're suffering as a result.

John S. Torday, MSc,PhD
Professor,
Department of Pediatrics and
Obstetrics and Gynecology
Director, The Henry L. Guenther Laboratory
for Cell/Molecular Research
Director,Laboratory for Evolutionary Preventive Medicine
Harbor-UCLA Medical Center
Phone: (310)222-8186
FAX: (310)222-3887

I do not believe that the peer review process is broken, all the problems that have been brought to the fore have resulted from the reduction in the NIH budget that has occurred under the current administration.

My own experience consists of 5 years on a study section (98-2003) and continual activity in various SEPs and site visit groups before and after. I am very concerned at some of the suggestions that are listed in the article. First and foremost, we need to maintain face to face reviews. As soon as we lose that aspect of accountability in our reviews, I think we are in a situation where the peer-review process has broken down. I find it hard to accept that we are not willing to spend several days a year away from our labs and office to review grants. We owe this to all our colleagues, and remember how much we all learn form the review process ourselves.

I do not see how a 2-level review process will benefit applicants. The stusy section should be the body that recommends funding, not another group above that with their own agenda.

Reducing the size of the application to 7 pages is a dumb idea. How will this help the reviewers to make decisions? Moreover, this will hurt the new investigators very hard, as expeirenced grant writers will likely be better at presenting in a redcued format. This to me is the worst idea proposed. The 25 page limit is not the problem, the funding reductions are.

Since scientific knowledge is advancing at a fast pace, the reviewer's service time should be restricted to one year to allow for alternative scientific points of view and perspectives on a particular field preventing a 'review style' to be maintained for three years.

In an article entitled ?Science by Consensus? published in The Scientist 8 years ago I stated that ?One of the foremost concerns of biomedical scientists in the United States is the difficulty in obtaining grant funding from NIH.? [1] I indicated that review panels expect so much preliminary data, that the major part of a discovery needs already to have been made, indicating that NIH is not funding actual discoveries, but merely their further characterization. I also pointed out that the tedious description of what a scientist is going to do five years from now is an unrealistic exercise in bureaucracy and is contrary to the true nature of scientific research.

The current crisis of the NIH grant system was described lucidly by Couzin and Miller in Science [2]. Bruce Alberts wrote in February that ??the careers of outstanding researchers can be terminated through bad luck in a chance selection process?one that resembles a game of Russian roulette? [3].

The multiple re-submission of grant applications has overwhelmed the NIH review system. Scientists are spending much of their time trying to obtain funding rather than focusing on their research.

This crisis cannot be resolved by attempting to incrementally improve a peer review system that has enormous shortcomings to begin with. The solution is a paradigm shift in the way we fund research, so that biomedical scientists can stop playing grantsmanship games, and focus on their research [4].

In the new paradigm, the majority of NIH funding will be allocated to researchers who have a track record of solid publications, as determined by large international panels of established and younger scientists.

The panel members will not travel, and the review process will not require tedious analyses of long proposals, but a simple scoring of accomplishments that have already been peer-reviewed for publications.

The grants will be limited to $300,000/year for 10-year periods. The awardees of these grants will not be permitted to submit R01 applications, unless they forfeit their grant within a year, placing them on an equal footing with other scientists vying for R01 support. They may, however, apply for shared instrument or non-federal grants.

These awardees will save much time by not having to submit numerous grant applications, avoid the anxiety of uncertainty, focus on their research, take risks, and afford to be creative.

NIH will save large sums of money and time on the review process. Current grant reviewers will save time that they can spend on their own research, and will save the world tons of carbon footprint by not having to travel to the NIH. They will, however, not be privy to their competitors' ideas!

Within the first year of the program, 10,000 such grants will be funded at a cost of $3 billion. The number of awardees will be ramped to 40,000 in 4 years. If 2000 new long-term awardees are added each year, the direct costs will increase by only $0.6 billion/year. This is more than double the number of new R01s awarded in 2005

NIH will then focus on evaluating large R01 and program grants that may be necessary for certain types of research.

Indirect costs will be limited to 30%, bringing the total cost for the 40,000 grants during the 4th year to $15.6 billion/year, slightly more than half the current NIH budget.

Young scientists starting out in an independent position, and with a proven post-doctoral track record, will receive $50,000 per year for 5 years to establish their own research programs. This steady source of funding will facilitate productivity, and help them obtain supplemental funding from other sources. Even if 10,000 new grantees are added per year, the cost during the 4th year would be only $2.6 billion (less than 10% of the NIH budget). Depending on their publications at the end of 5 years, the grantees will be eligible for intermediate funding of $100,000/year for 5 years or the larger 10-year grants.

The system proposed here will distribute more efficiently and equitably scarce resources to scientists with demonstrated merit and accomplishments, and will greatly reduce the time spent in crisis mode!

After 10 years, the productivity of scientists on the new funding system, measured by publications, citations, patents, and treatments developed, will be compared with that of investigators who choose to struggle through the traditional process.

Regardless of the outcome of the comparison, the new system will be a more humane and rational way of funding biomedical research.

1. N. Dzgnes (1999) ?Science by consensus: Why the NIH grant review system must be changed?. The Scientist 13 (12 April), p. 13.

2. J. Couzin and G. Miller (2007) ?Boom and bust,? and the accompanying sidebar ?Peer review under stress? (News Focus), Science 316 (20 April), p. 356.

3. B. Alberts, (2007) ?Peer-review processes at the National Institutes of Health?. ASCB Newsletter 30(2) (February), p. 2.

4. N. Dzgnes (2007) ?A new paradigm for NIH grants?. The Scientist 21 (August), p. 24.

I agree that putting the emphasis on innovation/impact would positively affect the review system. The trend at Study Section is to focus on feasibility and the use of new tools and techniques rather than ideas-this is more objective than judging merit on the basis of conceptual breakthroughs, but the latter is what we are striving for. If we were to consider the impact of the proposal conceptually, and judge the PI by his/her past performance, the process would be more conducive to driving the system forward than the current process is. Obviously the track record for young investigators has to be seen more in light of the environment they are in.

Although there are several good suggestions among the posted comments, these would only provide band aid type solutions. There is just not enough money in the system. Two suggestions to consider:

1. Smaller grants with more years of funding

2. Shift indirect costs into direct costs.

I would have to disagree to focus on people rather than projects. Where would that leave new PIs who are yet to prove their track record? I think that innovation and applicability to human health should be the main criteria for funding, followed by previous productivity (publications, preliminary data and such). There are a lot of postdoctoral fellows who are discouraged by the funding situation and are leaving academia and making the criteria based on the fame is not going to improve the situation. Also, if a PI receives a grant from the NIH, he or she should be required to serve on the review panel. I am assuming that the sole fact of them obtaining funding indicates their excellent abilities as scientists.