Nature on Benezra and St Noonan

Mar 25 2011 Published by under NIH funding

Nature has a bit up on the whole Benezra letter / St Noonan situation with respect to the decision of NIH to limit NIH grant applications to a single revision.

Daniel Noonan, a molecular biologist at the University of Kentucky in Lexington, wrote in response what he terms a “spontaneous post”, outlining what he believes to be problems with current NIH policies that have disproportionately affected funding for mid-career biomedical scientists.
His sentiments struck a chord — resonant to some, and off-key to others.
....
“If you lose that one grant-renewal opportunity, it's hard to recover in this day and age,” says Noonan.

I thought one parting shot in the Nature piece was kind of interesting.

Several scientist bloggers believe that Noonan's comments imply that scientists should have access to NIH resources regardless of ability or outcomes; they counter that meritocracy should rule.

I am not certain this sufficiently conveys what a lot of "scientist bloggers" that I read, anyway, were saying. Certainly not what I was saying about the situation. The trouble with "meritocracy" is that it implies there is a single unified standard for excellent grant proposals and I do not believe that at all. My criticisms were mostly that a sinecureocracy for established investigators should NOT rule.

Anyway, back to the A2 issue....
OER director Sally Rockey has an explanation post up at her blog as well.

We recently received a letter from a group of extramural scientists expressing concerns about the sunsetting of the A2 applications. I thought that the entire NIH research community would be interested in reading our response

(see BlueLabCoats commentary)

The point that Rockey doesn't make terribly well is one made by PhysioProf:

More importantly, however, there is a serious delusion that underlies this letter. There is only so much money available to fund competing applications, and the only effect changes in peer review in terms of actual funding of such applications could possible have is a change in which applications get funded. So the notion of “meritorious applications going unfunded because of this pernicious new rule” is nonsense. Limiting resubmissions can’t possibly change the number of “meritorious” applications that go unfunded.

I will admit that I had to think about this a little bit. Here's the way I understand it. People are very focused on a common scenario of the recent past. Let's say your IC of interest had a payline of 12%ile. And your A1 application came in at a 15%ile which resulted in, of course, the PO telling you to "Revise and Resubmit". You did so, and the study section handed you back a 2%ile score on your A2*. Whoo-hoo!

Now the Benezra petitioners are looking at the current 7-8%ile, no-A2 environment and saying "Hey, I have a near miss 10%ile score on my A1. Man if I just had that chance for the A2, I'd get funded, brah!"

They are missing a fundamental part of the previous picture.

The ONLY reason that A1 came in over the payline at 15%ile was because the study section had a whole stack of A2s on their plate that had also come in just over the payline on a previous round of review. If those hadn't been there, the A1 would have received a 9%ile and been funded.

The take home message here is that your just-miss score A1 score of today is not like your just-miss A1 score of 5 years ago. There are no longer any A2s in the queue clogging up the pipeline. Therefore your just-miss score is more like the "OMG, I pray the PO decides to reach way down for this one" score of before.

__
*The reason they did so, of course, had nothing to do with how your proposal was now objectively so much better than it was on the just-missed A1 version. It is much more a reflection of "Gee, I thought we gave this baby a score that would sneak over the line last time. Guess not. Well, we better make sure there is no way in hell Program can overlook it this time..."

23 responses so far

  • And not only that, but with no more A2s being submitted, the total number of competing applications goes down (although surely not by as much as the number of A2s, as there will be more A0s submitted in lieu of A2s that would have been). And that means that paylines become more generous.

  • drugmonkey says:

    True. One heck of a lot easier (and faster) to revise a decent scoring A1 and push it back out than to try to re-write a "substantially different" proposal on the same line of work or to create a newer line of thinking. This has to cut down the app numbers a little bit.

  • drdrA says:

    'but with no more A2s being submitted, the total number of competing applications goes down (although surely not by as much as the number of A2s, as there will be more A0s submitted in lieu of A2s that would have been)'

    If I were a cynical person I'd repeat that more strongly and say that the number of A0s funded is going up because people are turning their unfunded A1s over as "new" proposals.

  • drugmonkey says:

    sure, my suspicion too, as you know drdra. But it will be slightly *harder* and therefore the numbers should sneak down.

  • There is absolutely no question that--all else being equal--the overall denominator must be going down, as some of the grants that would have been funded as A2s are now being funded as A1s, and some of the grants that would have been funded as A1s are now being funded as A0s, and none of those grants are getting resubmitted or "resubmitted". It is also important to recognize that many ICs are applying more generous paylines to A0s than to A1s, thus further reducing the denominator.

  • becca says:

    "sure, my suspicion too, as you know drdra. But it will be slightly *harder* and therefore the numbers should sneak down."
    Or, you know, people don't just submit a shittone of vanity grants to NIH for shits and giggles, and 98+% of those Aps are just going to get turned around as A0s... Which is more than *slightly* harder, it's significantly harder. And in fact involves more reviewer work too. Isn't it possible, dear DM, that people will all respond to this acting intelligently individually (i.e. by having more grants in the hopper) and thus create a system with even more work done, and even less reward for it, and even lower paylines, and even more gnashing of teeth, then ever before?

  • whimple says:

    Why discount the possibility that peer review feedback from the A1 lead to a better proposal with the A2? Even if that funded A2 (sh/c)ould have been funded as an A0, the A2 might be reasonably expected to contain better science. Why allow revisions at all (NSF doesn't) if you don't believe the reviewers have anything useful and constructive to say? The assumption seems to be that all those grants being funded later were simply getting in line and treading water without any improvement. That might be true, or it might not, but the utility of an A2 shouldn't just be dismissed out of hand.

  • anon says:

    whimple, the NSF allows as many revisions as you want. You can even include a response to reviewers, but it has to be within the standard page limit. Each submission is treated as though it were new, but unlike the NIH, the applicant does not have to change the specific aims.

  • drugmonkey says:

    This is my considered interpretation of the scoring of my own revised proposals and from seeing what happens during an appointed stint on study section. I think it the rare case where a reviewer critique changes the eventual conduct of the science in any meaningful way. I.e, beyond what the normal advance of the field and the actually doing of the science (including *manuscript* review) provides.

    Even improving the proposal *as a proposal* has a spotty track record because much of it depends on the viewer- one persons unsupported risky Aim is the *interesting* part to another reviewer...

  • one persons unsupported risky Aim is the *interesting* part to another reviewer...

    Ain't that the motherfucken truth!

  • whimple says:

    If the reviewers can't improve the science, what is the evidence that the reviewers can assess the science? It's a problem.

  • Beaker says:

    Yes, that pretty much explains the widely disparate scores I received last round. One man's bold and innovative idea is another's high-risk folly.

  • DrugMonkey says:

    There is no necessary connection between those two things, whimple

  • atmos_prof says:

    This from a geoscience type... reviewers can't improve the science because once something is funded they have no more control. We learn to write what the reviewers want to get the proposal funded. Doing revisions to make the proposal "better" in the sense that reviewers become happier is in many respects just extra work for everyone.

  • Erik Nilsson says:

    Commenters above who think that reviews often improve science seem to be under the impression that PIs often respond to the criticism of reviewers with substantial and meaningful changes in the actual conduct of their science. Drugmonkey said it best "I think it the rare case where a reviewer critique changes the eventual conduct of the science in any meaningful way."

    OK, I don't quite agree with drugmonkey all the way down the rabbit hole. From the A0 reviews, I and people I work with fairly often make important changes suggested by reviewers, because reviewers often have good ideas. Reviewers often suggest additions to the research team that have worked out well. Reviewers note things I hadn't thought of and alternative approaches. Sometimes those insights are valuable.

    Some of the time, that is, from the A0 reviews. You also get a lot of personal crankiness, attempts to promote their own and their postdoc's research by complaining I omitted their tatty papers on peripheral subjects, and just random nonsensical weirdness. This work is done by people, so I expect nothing less. It's impressive that I actually sometimes get helpful suggestions from people who are effectively donating their time and tasked with shooting holes in my dream.

    But that's the A0 reviews. My experience is, funded or not, the A1 reviews are not informative by comparison to the A0 reviews. So I think the A0 reviews may make some contribution to science, but I don't think A1 reviews ever have.

  • Adriana says:

    The purpose of the new policies is to push scientists out, mostly mid career and later. NIH has less money to spend so they have to reduce the number of applications somehow. You can't just base the decision on great science because there still isn't enough money to fund all the great application.

    It is not any easier for a mid career scientist to shift gears than it is for a new investigators. You still have to have a pile of preliminary data to prove your point no matter what stage you are.

    Some of this seems counter productive. People that have spent maybe 15 or so years setting up model systems and generating hypotheses (paid for by NIH) are getting cut off right at the time their projects are starting to produce high yields. In a lot of cases the funds are diverted to new investigators with worse priority scores to set up more new models systems. Just when they get going they will probably get cut off too.

  • iGrrrl says:

    Why discount the possibility that peer review feedback from the A1 lead to a better proposal with the A2?

    Whimple, I think Erik Nilsson answered this best. In fact NIH policy officers say that part of the thinking behind the new, bullet-point review format was to keep the reviewers from re-writing the proposal for the applicant. Reviewers tell me that they like writing the shortened reviews, but they don't like receiving them so much because they are much less useful in understanding how to revise. As everyone gets used to them, however, I think applicants will get a feel for reading between the lines, and reviewers may say more in the new narrative paragraph on the drivers for the Overall Impact.

    But there are a couple of other wrinkles, starting with DM's footnote. In some study sections there has been almost a "wait your turn" culture, with good proposals being scored low-ish until the A2. One of the many reasons NIH did away with the A2 was to try to change that culture and get the best grants funded first, without those kinds of reviewer games. But those games still happen in some small ways, with a grant maybe getting higher scores to "keep a program of research going."

    Don't forget also, many applicants have the experience of putting in a grant that hung around 17-19th percentile on all three submissions. It was a pretty good proposal, but just wasn't ever going to raise reviewer enthusiasm, regardless of the improvements to the application or to the approach in response to reviewer comments. NIH also wanted to stop wasting reviewer and applicant time on those kinds of proposals, too. And it could be argued that without an A2, forcing the applicant to re-think their research program sooner in the process might be a good thing.

  • iGrrrl says:

    Adriana, even before the removal of the A2, the average age at first R01 was about 42, and the average age at last R01 was about 47. Mid-career people were already struggling. Part of the whole change in policy was to lower that first R01 age, but I don't know that the same level of thought has been given to the downstream effect on established investigators.

    But "shifting gears" isn't as dramatic as you might think. For example, if you're a yeast geneticist, and that's all you do, think about how reviewers might be looking to see more mechanistic biochemistry to back up the genetic results. They don't expect you to become a biochemist, but rather to collaborate with someone who is. Can you look at your central questions from another perspective? Is there something else in another field that could inform how you think about your system? You have to be creative to succeed as a bench scientist (and grant smith).

  • [...] Adriana suggests something very interesting on a prior post addressing NIH grant funding policies, the change to only a single revision, etc. [...]

  • drugmonkey says:

    In the latest Peer Review Notes [pdf] Acting Dir Nakamura claims:

    An early look at the data indicates that the policy is achieving its primary goal of increasing the number of A0 applications that are funded.

    Of course I continue to maintain this is utter tripe until and unless they can distinguish between genuinely novel proposals and those that have been reviewed previously in large part, if not in explicitly identical form.

  • drugmonkey says:

    many applicants have the experience of putting in a grant that hung around 17-19th percentile on all three submissions. It was a pretty good proposal, but just wasn't ever going to raise reviewer enthusiasm, regardless of the improvements to the application or to the approach in response to reviewer comments.

    "many"? Hmm. I can't recall having this experience, myself.
    Anything I've had in the 19th percentile range and revised has done significantly differently on revision.

    The only place I seem to stall out on is "triage".

  • iGrrrl says:

    DM, I think I could have chosen a better phrase than "many". How about "not uncommon", or "heard it often enough from different people that it stuck in my head as a category of non-rare application results"? It may not have happened to you, but you're also very plugged-in and savvy. Heck, I quote you and pimp your blog often enough.

  • drugmonkey says:

    It may not have happened to you, but you're also very plugged-in and savvy.

    that is kind but I am also a mere sample of one.

    Heck, I quote you and pimp your blog often enough.

    Many thanks for that!

Leave a Reply