The funding rate myth

May 04 2012 Published by under [Education&Careers]

7%. <10%. 12%. These are all funding rates I have seen in recent panel summaries and quoted widely on the intertubes and in faculty meetings. On the surface, these numbers are crippling. How is anyone supposed to be competitive when rates are so low!?!?

One thing the preproposal process gave me a greater appreciation for, however, is that those funding rates are misleading. I am not at all saying that we don't need more money for research in this country, nor am I here to say that 7% funding rates are anything but demoralizing. But those bemoaning 7% as a figure are making a flawed assumption: that all proposals are worthy of funding.

If all proposals were within a standard deviation of fundable, then a 7% chance of success would basically seem random. This is demonstrably not the case, however.

Take the preprosals as an example. Let's pretend that the panel normally fields 100 proposals and that it got the same number of preproposals (most panels saw >2X the number of preproposals than their regular proposal numbers, but work with me). Three panelists weighed in on each proposal and nearly 25% of the time, all three determined that a particular proposal was poor enough to reject outright. No discussion, it just wasn't what it needed to be. If a hypothetical panel had 100 preproposals, we're now down to 75.

Of the remaining number, another 25% just cleared the "cut" bar and were discussed, but no one felt they were in contention. Down to 50.

Of the remaining proposals, all of them had strong points and many had vocal advocates. Where they ultimately ranked was based heavily on the perceived weaknesses (if there were any) and how passionate their advocate(s) was. But even in this group, there was a clear sense of a spectrum on which they fell. And this is where some of the tough decisions start being made.

Enter the preproposals with a 20-25% invite rate.

More than anything, the 20-25% cut off represents the proposals that would be in the conversation for funding. In an ideal world we could fund these 25 proposals, but the percentage is an arbitrary number based on submission rate. So whereas a 7% funding rate is a scary number, the reality is that those seven proposals are only coming out of a 20-25 pool of proposals whether you make the cut at the preproposal stage or the full proposal stage. It doesn't matter if 100 proposals or 200 proposals are submitted, a lot of them are just not convincing or exciting to a panel.

I've heard POs make this argument more than once and never really made sense to me before going through the preproposal process. As much as I have some issues with this new process, I have to admit that it allows the culling of the bottom 50% with FAR less work (for writer and reviewer) than the previous system. This is a good thing for everyone.

So whereas funding rates are low, fretting over the funding percentage is to take an uninformed view of the process and what that percentage represents.

13 responses so far

  • Zen Faulkes says:

    "Not worth funding."

    It's worth looking at the assumptions behind that phrase. NSF wants "transformative" research, for instance. I'm curious as to how much science is not funded not because it's incompetent, but just because it's viewed as ordinary.

    Or do we agree that there are some kinds of research projects that are valuable even though they are not transformative?

  • proflikesubstance says:

    I'm talking about the proposal. More proposals than you would think are really terrible. A good proposal can sell almost any project. Even small, focused, projects can be exciting when placed in a larger context and made accessible.

    NSF is not looking to only fund "transformative" because those projects are actually quite rare (despite some people writing in their own proposal that it will be transformative. Don't do this). There is no question that both panelists and NSF want to fund projects that are not transformative, but are solid science.

    But that doesn't excuse a crappy proposal.

  • DrugMonkey says:

    Your discussions are focused by a presumption of the funding rate so I'm not convinced it really addresses the % apps that are worth funding over, say, one more Tomahawk missile launch.

  • Zen Faulkes says:

    Reassured. A teensy bit.

  • HCA says:

    Somewhat tangential but, given the experience of PLS and other folks here, how much would funding have to increase before you can say "We gave money to everything that was good science and had at least a decent shot of working"? Threefold? Fivefold?

    Essentially, how much "should" the US be giving the NSF and NIH to get the most high-quality (or even medium-quality) science done?

  • Zen Faulkes says:

    When the NIH announced Challenge grants as part of the American Recovery and Reinvestment Act of 2009, they expected to give out 200 awards.

    They got 20,000 applications.

    Let's say that 10,000 of those are not in contention (to use PLS's phrase above), and another 5,000 are okay, but not the most meritorious. That would leave 5,000 serious applications. If, ultimately, 7% of those were funded (continuing to use PLS's success rate), you'd have 1,400 awards.

    This suggests that we're out by a factor of 7.

    (Yes, there are many reasons why this particular example may not be representative. But it does say something about the how much people are chasing money.)

  • lylebot says:

    If you know how to write a proposal, your chances are better than 7%. If you don't, they're probably not.

    But learning how to write a proposal is not easy. I know lots of people that have good ideas, are good at writing papers about those ideas, but have not yet figured out how to write a proposal to get them funded.

    In my experience as a reviewer, the worst proposals people submit are a lot better than the worst papers people submit. Clearing that bar for proposal preparation that gives you a better-than-7% chance is (or at least seems to me to be) much harder than clearing the same bar for paper preparation.

  • Isabel says:

    "So whereas funding rates are low, fretting over the funding percentage is to take an uninformed view of the process and what that percentage represents. "

    I still can't seem to wrap my head around the idea that 90% of applicants, who have climbed the difficult path to tt positions, are dolts who are not doing solid science and can't manage to write a decent proposal and are therefore not worth funding. How did they get to where they are in the first place without a track record, good ideas and a clear vision and publications galore etc? Suddenly they are idiots?

    Fortunately my project was funded by a private organization (equally competitive and finally something for my CV) so thankfully I will have a short break from dealing with NSF until I have to apply for postdoc fellowships. Though I've been wondering- why does NSF fund so many expensive graduate fellowships when grad school is free, when they could fund the research projects of ten times as many grad students with the same amount money?

  • odyssey says:

    why does NSF fund so many expensive graduate fellowships when grad school is free

    Grad school is not free. It may be no (financial) cost to the student, but someone is paying for it. Sometimes the institution. Sometimes the PI out of her grant funds. Either way, the fellowships funded by the NSF frees up money that can be used to support more grad students* or for other purposes.

    _________

    * Whether or not that is a good thing is of some debate.

  • odyssey says:

    And as to why the NSF Graduate Research Fellowships pay so much, it's because the NSF is trying to encourage the best students to enter STEM careers. These fellowships are very prestigious and quite difficult to get.

  • Pascale says:

    When I have looked at NIH funding, initial scores for the top 25% or so of proposals fell between 1.0 and 2.0 (old scoring system), the range for excellent to outstanding. Given the quality of folks who have pubs and preliminary data, it makes sense that at least this many are submitting proposals without major holes or issues. Most people who submit these grants know what they are doing!
    Based on my own study section experience, 20-25% of NIH proposals would be funded in a perfect world.

  • Isabel says:

    "Grad school is not free. " wait I thought grad students were cheap labor, etc. Only a small percentage of students have fellowships. Since grad school is so competitive especially at top schools, I don't see that the fellowships are serving a purpose except to increase the number of grad students and there is certainly considerable controversy over that. If they are supported on a PIs grant they are working on the research. right? If they are supported with TAships they are cheap labor, right? They don't even take classes the last three years.

    Is there any evidence that science would suffer without these fellowships, instead of directly funding the research projects of many more students? It also creates a weird two-tier situation for grad students. And the students who work all the while are expected to produce as much as the fellowship students, and to produce equally polished grants proposals....I wonder how even the playing field is to get these fellowships in the first place.

Leave a Reply