What I learned at an NSF Bio preproposal panel

It was my intention last week to blog a bit about the NSF Bio Preproposals while I was in DC, but that just didn't happen. What did I learn? Well, I can tell you what happened on one panel, but I got the impression that there is some decent variability in the system right now. Below are a few observations.

- The Big Idea is no more important than before. In talking to people leading up to writing the preproposals, there was a lot of emphasis on selling your Big Idea. Theoretically there was going to be less emphasis on methods and more on potential. Well, that kinda turned out to be bullshit. They were essentially judged like mini-proposals.

Cut rate. Our target cut was supposed to be 80%, with 15% landing in the category of "High Priority Invite" and 5% in the "Low Priority Invite". In the end we were closer to 20% HPI and 15% LPI, with everything else landing in "Do Not Invite". How many from the LPI category will actually get invited was not clear.

Two flavors. A lot of good science went into the DNI category. If you get a panel summary you will know why: The DNIs with a summary were discussed, the rest were triaged.

Triage. Anything that got less than three ratings of "good" in the preproposal stage was not discussed. Slapped a boiler-plate panel summary on those and moved on. Roughly 25% of the initial pool went undiscussed.

- BI still matters. Thought you could short the Broader Impacts section just because it was a preproposal? Wrong. A crappy BI section bumped several proposals to DNI.

- Small proposals get killed. For a long time there has always been the party line at NSF that there was no reason for a small grant mechanism because you could always send in a small proposal. Well, guess what happens when you remove the budget and measure all proposals with the same stick? Yeah.

- NSF is worried about new PIs too. Much was made of the concern for the N00bs, and NSF is watching this closely. If new PIs get disproportionately whacked, there will be a correction (This goes for RUI as well). BTW, current average to grant is three years.

- Possible funding from preproposals. One possibility that came up was that the very top preproposals might, in future years, just get funded without a full proposal. Things are still in flux, but my panel felt it could make some awards now.

- Incorporation of preproposal panel members for the full proposals is being discussed. There was some concern by panelists over having two very different hoops for the PIs to jump through, between the preproposals and the full proposals. One way to keep some continuity could be if preproposal panelists agreed to serve as ad hocs for the proposals that got invited and for which they were the primary reviewer in the first round. Everyone started with eight primaries, but I don't think anyone had more than two or three make it to the invite stage, which would be a manageable load to ad hoc.

- Shrinking everything. The full proposal may be going on a diet soon, too.

- Three's often enough. Most proposals got a pretty fair treatment with three people having read them and no ad hocs. There may have been some that could have benefited or been hurt, but overall a panel-only review didn't seem to be an issue.

- The long month. The time line to notifications is roughly a month for those getting an invite, longer for those that got bumped.

So, do I think this is going to be an effective process? It's too early to tell. Just based on my preferences as a PI, I think I would prefer something more along the lines of the 8 month cycle that MCB has gone to, so that you can more effectively manage one's grant load. As it stands now, anyone who can apply to DEB and IOS could have 4 preproposals going in every January. The 8 month cycle with 1 proposal per round would spare people the year between submissions and spread the load out a bit.

But we will have to see how it all goes. One thing was clear: no one, at NSF or as PIs, really know how this is all going to play out.

39 responses so far

  • Dr24Hours says:

    I don't live in the NSF world much, but I have considered applying there. It seems that part of the problem in general, for all the agencies, is being able to legitimately respond to the massive flood of applications. I'm not sure what can be done about that. Except for restricting eligibility, or massively increasing the budget (which might not produce better science).

    What do your acronyms stand for?

  • Zen Faulkes says:

    Thank you very much for this. This info is gold. I look forward to my rejections.

  • proflikesubstance says:

    Division of Environmental Biology (DEB)

    Division of Integrative Organismal Systems (IOS)

    Division of Molecular and Cellular Biosciences (MCB)

  • Jen says:

    Thank you very much for sharing your thoughts on the process - this is so helpful! I will be moving into a faculty position at a PUI this fall, and intend to eventually submit a RUI. From the context of your post, it sounds like RUI preproposals are reviewed with the other Bio proposals - is that correct? If so, do the panelists know it is a RUI? I am completely new to this process, and wonder if RUI funding lines will be changed by the preproposal process.

  • anon says:

    Great post, thank you. What is the rationale for restricting preproposal/proposal submission to one cycle per year? As you say, time to first award is 3 years, which for many of us, is way too long. Do you think it would be possible to have two pools of applications - one is full-length proposal, and the other, a preproposal. The intention would be to reduce the cycle length to 6 months.

  • Zen Faulkes says:

    Jen: RUI proposals have *always* been reviewed with other proposals.

    Panelists know it is an RUI because people are supposed to put "RUI" in their titles (including pre-proposals).

  • Zen Faulkes says:

    Anon: The rationale was to cut the work load of the reviewers. NSF was having a harder and harder time getting people to review full proposals.

    Too many scientists, not enough money. More and more proposals to review.

  • proflikesubstance says:

    Jen,
    The RUI proposals have always been in the normal panels, as Zen said. My understanding is that the funds mostly come from that panel's pot, but there may be some additional money that can be channeled to those proposals from a central pot.

    Anon,
    Essentially there are two rounds now - you need to get your preproposal invited for a full proposal or you can't submit in August. If people didn't have to write the preproposal, no one would.

  • mmd says:

    Thanks for the info. I have a bad feeling I've got a DNI coming for my small proposal, but so be it. I put in 2 preproposals to DEB in January, but the program officer decided that one was better suited for MCB, so I'll be trying out that process in May. While I wasn't that excited about the change to preproposals, I'm finding that I'm now pretty resentful of the time I'll need to sink into a full proposal with a low chance of success. It's not just the additional text, it's also all the budget crap that needs to get hashed out for a full proposal that didn't have to go into the preproposal.

  • Cyn says:

    "Theoretically there was going to be less emphasis on methods and more on potential. Well, that kinda turned out to be bullshit." = Why do you think that happened? Very interesting.

  • proflikesubstance says:

    The panelists looked over the preproposals and became wary of someone selling a project on hand waving, and there was a lot of it. The better proposals had a Big Idea and backed up the analysis ideas with data, much like a typical full proposal. The POs can say "sell your Big Idea" all they want, but if the panelists aren't buying you are not getting to the next round.

  • Drugmonkey says:

    Same problem with the NIH R21 "Exploratory/Developmental" proposal. I think it sad and poorly thought-out to insist that everything has to be proposed as if it was a slam-dunk. It means only the comfortably-supported investigator has the freedom to go off on wild-haired ideas.

  • proflikesubstance says:

    DM,
    To a certain extent, I agree, but you have to realize that this is the only funding mech for a lot of people. I completely support an R21-like mech for NSF (and no, people, EAGERs are not the same), but reviewers want to see some evidence of likely success and judged accordingly.

    As a corollary, we were given NO official directions on how to review these things or that it should be a different process than prior. There was a webinar, but it mostly dealt with the process. If NSF wants to move to a Big Idea preproposal, followed by a nuts and bolts full proposal, they need to pound that home to reviewers first.

  • Neuropop says:

    I felt it went the other way in the panel I served on. Most of the newbies focused a little too much on detail (small town grocer stuff) but not enough on ideas. As long as the panel felt that
    the PI had had adequate training (for noobies) or had a track record, feasibility was a given. So the ones that ended up high on the invite list were precisely ones that were able to lay out the significance and approach (not the nitty gritty details but a broad plan of experiments that advanced the stated question/hypothesis) succinctly. The premium was on illustrating the kind of data that a set of Aims would yield and how they would be analyzed and interpreted. Whether such data was already present was deemed useful but not necessary. I thought the panel adhered to the "pre" part of the "pre-proposal" quite rigorously.

  • proflikesubstance says:

    That's really interesting, but not surprising. Each panel is going to have a "house flavor" right now. In the panel I was on, the ideas were key, but feasibility was not so generously assumed.

  • LD says:

    Did the PO's explain why it will take them a month to send out the invites? It's not like they are promising anything. This will limit the time we all have to write the full proposals. I certainly don't want to start down that rabbit hole until I have a go ahead...
    LD

  • CoR says:

    I wrote mine like NeuroPop describes, based on the recs of my PO. I hope the panel doesn't get all wanky on me.

  • proflikesubstance says:

    LD, there is actually a substantial process to any decision, much of which seems out dated to me. Each Division Directory and the deputy DD read the project summaries, panel summaries and reviews of EVERY proposal in the division as the final stamp on each decision. You can imagine there are similar hoops before it even gets that far. This all takes time.

  • Isabel says:

    Are there any data to back up the NSF selection process? That is, regarding the panels concerns about "hand waving" without preliminary data etc? Were unsuccessful projects a plague in the past because people were using crazy untested methods and is this some kind of correction? This is another area that is hard on grad students who are not following in their advisor's footsteps. Even when relatively small amounts of money are at stake, panelists seem completely neurotic about this aspect.

  • drugmonkey says:

    I bet the answers are 1) no data and 2) no credible prior problem, Isabel. Just like in NIH land.....

  • proflikesubstance says:

    Isabel, if there is, I haven't seen it. The general idea was for the short preproposals to serve as an elevator pitch for the full proposal, with a focus on the exciting aspects rather than the experimental conditions. In the panel I served on, however, the field is seeing a change-over in some of the core technology. Because of that, a lot of people were proposing to use new shiny machines without an evidence they could handle the resulting data (which is a major issue, in this case). So very quickly panelists wanted to see some evidence of feasibility before trusting that they weren't just green lighting a bunch of proposals that are going to look like shit in the full round. The question "but can they do it?" was asked often. In the absence of demonstrating it, you're leaving it to the panelists to decide.

    I don't know that that was how NSF wanted this to play out, but to a certain extent PIs are like salespeople and the best ones can sell an idea. Anyone can say they will do the latest cutting edge stuff, but...

  • ecologist says:

    "They were essentially judged like mini-proposals. "

    Thanks for this very interesting and informative post. But, I have to say, that to the extent that your statement quoted above is correct, the pre-proposal idea is a total failure.

    "Anyone can say they will do the latest cutting edge stuff, but..."

    "But", indeed. It sounds like your panel forgot that the decision they were making was not whether or not to fund the proposal and give the PI money to do the latest cutting edge stuff. It was whether or not to give the PI a chance to write a full proposal and put in all the usual stuff to prove that he/she can actually do the cutting edge stuff.

    If NSF wanted the pre-proposals to be something really new and useful, they would have used the following logic:

    1. Is the idea interesting? If the answer is "no", then the ability of the PI to do cutting edge stuff, or any kind of stuff, is irrelevant, because no one cares about the ability to do good methods for an uninteresting idea.

    If the answer is "yes", then give the PI the opportunity to fully document the following series of questions:

    2a: how will he/she do the research?
    2b: does he/she actually have the ability to do the cutting edge stuff?
    2c: will the cutting edge stuff actually answer the question?
    2d: what will the broader impacts be?

    The components of part 2 are what we always write proposals to document. If the preproposal process is to work, it should be designed to make sure that all that detail is ONLY submitted when it addresses truly interesting questions.

    "The question "but can they do it?" was asked often. In the absence of demonstrating it, you're leaving it to the panelists to decide. "

    This is stupid. Why not say that, in the absence of demonstrating it, you're leaving it to the PI to document that in a full proposal, as it ought to be?

    I have a lot of experience and a lot of investment in the NSF funding process. The idea of a preproposal stage is potentially a good one, but if it got distorted in this way in all the panels, then it has been, for this round anyway, a failure. It sounds like more guidance from the program managers and above is needed, and I hope it is provided next time around.

    Again, thanks for the post. It is truly useful to find out this kind of report from within the process.

  • proflikesubstance says:

    It sounds like your panel forgot that the decision they were making was not whether or not to fund the proposal and give the PI money to do the latest cutting edge stuff. It was whether or not to give the PI a chance to write a full proposal and put in all the usual stuff to prove that he/she can actually do the cutting edge stuff.

    OR, perhaps this panel did not want to send a large number of proposals forward to the next panel that have a low probability of turning into solid full proposals. In 4 pages you can get a decent feel if you are being sold a good idea or a bridge in Brooklyn.

    Your idea of How This Should Be Done was essentially how it worked out. Do you really think a panel of experienced PIs are going to favor boring science with a high probability of success over great ideas with a decent chance? Really? When I said the Big Idea is no more important than before I actually meant that, not the Big Idea is less important than before.

    Cool science was the first factor, one's ability to do what they claimed was probably the next. If you "have a lot of experience" then you know a lot of cool ideas end up on the cutting room floor in favor of cool ideas that are likely to work. "Can they do it?" was asked often, because we had a lot of good proposals. At least in the panels I have been on, the line between funding (or getting invited for a full) never divides "good science" and "bad science", but splits hairs somewhere high up in the "good science" category because resources are that limited.

  • ecologist says:

    Well, since you asked ... Yes, I do worry about a panel of "experienced PIs" making decisions that put more weight on boring-but-safe and less weight on interesting-although-risky than they should. This happens all the time in NSF's regular panels. When you say that the Big Idea is "no more important than before" and reassure that you actually meant that, I repeat that this is a failure of the preproposal idea.

    No, I'm not saying that the idea should be the ONLY criterion (let me tell you about my cool perpetual motion machine ...), and maybe I gave that impression in my comment. But in this preproposal process it should have been more important than it was before, and the details of how to do it should have been less.

    I am saying that the BALANCE between the priority given to ideas and given to methodology should have shifted in the preproposal evaluation. If the balance isn't going to shift, if these are going to be judged on the same balance as full proposals (or, as you put it, like "mini-proposals"), what's the point? You are now trying to make the judgement on the basis of about 75% less information than in a full proposal. On what rationale is that an improved way of making what you acknowledge is a decision that comes down to splitting hairs?

  • [...] with their approach to preproposals. The panel I applied to had considerably more applications than the one I served on. So many, in fact, that the had to deal with them in two panels. So, not only is there the [...]

  • @BrunaLab says:

    Thanks for a great summary! One question for you, which on the surface may seem trivial - what did successful proposal's look like? By that I mean did authors cram as much as possible in, with small font, limited white space? This was the subject of endless discussion here - to cram or not to cram - especially if you wanted to include figures, which consume much space that could instead be devoted to text or font size that is easier on reviewers.

    Good luck with the full proposal,
    EB

  • @BrunaLab says:

    PS, not sure how that apostrophe got in there...*proposals*

  • proflikesubstance says:

    BrunaLab, more information in yesterday's post: http://scientopia.org/blogs/proflikesubstance/2012/05/02/nsf-preproposals-now-what/

    But, I would say that cramming is NEVER a good idea. It make reviewers cranky, which is bad. Better to tighten your wording and simplify to get the point across.

  • @BrunaLab says:

    That was the final decision we all made here as well, though there was definitely angst about sacrificing data and figures for (what we perceived as) clarity and readability.

  • proflikesubstance says:

    Keeping the reviewer happy is about as important. Adjusting to telling a logical story and supporting it with data/figures in a smaller space is more about simplifying the story to the highlights than trying to fit 6 pages into 4. The best preproposals walked the line between detail and story.

  • [...] No, go to The Spandrel Shop and read Prof-like Substance’s post and [...]

  • [...] if reviewers still focus on primarily on methodological details (as seemed to be the case on Prof-like substance’s panel) then the new system could end up putting even less emphasis on big ideas, because the 4 pages will [...]

  • [...] on the big idea focus and thought it went well. Their first entry was here. Prof-Like Substance, here, was more discouraging to me. He said the proposals were treated like mini full proposals, and [...]

  • [...] seen NSF make some major changes in the way they review proposals and I went to DC to see how this would work, first hand. After much debate over this process, I offered up a [...]

  • [...] Lots of other folks with better first-hand knowledge have written about the new process. The key change is that, where formerly NSF offered two opportunities per year to submit a proposal for funds, the new procedures introduced a “pre-proposal” stage in which biologists write a much shorter pre-proposal first. If this mini-proposal is judged worthy, the applicant is then invited to submit a full proposal several months later. [...]

  • [...] Lots of other folks with better first-hand knowledge have written about the new process. The key change is that, where formerly NSF offered two opportunities per year to submit a proposal for funds, the new procedures introduced a “pre-proposal” stage in which biologists write a much shorter pre-proposal first. If this mini-proposal is judged worthy, the applicant is then invited to submit a full proposal several months later. [...]

  • biologist says:

    An 8-month cycle seemed long enough when I submitted my preproposal in January, but now December's almost here and I still have no answer on my full proposal, so it's looking more like it's going to be an 11-month cycle. I'm expecting to spend my Christmas break thinking about how I'll shape up a new preproposal with new preliminary data based on comments I haven't seen yet.

  • proflikesubstance says:

    I know it is frustrating and some of the delay is budgetary. Remember that they moved the preproposal deadline back to around the 20th, so you'll have some extra time in Jan this round.

  • [...] what made a successful application. After attending a panel, I've expressed what I thought made a good preproposal and so have [...]

Leave a Reply