Reconsidering the "too many mouths at the NIH Grant trough" hypothesis

I keep mulling over the data presented in this entry at the Rock Talk blog. I originally concluded that this, combined with the revelation that applications-per-PI only went from 1.2 to 1.5 across the FY98-FY11 interval, showed that the RealProblemTM at the NIH was the growth in the number of Principal Investigator mouths at the trough.
As a reminder, these are data for the investigator-initiated Research Project Grants only and exclude the ARRA largesse. The graph shows change data from the baseline of Fiscal Year 1998. As a brief summary of my prior thoughts:

the post indicates an increase from an average of 1.2 applications submitted per investigator to an average of 1.5 per investigator from 1998 to 2011...This surprised many of us on the Twitters. I don't think I know of any active scientists who are submitting less than several NIH grant applications per year...if we harken back to some data on..the Rock Talk blog (maybe; UPDATED, it was RockTalk) which showed the average NIH PI had only about 2 grants concurrently then we must consider that there are still a LOT of folks out there on a single grant at a time. Especially if they have long-term continuations going, sure, maybe there are a lot of PIs who only have to submit an application once every 5 years. The post also indicates that there were 19,000 applicants in 1998 and this grew to 32,000 in 2011. Some 13,000 new mouths, a 68% increase in PIs seeking money from the NIH.

I've added emphasis to highlight what has been bothering me.

The notion that we have 68% more PIs seeking money from the NIH should have been more of a warning. The thing is, it dovetails nicely with one of the very truthy memes that we have going about the effects of the NIH Doubling interval. More people in the system made a certain sense. Particularly for those of us who were entering the system approximately during the doubling interval and did not feel as though it was easy to get a grant funded. Certainly, success rates did not double. In the historical sense the success rates were only moderately restored from a slide that ran from the late 80s (40% for experienced applicants. Think about that.) to about 1994 (25% for experienced applicants). So if the budget was doubled and success rates were far from doubled, there must be more people seeking funding. Right?

What never seemed possible to me was that traditional research-heavy Universities, who were already deep into NIH-addiction, were throwing up that many new jobs. Sure, they expanded their soft-money faculty positions a little bit...and let the occasional word-salad-position Assistant Adjunct Research Project Professor of Bunny Hopping upjumped postdoc submit a grant or two. But it didn't seem likely to me that this explained the budget/payline disparity. Nor, in context of Rockey's data, did a 68% increase in PIs at such places seem likely. So I was always asserting that the growth came in large part from the entry of new institutions into the system. In the sense that smaller, less research intensive Universities were, perhaps, putting on a big push to get in the NIH game. Perhaps this was by hiring new NIH-honcho faculty. Perhaps by pushing hard from the deanlet level to get the existing faculty to submit more grants, bring in more NIH moola. This latter hypothesis was fueled by rumor of this kind of behavior from some of my colleagues and friends so I was primed to believe it.

My new realization of the week is that the data from Rock Talking are misleading. The denominator for the grants-per-PI is calculated on a per-FiscalYear basis. It has to be, even though they don't say this. So you only get counted if you've submitted at least one grant in the FY. Similarly, the growth in the number of PIs from the 1998 baseline is likewise a reflection of the number of PIs submitting at least one competing grant application in a given FiscalYear. Again, they don't specify. I was perhaps assuming that this reflected the number of PIs in the system, i.e. submitting competing or noncompeting applications. In some senses, we also have to keep in mind the number of occasional applicants to the NIH...hard to believe from my perspective but sure, why not consider that there is a pool of PIs who may have repeated, but not continuous funding from the NIH across their careers?

Keep in mind that I'm eventually getting around to the consideration of the massive decrement in the purchasing power of the standard, $250K direct cost, full modular award.

As you can see, a full modular $250,000 year in 2011 has 69% of the purchasing power of that same award in 2001.

We'll return to this.

Let us start with consideration of what appears to be, going by disgruntleprof comments on various blogs and opinion pieces, the shining virtue of the NIH system...the one-R01 small town grocer. This PI submitted a grant application once every five years to continue her R01...in the old days. So on average this person would be submitting 0.2 grants per FY but in the Rockey analysis would only count as 1 grant-submitted...every five years. Over time, however, she is now facing a decreased probability of getting funded the first time and, let us say, submits an application three times (A2 scenario, not unlikely at all by the end of the doubling), a year apart, during her 5 year window. Her Rockey number is still 1 application per year but her 5 year average has increased to 0.6. Similarly, since we're dealing with the one-grant scenario, the appearance in the Number-of-Mouths data is likewise affected by the frequency of submissions. Taking the 3 tries case again, if she only had to apply twice every 10 years in the past but is now applying 6 times to maintain funding, she has tripled her presence in the Rockey way of looking at the number of applicants. If we're talking about an overall 68% change over time...this kind of behavioral change is significant if it occurs in any appreciable part of the PI distribution. It makes it look like there is a big change in the number of PIs that need to be fed when there have not, in fact, been two more PIs added to the system.

Getting back up to my original thoughts on where the RealProblemTM lies, however, this is all critical. Is the NIH in fact supporting 68% more investigators in 2011 vs 1998? This is what Sally Rockey's post would imply. It certainly implied this to me. However, it may simply reflect the same number of overall PIs in the NIH-funded extramural workforce who simply have to submit more grant applications to maintain the same number of grants.

Which brings me to my next point. Note that I said "same number of grants" but not "same amount of funding". Because it is also clear that over this self-same interval when SmallTownGrocerPI was forced to submit applications more frequently to sustain her funding, she was also forced to try to get more awards simply to maintain the same level of operation. Because the purchasing power of the grant dollars had fallen by so much and yet the full-modular cap still imposed a de facto limit on budget escalation. grants_per_pi_allNow true, the "myth-busting" data from Rockey show only a 4-5% shift in 1-grant to 2-grant PIs from FY1986 to FY2004 when the doubling was rolling hard. This is where the simple case we are discussing really breaks down. Obviously there are many varieties and mixtures of PIs in terms of the number of applications submitted, the stable-versus-growth aspirations, the amount of NIH funding that represents stable state, the mixture of R01 and "other" funding, etc.

So obviously it would be a complex modeling job in the NIH databases to get the best understanding.

But it strikes me that one of the simplest and most productive things for the NIH (read: Sally Rockey's data mining minions) to do would be to take a closer look at the number of PIs applying instead of the number of applications. The number of PIs over an extended window of time, not just on a per-FY basis.

This reason that this is important to know is that the success of any proposed fixes to the NIH depend on this reality. If there has genuinely been an increase in the number of PIs then shelling some of them out of the system permanently (including by preventing entry) is the only way to have sustained effect. Within that category, it may be necessary to see if the growth in PIs has come from the top research Universities or from increases in the lower-tier Universities.

If the main trouble is the uncertainty of maintaining one award, then the solutions are to extend the interval of non-competing and/or give a much larger payline break to competing continuations versus new applications.

If the trouble is that the purchasing power of the full-modular has decreased, then boost the limit to $375K per year in direct costs. [ETA: per comment from Grumble, note that the purchasing power has also been eroded by habitual budget reductions upon funding. Some ICs cut a whole year. Some have made 1-2 module ($25K per module) reductions the SOP. Some hit even non-competing renewals with additional reductions because of budgetary uncertainty. They do this to artificially prop up the success rates. Take one module from 9 awards and you can fund 10.]

It is incredibly frustrating for those of us who watch from the outside since these data are clearly available within the NIH databases and they simply seem to be looking* in the wrong direction.

__
*I realize that Sally Rockey may have a ton of analyses that she simply has not put up on the blog. Somehow, given her little oopsie with the alternative career fate of trainees, I doubt it.

97 responses so far

  • miko says:

    So does the NIH...

    1. Not understand it's own data?
    2. Purposefully wish to misrepresent/confuse these issues?
    3. Somewhere in between

  • Grumble says:

    When is the last time you or anyone you know actually got awarded the full $250k/year modular limit? Everyone applies for this level, and nowadays you're extremely lucky if they only cut one module - typically it's two. So the purchasing power of a grant is now closer to 50% less than it was 10, 12 years ago, not 69%.

  • drugmonkey says:

    Sure Grumble. That is another factor. And a clear choice point for the NIH.

  • physioprof says:

    give a much larger payline break to competing continuations versus new applications

    YEAH!!! KILL THE YOUNGS!!!11!1!!

  • kevin. says:

    Yeah it seems like a reasonable explanation is that 80% of PIs w/ one grant or so get it renewed pretty much every time (small-town grocers or otherwise), and it's the 20% who have more trouble that are responsible for the "churn."

  • drugmonkey says:

    to be clear kevin, this is by no means the case anymore. it is the growing uncertainty in continuing a "reasonably productive" single grant even during the doubling interval that I was trying to describe here.

  • Bill Hooker says:

    "one of the simplest and most productive things for the NIH (read: Sally Rockey's data mining minions) to do would be..."

    ...to release the fucking data into the public domain so the people with a compelling interest in really understanding it could get hold of it and figure shit out.

  • whimple says:

    Silly for the NIH to see how they're doing measured in grants per PI. Obviously should be dollars per PI. Hint: the BigFish(tm) are not submitting modular RO1s.

  • miko says:

    Would this data be FOIA-able?

  • zb says:

    Isn't the data in NIH Reporter already? I'm not altogether sure of exactly what question you want to answer here, but given that one can create XL spreadsheet of all RO1 grants at NIH Reporter, it seems that the data is available for outside crunching.

    My gut impression is that there are more PI's applying for grants and that more PI's are applying for more grants (both to sustain the level of funding with cuts & uncertainty & because more costs are being shifted into direct costs of grants, including the PI salary), and that the main increase comes from an increase in the population of applicants from major research institutions (i.e. the opposite of your gut).

    A spreadsheet at NIH reporter listing the number of higher ed institutions funded (under the organization tab) suggests that the number of organizations receiving funding has decreased between 2003-2012 (from 505->451) and the number of grants/organization has increased (for 70->76). That's informing my gut a little bit, though it doesn't take into account all the data.

  • Pinko Punko says:

    If you look at concentrated research centers, my question would be who is filling the giant new buildings that have gone up in the last 10 years. But yeah, they are not even close to looking at the data in a meaningful way.

  • Grumble says:

    "Isn't the data in NIH Reporter already?"

    What's missing from Reporter is the denominators: how many PI's have applied? for how many grants each?

  • SidVic says:

    I haven't seen anyone mention this, so I apologize if it’s been covered before, but i suspect that clinical trial grants, SBIR and various set-asides have eaten into the NIH extramural budget in the past ten/twenty year period. While it’s all arguably a good use of government money- a case can be made that the private sector should be doing a lot of more of the clinical trials and biotech start-up stuff; or that they shouldn’t be done at all.

  • drugmonkey says:

    SBIR is a Congressional requirement. Best to think of those types of monies as off the books.

  • drugmonkey says:

    miko- should be . Might have to navigate the "proprietary interest" and "personally identifiable info" dodges though

  • neuromusic says:

    FOIA! FOIA! FOIA!

  • MorganPhD says:

    @zb: The number of PI's applying for grants is not in the NIH Reporter, only those funded.

    To get this information, we'd have to ask the NIH nicely, or in lieu of that, FOIA the data (as @Miko suggested).

  • MorganPhD says:

    There are likely a few problems with doing an FOIA request for this info. I suspect they might first try to hide behind exemptions b(4) or b(6) which are confidential business information or personal privacy.

    Also, unless the request is deemed to be important to the public good and not due to financial interests of the requesters, they could charge for it, and I bet the $$$ would be severe.

  • Beaker says:

    Do the budget calculations pertaining to "the doubling" take into account possible changes in overhead rates? I can't find the historical data on indirects. Did the average overhead rate change during the doubling? If this is so, it helps explain why twice the money probably did not translate into twice the research but instead promoted the proliferation in administrators, Jr. deans, and buildings.

    Obviously, overheads are irrelevant for analyses based on "number of applicants." But many of the graphs in your last post on this topic had the NIH budget as the Y-axis, so I assume these data include overheads. In addition to adjusting for inflation, should we be correcting for "overhead creep?"

  • drugmonkey says:

    Have you seen IDC rates go up? I haven't.

  • SidVic says:

    DM-SBIR is a Congressional requirement. Best to think of those types of monies as off the books.

    Well… it is part of the 30B budget under discussion, was pumped disportionately during the “doubling” ,and thus is on the books. It may well be that SBIR was sold to congress as separate additional funds (good). Or was it the politicians telling NIH how to divvy their money among the constituencies (not as good). Don’t get me wrong, personally, I love SBIR money, wish I had more of it.

    PS, Haha Forget FOIA requests and get back to submitting grants ...

  • Susan says:

    "the average NIH PI had only about 2 grants concurrently"

    Please tell me where you get 2. The mean, median and mode of the RockTalk figure are clearly 1 (1.2 at best by an eyeballing). One. So: The average NIH PI had only 1 grant concurrently. One.

    Even if I have 1 .... there's no way I only reapply once in 5 years. More like I keep trying every cycle for that 2nd R01 and that R03 and that R21 and and and ... because I have to. Because that's what my P&T wants to see. Because my one R01 buys me less. Because funding lines are going down. Because my department funds less students and less bridge and the only solution is moar moneys and the only way to get it is spaghetti at the wall.

  • kevin. says:

    to be clear kevin, this is by no means the case anymore. it is the growing uncertainty in continuing a "reasonably productive" single grant even during the doubling interval that I was trying to describe here.

    Agreed. At least in our lab's case, the R01 has been continuously renewed, but the PI is submitting a new proposal to support an interesting new direction (paid for by grant #1). Whether this should be considered necessary in light of declining $ or is part of the natural growth of the lab isn't clear.

    I would enjoy knowing whether it's the same pool of investigators who are putting together the winning and the failing grants, or if it's an even split. I could see it being that some people are just really good or really bad writers.

  • Beaker says:

    DM--I haven't a clue if average indirect rates have changed, hence my question. I just figured that during the doubling, institutes would try to justify getting a bigger piece of the pie. I doubt that the average for all institutes has gone down; this seems about as likely as sales tax rates going down.

  • drugmonkey says:

    Susan- it is possible that I was looking at the top20% chart on Rock Talk when I made this prior comment.

    Kevin- I am likewise interested in the failures. How many are once-applied, never-seen? How many apply 5 times and give up? How many apply 10, 20, 30 times without a hit? Etc. and how do those numbers break down by various PI and institutional parameters.

  • drugmonkey says:

    SidVic- these Congressional directives should be thought of as money that otherwise would not be in the NIH control at all. So blame Congress but not the NIH.

  • iGrrrl says:

    This is really interesting. NSF is much more transparent, I think, about their data. (The latest report to the NSB is here: http://www.nsf.gov/nsb/publications/2012/nsb1228.pdf)
    Figure 9 "provides the funding rate for investigators (the number of investigators receiving a grant divided by the number of investigators submitting proposals)." Digging around, the percentage of proposals funded is 22% for FY 2011 (Table 2), but the percentage of PIs funded was given as 38% (Figure 9).

    They had an interesting analysis of how many proposals until a PI was funded, which seems to be about 2.3 at NSF, but the figure (and Appendix 8) don't seem to say exactly how the numbers were generated year to year.

    I would love to see these kinds of analyses for NIH.

  • drugmonkey says:

    This chart
    http://report.nih.gov/NIHDatabook/Charts/Default.aspx?showm=Y&chartId=167&catId=22

    suggests about 7,000 investigators supported on competing RPGs (as PI) in 1998 and maybe 9,000 in 2011.

    footnotes are scant so it is not impossible that this only accounts for new grants each year. They funded about 7,000 RPGs in 1998 and about 9,000 or so according to
    http://report.nih.gov/NIHDatabook/Charts/Default.aspx?showm=Y&chartId=20&catId=2
    so at a guess I'd say this doesn't include PIs supported on the ongoing, noncompeting awards.

    Obviously we'd like to see the number of PIs supported by any RPG by the FY, regardless of whether it was competitively awarded in a given FY or not.

  • zb says:

    I realized after I posted that NIH reporter only has funded gravy infO. But, that got me started on thinking why we want to know number of people tryingto feed at the trough?

    This "review" of the NIH analysis had the feel, to me, of one of those manuscript reviews where the reviewer asks you to break down the data in a thousand other different ways.

    What is the first question you want to see answered? is the data inthe NSF PDF what folks are looking for? How would the answer to the putative question inform the advice we'd want to give NIH?

  • AcademicLurker says:

    What never seemed possible to me was that traditional research-heavy Universities, who were already deep into NIH-addiction, were throwing up that many new jobs.

    I'm not sure I agree with your intuition here. Certainly my former institution (R1 Medical School) threw up an expensive new research building* and dreamed that it would soon be filled with new PIs, all with X RO1s a piece. Conversations with my peers at meetings suggests that many other R1 medical schools did the same during the early 2000's.

    *Not that the faculty ever suggested that we needed such a thing. The decision was made entirely at the dean and above level.

  • drugmonkey says:

    why we want to know number of people tryingto feed at the trough?

    Did you read through to the end of the post? Did you read my first one on this topic?

    This discussion is premised on the idea that there is something wrong with the NIH extramural granting process and the state of being an NIH-funded investigator at present. If you don't think this is true, move along.

    If you think this is true, then you are either concerned about fixing it, or not. If not, move along.

    If you are concerned about fixing it, then you should be (my assertion here) concerned about understanding just where the problems lie, what are the implications of different proposed fixes, etc. If you are not, move along.

    The reason is that different fixes may be able to address one problem but not another. Or they may be differentially effective at solving different aspects of the problem. etc.

    This was the point of Rockey's "myth-busting" post. If the "problem" is too many investigators with >2, >4, >6 grants (or you can make it total direct costs if you like)...then you need to know how many of them there are and whether taking the money / grants from them will have any effect on "the problem".

    What I am trying to grapple with here is the conventional wisdom that there are too many investigators. More pertinently, that the doubling led to a massive increase in the total number of investigators. Reason being, if this is supported, then we need to get a significant number of warm bodies out of the system.

    If a lot of the problem (and here I am addressing grant success rates, paylines, etc) is that the same investigators have to write a lot more applications to get the same amount of money at a stable and predictable rate, then there are other fixes that will suffice.

    Look at it this way...if you have a lab that has enjoyed, on average, $500K in direct costs almost uninterrupted for the past 20 years then the system has validated this lab's presence and contributions within the system. Any time the PI has spent chasing the grants, instead of doing the science, in the past 20 years is an inefficiency in the NIH process. and a waste of money.

    One on one you can say that "who can predict so we need this process". But if you said "huh the data show that we have something like 75% of our 20 yr funding vets in this category of continuous stable funding" well then you can start to assume you could stand some reform.

  • drugmonkey says:

    I'm not sure I agree with your intuition here.

    68% more though?

  • Jim Woodgett says:

    In Canada, ~70% of CIHR grantees hold a single operating grant they represent about 55% of the total funding in that category (bear in mind avg CIHR grant is about $120K but doesn't contribute to PI salary). Would be interesting to know if the distribution of RO1's in terms of # per PI has changed as the funding line has dropped. This will have impacts on total applicant #, consistency of funding and application numbers. It's a downward spiral fueled by reduced funding lines (and reduced purchasing power per funded grant).

  • Grumble says:

    "Look at it this way...if you have a lab that has enjoyed, on average, $500K in direct costs almost uninterrupted for the past 20 years then the system has validated this lab's presence and contributions within the system. Any time the PI has spent chasing the grants, instead of doing the science, in the past 20 years is an inefficiency in the NIH process. and a waste of money."

    Bingo.

    It's a simple question: are even old NIH hands writing more grants per awarded grant nowadays than 10 years ago? Or is that a problem specific to noobies? If NIH won't part with this data, one way to get it would be to survey PIs. But that takes time and money. Hey, maybe we can write a grant to NIH to get it...

  • zb says:

    I was actually trying to ask a much more specific question -- not why is this all importsnt anyway, but what number do people need to know to address the problem. (well, and what the problem'is -- is it the time wasted writing and reviewing grants? is it the insecurity of the grant-funded career? is it a decrease in the quality of science done?)

    Is the number we want to know the number of grants submitted by funded investigators with different lengths of funding history? (and the their success rate ) and the changes in the number over time?. I guess I'm looking for a mockup of the graphs that would answer the critical questions.

    I actually think NIH is not purposefully obscuring the information, though I think they have different interests than the investigators they fund. Therefore, i think if a particular data set was requested, there's a reasonable probability ofgetting the info. I do think that NIH correctly believes that any system that sounds like tenure for government funding will not be politically viable.

  • HP says:

    As a recipient of steady RO1 funding over >20 yrs, usually at >$500k direct, I have had to submit more applications (RO1, R21, PO1) in recent years -and spend the time writing them- than I would think reasonable to keep the ship afloat. If productivity is to be measured by the number of publications in decent journals, then the NIH has been getting a very decent return on their investment in my lab (publications since '92: >350; h-index:>100) based on all reasonable criteria. This is true especially in comparison with the accomplishments of IRG panel members who shoot down applications that -in my own judgment- are at least as good, if not better than what I submitted 3, 5, 10 years ago. It is galling to see that -over the period of a single RO1 5-year cycle - my lab typically publishes more papers in decent journals (IF>8) than some of the panel members who assess my proposals and productivity have produced over their entire careers. The IRGs that I have served on generally have low participation of seasoned, senior investigators, and so the argument that 20 years of continuous, sustained productivity -including the most recent progress reports and publications- should stand for something mostly falls on deaf ears. I am convinced that being well-resourced is a strike against an otherwise productive applicant, and that IRGs would rather reward pedestrian, solid work from the single and only RO1 renewal applicant (the sympathy vote) than take a real hard look at the expected return on the investment.

  • drugmonkey says:

    Maybe the disdain in which you hold them comes across on the page? (Or your competition is with the other applicants, not an epic struggle with reviewers.)

  • Dave says:

    To be fair he does have a point about some reviewers. When your grants are evaluated by oldies who haven't had an NIH grant for years and rarely publish, I'm not sure that's ideal and its understandable that resentment sets in. I think reviewers should have to have NIH funding to sit on study sections.

    And not everything can be solved by better "grantsmanship".

  • drugmonkey says:

    I have never been on a panel with anyone who matches that description. Nor can I recall ever seeing such a person pop up in rosters reviewing my proposals.

    If anything, the less vigorously engaged scientists are more likely to be snowed by the latest, greatest cutting edge shine jobbe. IME.

    OTOH I am quite familiar with an entitled, senior PI phenotype that doesn't realize everyone else has a great idea and does great science. And that we can't fund all of them. And that junior and mid career folks are also spending far too much time writing applications for funding. Those types most certainly do exist. IME.

  • Dave says:

    I'm sure that's true as well. No doubt.

    But you saw my summary statement. Only 1 of the 5 (not including the chair) on the SEP had a PI RPG, at least according to reporter, and the 1 that did is a bit of a BSD-type and is the only one publishing regularly in relevant journals (duh!). I had never heard of any of the others and nor had my boss.

    All were full professors and one was even a fucking dean of god-knows-what so probably doesn't need grants anymore.....or something. I just think it's an interesting observation when non-NIH grant holders are reviewing NIH grants.

  • HP says:

    IME, a lot depends on the ability of the SRO's to rope in competent reviewers. I think that IRGs with a long and strong tradition of involving leaders in their fields in the review process do a better job than IRGs thrown together haphazardly (and have an easier time recruiting reviewers that have the necessary credibility among the applicants) . IRGs that have shown a historic tendency to protect their turf and exclude newcomers are not exactly unknown. I'd be curious to know how the stats work out for IRGs in established fields and active for a long time, versus the panels put together on a more ad hoc basis

  • HP says:

    DM:" OTOH I am quite familiar with an entitled, senior PI phenotype that doesn't realize everyone else has a great idea and does great science."

    define great science

  • Spiny Norman says:

    ...though I am hoping to work my way up to this.

  • Pinko Punko says:

    My limited experience is in line with DM RE: SRG. I do regularly hear comments like HP's (for example, from former PI) about incompetent retread study sections. Conversely, my experience with the seasoned, experienced, senior reviewers is that they are also the ones more likely to favor track record over actual proposal and names on the grant (or pedigrees) with a little more weight than the words on the page. So I don't think we solve problems this way. I think we solve some people's problems. At 10% paylines, once you hand out all of the "trust this dude, great track record" grants, there is nothing left. And did I say "dude"? That is another likely subconscious problem with the masters of the universe review panels.

  • drugmonkey says:

    once you hand out all of the "trust this dude, great track record" grants, there is nothing left

    And ain't that the truth. Nobody of the BSD variety seems to understand this as it applies to them.

  • HP says:

    reading his posts, it's hard to escape the conclusion that DM, too, is a proud charter member of the BSD variety....we could also do with fewer of the sophomoric phallic allusions

  • whimple says:

    HP: Define great science

    "Great science" is science that founds new fields of exploration rather than extending existing fields, no matter how competently. Great science changes how we think, not just what we know.

  • Spiny Norman says:

    That's pretty Kuhnian. But paradigm-breaking science cannot occur in a meaningful way without periods of consolidation.

  • whimple says:

    True. Apparently DM was more talking about "good" science rather than great science. You can't throw a rock without hitting at least three people doing good science.

  • Drugmonkey says:

    Great in the sense that the BSDs think that their stuff is great and deserves funding at whatever current payline exists. I think equivalent science is proposed to at least 2X the payline at present. Maybe 3X.

  • Pinko Punko says:

    Also keeping in mind that a healthy scientific ecosystem builds the framework for major new discoveries- you know all the good not great science that the great science stands on top of- a balance between incremental and revolutionary is good.

  • Grumble says:

    once you hand out all of the "trust this dude, great track record" grants, there is nothing left

    That's no reason not to consider a track record-based grant system - or, more accurately, a system that reduces grantwriting workload for experienced PIs by awarding some portion of funding via grants that don't require a detailed proposal, but are evaluated solely on track record. Such a system would not work at all if PIs were simply awarded every cent they asked for. (I'd just ask for $1 billion.) There would have to be an upper limit per PI, and that limit should take into account how much is leftover for traditional grants.

  • Pinko Punko says:

    Grumble, this system already exists in practice, because grants are written with obviously little effort and state things like "our track record is convincing". A 12 page executive summary is not too much to ask for accounting of dollars. There are also Merit grants.

  • drugmonkey says:

    Not only does the program-based system work in practice, there's a good chance that wagon circling behavior in these tough times is worsening this trend. And yet the old established types who think they deserve everything that they request (as they have been accustomed to expect) yammer for even *more* program-based evaluation/award.

    Mid career folks fall into this trap because they think it is *their turn* and that formalizing program-based support will give them the bennies the older generation enjoyed. I agree it is *my* turn (natch) but am not certain expansion of program-based support is the way to go. I have other fixes that I think would make it fairer for all, including me.

  • SidVic says:

    HP “As a recipient of steady RO1 funding over >20 yrs, usually at >$500k direct, I have had to submit more applications (RO1, R21, PO1) in recent years -and spend the time writing them- than I would think reasonable to keep the ship afloat. (publications since '92: >350; h-index:>100)”
    Well, that is obviously a good run! But, HP, what does that put you at -60 ish with approximately a paper published every month for the last 20yrs? Personally I’ve noted a decline in my creativity and the flexibility of my intellect from my 20s (now eary40s). I’m looking for an exit strategy by my 50s. It looks something like 100-150k of non-competitive funding and hard money salary for my golden years (60yrs on). You know, a technician or graduate student if they are bright and don’t get on my nerves too much.
    Of course I’ll probably sing a different tune 20yrs on. With the crunch in funding it is natural that we are at each other with knives drawn. That crunch is the underlying problem. OTOH, after witnessing the "doubling" i have faith that the biomedical complex could phagocytise 10X the amount and we would be in the same place in a couple of yrs.

  • Spiny Norman says:

    Another problem with HP's view is that the current cohort is not, on average, getting its FIRST RO-1 until about age 44. It is a whole lot more likely that one will rack up 200+ papers and an H-index >50 if one has had a hive of 5-20 worker bees making honey for the PI to put his name on. The difference between building a productive group in one's early or mid 30s vs. one's mid to late 40s is vast. The major reason that the latter is more common now has nothing to do with today's investigators sucking and everything to do with structural trends in the workforce.

    Me? I'd kill off most of the PO-1, UO-1 etc grants...

  • drugmonkey says:

    Right up until you get senior enough to compete as P01 PD, amirite Spiny?

  • Spiny Norman says:

    No. Not correct. We are in the preliminary phases of putting one together now. You go where the money is but you surely do not have to like it.

  • drugmonkey says:

    Do as I say, not as I do?

  • Spiny Norman says:

    I don't determine how NIH structures its funding mechanisms, DM. If you're a basketball player and you don't like a particular ref, it's not like you're going to walk away from the game.

  • Grumble says:

    Pinko:

    "grants are written with obviously little effort and state things like "our track record is convincing"."
    It's only the biggest of the BSDs that can get away with this. In fact, I've never seen it, and I've reviewed grants from some pretty big pooh-bahs. So I totally disagree with you that the system in place is anything at all like what I proposed.

    "A 12 page executive summary is not too much to ask for accounting of dollars."
    It's too much to ask if you have to submit 10 of them to get 1. You make it sound like a 5 minute breeze to put together a 12 page grant. Even for experts, it's not.

    "There are also Merit grants."
    There aren't enough of them, and again only the biggest of BSDs get them. Expansion of this program might be a good thing, though.

    DM:
    "Mid career folks fall into this trap because they think it is *their turn* and that formalizing program-based support will give them the bennies the older generation enjoyed. "
    Trap? You have yet to explain why it's a trap, or a bad idea. You've complained that it's not fair to the young 'uns, but I don't think it has to be unfair to them. If one just took the current allocation of $ to old farts, mid-career aspirants, and total noobies, and then simply freely awarded (based on track record) some of the money that old farts/mid careers now have to waste gobs of time writing grants to get, would that really be so unfair?

    For instance, there could be a piece of the pie dedicated just to noobies, so that the funding rate would be exactly the same under that system as it is now. Then, as they prove themselves, they are rewarded with more stable funding and less grantwriting effort. Why not?

  • Pinko Punko says:

    Grumble, I just don't see how you make sense. PIs with very good track records do not have to write 10 grants to get 1. Perhaps we are not clear on what level of track record gets you into the magic money pile. The less money there is, it makes sense to justify the funding to greater and greater extents. If paylines were 30%, it might be possible for what you say to be reasonable, but I don't see how it can be justified when there are no dollars. Many more people have proven themselves than there is funding for.

    In my experience, I have seen a lot of "trust me" grants, so we differ on that one.

  • Grumble says:

    Maybe experienced PIs have a better than 1:10 funding ratio, but the whole discussion was started with the point that we don't know what that ratio is, and whether it has gotten worse lately even for experienced PIs.

    And I don't really agree with the sentiment that "the less money there is, [the more] it makes sense to justify the funding." If everyone is wasting time writing grants that go unfunded rather than doing science, this just makes for less productivity under precisely the conditions where we need every dollar to stretch farther. Unlike the stock market, in science, past performance is a strong predictor of future results, so why not use that indicator?

    (Here's an idea: give experienced PIs a choice between (a) small-but-consistent awards that come with the stipulation that the PI can't have concurrent R01s - or can't have more than $X in additional NIH funding - or (b) the freedom to compete for R01s in the pool with the rest of us. Again, the basic idea is to keep allocation of money to about the same proportions among different classes of competitors as currently, but reduce the workload and anxiety level for at least some people.)

  • [...] Against the cultural anthropologists 3 British Moths Extinct; Most Other Species in Decline London’s new ‘Micrarium’ aims to showcase the tiniest members of the animal kingdom (this sounds amazing) What You Need to Know About Genetically Engineered Food Reconsidering the “too many mouths at the NIH Grant trough” hypothesis [...]

  • Pinko Punko says:

    Grumble,

    Who will decide and what are the metrics? How will it not be a star chamber? If it makes the pool smaller for those competing for the rest by writing grants, doesn't it make their grant writing more onerous? I'm not trying to be argumentative. It sounds vaguely like college football.

  • Grumble says:

    "If it makes the pool smaller for those competing for the rest by writing grants, doesn't it make their grant writing more onerous?"
    Past performance-based grants would not necessarily make the pool smaller for everyone else if the pool of money dedicated to such grants were limited, and if recipients' ability to obtain further funding through the traditional proposal system were limited. The choice ought to be up to the PI: slow and steady funding based on record, or compete with everyone else based on proposal and promise.

    "Who will decide and what are the metrics? How will it not be a star chamber?"
    Imagine being asked to review a grant based solely on the Investigator and Environment criteria. If an applicant has solid publications and a collaborative environment with good equipment for what you know the PI does, you'd give her a good score. If she consistently publishes outstanding papers that you think move the field forward by leaps and bounds, you'd give her a great score. If she's a hack who publishes the same study over and over again with minor variations, you'd give her a poor score. Then program staff and council would decide who gets funded and by how much based on the scores and available funds. Under such a system, "good" but not "great" PIs could still get funded just as they do now, so it doesn't have to be a "star chamber" any more than the current system is.

  • drugmonkey says:

    This will increase the dependence on the three assigned reviewers highly subjective opinion, won't it? Either that or accelerate GlamourDouchery. Either way, not good.

  • Ola says:

    As has been said before further up the page, I would bet a significant portion of additional applications are directly attributable to increased pressure from deanlets. Expectations for salary coverage from grants never seem to go down, only up. Sure, the actual # of funded 1RO1 vs. 2RO1 PI's is not much different nowadays, but more people are "trying" to become 2RO1 PI's.

    I used to be one of those Small-Town-Grocer single RO1 type PI's until about a year ago, when it came time to renew my single RO1. At that time our administrators revised the rules on salary expectations, writing in rules on salary cuts if coverage drops below the required amounts. I am now wholeheartedly a fan of the "crap-shoot approach" - when they're funding 1 in 20 grants, you'd better submit 20 grants even if you want to stay a single Ro1 PI.

  • hidde ploegh says:

    "This will increase the dependence on the three assigned reviewers highly subjective opinion, won't it? "

    Isn't that already the case? How many IRG members read applications other than those assigned to them? How many can judge the validity of R1,R2's assessments based on R1's honey-tongued oratory or R2's poor syntax/broken English? How often does one run into an IRG member who casts a quick glance on the biosketch, raffles through the specific aims and then endorses one or the other extreme opinion?

  • Grumble says:

    IMHO, grant scores are already based on three reviewers' highly subjective opinions. I think one can objectively rate a grant as Good or Bad, and everything within those levels is entirely subjective. When fine differences are used as a basis for funding decisions, the end result is a bullshit-acious crapshoot.

    The same Good vs Bad objectivity resolution level would probably be the case with scoring based on past performance - but the critical difference is that presumably a much greater proportion of applicants would get funded. So the system would be less subject to subjectivity.

    As for GlamourDouchery, it really doesn't have to be the case that glamour mag pubs become a requirement for funding, just as it isn't the case now.

  • drugmonkey says:

    hp-

    Yes and no. The reason that StockCritiques appear so commonly is that they are reinforced by the communication with the rest of the panel. These tend to focus on rapidly verifiable parts of the app that the panel can rapidly identify. Values of "good but not great contribution to subsubsubfield" do not help to decide when the assigned reviewers are 3 pts apart on score.

  • Pinko Punko says:

    Hi Hidde,

    Your points are well taken, but arguing we already have a system that is somewhat arbitrary based on human failings as possible support to make it more arbitrary (increase the subjectivity of the criteria by reducing the paper trail or making it more cosmetic). If you are the famous and highly accomplished Hidde Ploegh, when you are on study section do you read additional grants?

    I definitely try to.

    The points you mention about grant scores being sensitive to the oratory of the reviewer and possible bias (whether subconscious or conscious) against non-english speakers as presenters I do think are valid, based on my limited experience. One would hope that the section chair would be active in supporting the discussion, and that other panel members would speak up. If I were the boss of the section I would have quaternary review assignments where the assignment would relate to having to read x grants outside of ones pile and the submitted critiques, with no written assignments except possibly a check box or form ranking the critiques as applicable. And these assignments would be much closer to the actual expertise of the reviewer. It would be more work, but it would essentially be the work that you would want to happen in the section anyway.

  • drugmonkey says:

    Reviewers who are "more expert", if present, chime in during discussion frequently, Pinko. IME. Obviously they have to be polite about it.

    But in the broader scheme your proposal misses a fundamental GoodThing about having some slightly peripheral expertise. These people can often do the BigPicture in a way the subfield pedant cannot.

  • dsks says:

    The more I think about it the less facetious it seems to simply use the peer process to narrow down to the best 50% of grants and then just have the PO pick the winners out of a hat. The only other necessary rule would be that a PI is restricted to one application per grant mechanism per cycle (e.g. 3 RO1 applications per year).

    Arguments:

    1) The empirical data suggests that the current system already operates as a lottery above the basic triage of the objectively obvious dross applications (which is consistent with historical evidence and philosophical theorizing that scientists are as useless at predicting where, when and how significant scientific breakthroughs will happen as anyone else and their Magic 8-balls).

    2) We wouldn't have to argue about A1s and A2s because nobody would care how many times the same application has been submitted (so long as it is still good enough to make the top 50%). This would mean a lot less grant writing because the same grant could be submitted with very few alterations for as long as it consistently gets rated in the top 50% making it into the hat. And the incentive to simply use the extra time to write more grants in order to up one's chances would be restricted by the 1 application per cycle criterion.

    3) The whole process would be race/gender/age/sexiness blind.

    4) Savings on administrative costs by removing the need for study section meetings would probably fund a good few extra RO1s or so

    Arguments against:
    1) It's a ridiculous way of running the scientific endeavour

    Rebuttal to the arguments agains:
    1) Yes, it is ridiculous. And yet arguably no less so than the current model, but a damn site cheaper in terms of time, labor and capital.

  • Pinko Punko says:

    DM,

    My proposal doesn't mean to do that- I like peripheral expertise as main reviewers (not all), but if there is one reviewer that is much more on top of the subfield, that person can dominate the other two, and if the panel has not really followed along, they might go with that dominant reviewer. My proposed system is a little bit of a backstop on that. In my limited experience, I thought the section was a little bit more on the quiet side than I expected, but I also though reviewers were extremely good. That said, I felt that reviewers who were quieter or english was sub-optimal did seem to generate less enthusiasm from the panel.

    As to dsks, I don't think it is a lottery among the top 40%. I think it is a lottery between 10ish-30ish.

  • Pinko Punko says:

    Adding, the point was not to change how first three reviewers were assigned, but to make sure quaternary review was not exactly random.

  • Grumble says:

    I think it's a fine idea, dsks. Why should NIH pretend that its funding process is rational, when we all know that it's largely stochastic? Why not just make it stochastic explicitly?

  • drugmonkey says:

    Why not just make it stochastic explicitly?

    "Congress"

  • dsks says:

    "As to dsks, I don't think it is a lottery among the top 40%. I think it is a lottery between 10ish-30ish."

    You're probably right, I was rather over-extrapolating Berg's graph. Top 30% then.

  • Grumble says:

    Has anyone ever asked Congress? No, because NIH bureaucrats utterly lack the creativity and insight to come up with a plan to change the agency and then ask Congress for approval.

    Not that said approval is even necessary - is the current review system REALLY written rigidly into law? I doubt it.

  • Pinko Punko says:

    Changing anything with money like this is pretty meaningless, I have a feeling. I also think the perception is that the Grumble method will maintain or extend any current disparities. I guess it doesn't hurt to talk about it.

  • miko says:

    I thought we were going to fix the NIH.

  • lurker says:

    Will the real HP please stand up? Certainly not the BSD otherwise know as Hidde Ploegh!

    As of now, the Ploegh-meister has, count them, 3 R01's, only one is modular ($1.6M) AND a DP1 ($975K) AND a R21 ($292K) as a little cherry on top. One lab gets to eat $2.87M in directs (IDC's - cha-ching!) from the NIH teet.

    The guy is a grant writing god, and meanwhile he laughs upon all of us piglets squealing on this blog. Especially @ Pinko Punko. I'm sure a BSD like him would grace us with a posting on DM's blog!

  • Pinko Punko says:

    Oh HP and Pinko P go waaaaaaay back. Kidding, but he used to wear these glasses that we joked made him look like a certain Raiders of the Ark, medallion-hunting, face got melted bad guy.

  • Spiny Norman says:

    "is the current review system REALLY written rigidly into law? I doubt it."

    You might be *very* surprised.

  • [...] many people are applying for NIH funding, [...]

  • Dave says:

    Nice post. (See? I'm not always a critical crank)

  • Eli Rabett says:

    Very late, but we have discussed this before and Eli just found the smoking gun. Something like 80% of all doctorates are awarded by the very high research activity universtities. Cutting down the number is going to have to start at the top

  • Grumble says:

    'Something like 80% of all doctorates are awarded by the very high research activity universtities. "

    Is this supposed to be surprising? Universities with research labs have graduate students in the sciences. Those without labs don't have science graduate students. Graduate students eventually get degrees.

    Duh?

  • Eli Rabett says:

    No, fucker, it means that if you want to do something about this you are going to have to shut down the University of California, or restrict the size of graduate programs at a whole bunch of R01s. Closing up the weaker programs don't do shit.

  • qaz says:

    New data from RockTalk... it's not new institutions showing up. It's old institutions adding applicants.

    http://nexus.od.nih.gov/all/2013/03/22/more-applications-but-how-many-research-institutions/

    Which supports the theory that already-NIH-funded institutions added faculty. (I suspect soft-money faculty. Following the theory that if the new faculty got grants, they would increase the institution's bottom line, but if they didn't then the institution wasn't out too much. Since they didn't have to pay them.)

    Anyway, this was a big debate earlier in this thread. Wanted to point out that new data was here.

  • drugmonkey says:

    Not sure how you come to this conclusion.

    First, there was a 58% increase in applicant institutions for RPGs from late 90s to now. Interesting bit is that there was only a 15% increase in *successful* institutions. (obvious bubble in between seems to have deflated, of course).

    But even so, if the breakdown here is between institutions with 1 or more prior application and institutions with zero it doesn't give enough resolution. I'm would be unsurprised if there is a big threshold/hurdle at going from zero to one. Think of all the administrative stuff that has to be in place.

    The real question is if the small time university has ramped up the number of applications. This needs to be broken down by ranking Universities by the number of apps submitted. Then we can see if the increased applications are proportional, coming from the traditional greedybags universities (TGBU) or coming from the wannabes.

  • [...] at DM’s there is an ongoing discussion about mouths at the trough. The latest bit has generated some good discussion (and the some of the [...]

  • Rolf Kilker says:

    This is usually a excellent A to Z on how to establish a strong online presence! Good info, very newbie-friendly, comprehensive and thorough!

  • Sweet blog! I found it while searching on Yahoo News.
    Do you have any tips on how to get listed in Yahoo News?
    I've been trying for a while but I never seem to get there!
    Cheers

  • uncynikal says:

    casi ...naro

  • […] slipping purchasing power*) with dismal success rates. Too many mouths at the trough, is the way I put the situation, even if I don't specifically blame the doubling interval for […]

Leave a Reply