I keep mulling over the data presented in this entry at the Rock Talk blog. I originally concluded that this, combined with the revelation that applications-per-PI only went from 1.2 to 1.5 across the FY98-FY11 interval, showed that the RealProblemTM at the NIH was the growth in the number of Principal Investigator mouths at the trough.
As a reminder, these are data for the investigator-initiated Research Project Grants only and exclude the ARRA largesse. The graph shows change data from the baseline of Fiscal Year 1998. As a brief summary of my prior thoughts:
the post indicates an increase from an average of 1.2 applications submitted per investigator to an average of 1.5 per investigator from 1998 to 2011...This surprised many of us on the Twitters. I don't think I know of any active scientists who are submitting less than several NIH grant applications per year...if we harken back to some data on..the Rock Talk blog (maybe; UPDATED, it was RockTalk) which showed the average NIH PI had only about 2 grants concurrently then we must consider that there are still a LOT of folks out there on a single grant at a time. Especially if they have long-term continuations going, sure, maybe there are a lot of PIs who only have to submit an application once every 5 years. The post also indicates that there were 19,000 applicants in 1998 and this grew to 32,000 in 2011. Some 13,000 new mouths, a 68% increase in PIs seeking money from the NIH.
I've added emphasis to highlight what has been bothering me.
The notion that we have 68% more PIs seeking money from the NIH should have been more of a warning. The thing is, it dovetails nicely with one of the very truthy memes that we have going about the effects of the NIH Doubling interval. More people in the system made a certain sense. Particularly for those of us who were entering the system approximately during the doubling interval and did not feel as though it was easy to get a grant funded. Certainly, success rates did not double. In the historical sense the success rates were only moderately restored from a slide that ran from the late 80s (40% for experienced applicants. Think about that.) to about 1994 (25% for experienced applicants). So if the budget was doubled and success rates were far from doubled, there must be more people seeking funding. Right?
What never seemed possible to me was that traditional research-heavy Universities, who were already deep into NIH-addiction, were throwing up that many new jobs. Sure, they expanded their soft-money faculty positions a little bit...and let the occasional word-salad-position Assistant Adjunct Research Project Professor of Bunny Hopping upjumped postdoc submit a grant or two. But it didn't seem likely to me that this explained the budget/payline disparity. Nor, in context of Rockey's data, did a 68% increase in PIs at such places seem likely. So I was always asserting that the growth came in large part from the entry of new institutions into the system. In the sense that smaller, less research intensive Universities were, perhaps, putting on a big push to get in the NIH game. Perhaps this was by hiring new NIH-honcho faculty. Perhaps by pushing hard from the deanlet level to get the existing faculty to submit more grants, bring in more NIH moola. This latter hypothesis was fueled by rumor of this kind of behavior from some of my colleagues and friends so I was primed to believe it.
My new realization of the week is that the data from Rock Talking are misleading. The denominator for the grants-per-PI is calculated on a per-FiscalYear basis. It has to be, even though they don't say this. So you only get counted if you've submitted at least one grant in the FY. Similarly, the growth in the number of PIs from the 1998 baseline is likewise a reflection of the number of PIs submitting at least one competing grant application in a given FiscalYear. Again, they don't specify. I was perhaps assuming that this reflected the number of PIs in the system, i.e. submitting competing or noncompeting applications. In some senses, we also have to keep in mind the number of occasional applicants to the NIH...hard to believe from my perspective but sure, why not consider that there is a pool of PIs who may have repeated, but not continuous funding from the NIH across their careers?
Keep in mind that I'm eventually getting around to the consideration of the massive decrement in the purchasing power of the standard, $250K direct cost, full modular award.
As you can see, a full modular $250,000 year in 2011 has 69% of the purchasing power of that same award in 2001.
We'll return to this.
Let us start with consideration of what appears to be, going by disgruntleprof comments on various blogs and opinion pieces, the shining virtue of the NIH system...the one-R01 small town grocer. This PI submitted a grant application once every five years to continue her R01...in the old days. So on average this person would be submitting 0.2 grants per FY but in the Rockey analysis would only count as 1 grant-submitted...every five years. Over time, however, she is now facing a decreased probability of getting funded the first time and, let us say, submits an application three times (A2 scenario, not unlikely at all by the end of the doubling), a year apart, during her 5 year window. Her Rockey number is still 1 application per year but her 5 year average has increased to 0.6. Similarly, since we're dealing with the one-grant scenario, the appearance in the Number-of-Mouths data is likewise affected by the frequency of submissions. Taking the 3 tries case again, if she only had to apply twice every 10 years in the past but is now applying 6 times to maintain funding, she has tripled her presence in the Rockey way of looking at the number of applicants. If we're talking about an overall 68% change over time...this kind of behavioral change is significant if it occurs in any appreciable part of the PI distribution. It makes it look like there is a big change in the number of PIs that need to be fed when there have not, in fact, been two more PIs added to the system.
Getting back up to my original thoughts on where the RealProblemTM lies, however, this is all critical. Is the NIH in fact supporting 68% more investigators in 2011 vs 1998? This is what Sally Rockey's post would imply. It certainly implied this to me. However, it may simply reflect the same number of overall PIs in the NIH-funded extramural workforce who simply have to submit more grant applications to maintain the same number of grants.
Which brings me to my next point. Note that I said "same number of grants" but not "same amount of funding". Because it is also clear that over this self-same interval when SmallTownGrocerPI was forced to submit applications more frequently to sustain her funding, she was also forced to try to get more awards simply to maintain the same level of operation. Because the purchasing power of the grant dollars had fallen by so much and yet the full-modular cap still imposed a de facto limit on budget escalation. 
Now true, the "myth-busting" data from Rockey show only a 4-5% shift in 1-grant to 2-grant PIs from FY1986 to FY2004 when the doubling was rolling hard. This is where the simple case we are discussing really breaks down. Obviously there are many varieties and mixtures of PIs in terms of the number of applications submitted, the stable-versus-growth aspirations, the amount of NIH funding that represents stable state, the mixture of R01 and "other" funding, etc.
So obviously it would be a complex modeling job in the NIH databases to get the best understanding.
But it strikes me that one of the simplest and most productive things for the NIH (read: Sally Rockey's data mining minions) to do would be to take a closer look at the number of PIs applying instead of the number of applications. The number of PIs over an extended window of time, not just on a per-FY basis.
This reason that this is important to know is that the success of any proposed fixes to the NIH depend on this reality. If there has genuinely been an increase in the number of PIs then shelling some of them out of the system permanently (including by preventing entry) is the only way to have sustained effect. Within that category, it may be necessary to see if the growth in PIs has come from the top research Universities or from increases in the lower-tier Universities.
If the main trouble is the uncertainty of maintaining one award, then the solutions are to extend the interval of non-competing and/or give a much larger payline break to competing continuations versus new applications.
If the trouble is that the purchasing power of the full-modular has decreased, then boost the limit to $375K per year in direct costs. [ETA: per comment from Grumble, note that the purchasing power has also been eroded by habitual budget reductions upon funding. Some ICs cut a whole year. Some have made 1-2 module ($25K per module) reductions the SOP. Some hit even non-competing renewals with additional reductions because of budgetary uncertainty. They do this to artificially prop up the success rates. Take one module from 9 awards and you can fund 10.]
It is incredibly frustrating for those of us who watch from the outside since these data are clearly available within the NIH databases and they simply seem to be looking* in the wrong direction.
__
*I realize that Sally Rockey may have a ton of analyses that she simply has not put up on the blog. Somehow, given her little oopsie with the alternative career fate of trainees, I doubt it.
