What fraction of the stuff proposed in funded grants actually gets done after feasibility and field movement come to play?
Archive for the 'NIH Careerism' category
Datahound has some very interesting analyses up regarding NIH-wide sex differences in the success of the K99/R00 program.
Of the 218 men with K99 awards, 201 (or 92%) went on to activate the R00 portion. Of the 142 women, 127 (or 89%) went on to these R00 phase. These differences in these percentages are not statistically different.
Of the 201 men with R00 awards, 114 (57%) have gone on to receive at least 1 R01 award to date. In contrast, of the 127 women with R00 awards, only 53 (42%) have received an R01 award. This difference is jarring and is statistically significant (P value=0.009).
So per my usual, I'm very interested in what the ICs that are closest to my lab's heart have been up to with this program. Looking at K99 awardees from 07 to 09 I find women PIs to constitute 3/3, 1/3 and 2/4 in one Institute and 1/7, 2/6 and 5/14 in the other Institute. One of these is doing better than the other and I will just note that was before the arrival of a Director who has been very vocal about sex discrimination in science and academia.
In terms of the conversion to R01 funding that is the subject of Datahound's post, the smaller Institution has decent outcomes* for K99 awardees from 07 (R01, R21, nothing), 08 (R01, R01, R01) and 09 (P20 component, U component, nothing, nothing).
In the other Institute, the single woman from 07 did not appear to convert to the R00 phase but Google suggests made Assistant Professor rank anyway. No additional NIH funding. The rest of the 07 class contains 4 with R01 and two with nothing. In 08, the women PIs are split (one R01, one nothing) similar to the men (2 R01, 2 with nothing). In 09 the women PIs have two with R01s, one R03 and two with nothing.
So from this qualitative look, nothing is out of step with Datahound's NIH-wide stats. There are 14/37 women PIs, this 38% is similar to the NIH-wide 39% Datahound quoted although there may be a difference between these two ICS (30% vs 60%) that could stand some inquiry. One of 37 K99 awardees failed to convert to R00 from the K99 (but seems to be faculty anyway). Grant conversion past the R00 is looking to be roughly half or a bit better.
I didn't do the men for the 2009 cohort in the larger Institute but otherwise the sex differences in terms of getting/not getting additional funding beyond the R00 seems pretty similar.
I do hope Datahound's stats open some eyes at the NIH, however. Sure, there are reasons to potentially excuse away a sex difference in the rates of landing additional research funding past the R00. But I am reminded of a graph Sally Rockey posted regarding the success rate on R01-equivalent awards. It showed that men and women PIs had nearly identical success rates on new (Type 1) proposals but slightly lower success on Renewal (Type 2) applications. This pastes over the rates of conversion to R00 and the acquisition of additional funding, if you squint a bit.
Are women overall less productive once they've landed some initial funding? Are they viewed negatively on the continuation of a project but not on the initiation of it? Are women too humble about what they have accomplished?
*I'm counting components of P or U mechanisms but not pilot awards.
Excellent comment from eeke:
My last NIH grant application was criticized for not including a post-doc at 100% effort. I had listed two techs, instead. Are reviewers under pressure to ding PI's for not having post-docs or some sort of trainee? WTF?
I think it is mostly because reviewers think that a postdoc will provide more intellectual horsepower than a tech and especially when you have two techs, you could have one of each.
I fully embrace this bias, I have to admit. I think a one-tech, one-PD full modular R01 is about the standardest of standard lineups. Anchors the team. Best impact for the money and all that.
A divergence from this expectation would require special circumstances to be explained (and of course there are many projects imaginable where two-tech, no-postdoc *would* be best, you just have to explain it)
What do you think, folks? What arrangement of personnel do you expect to see on a grant proposal in your field, for your agencies of closest interest? Are you a blank slate until you see what the applicant has proposed or do you have....expectations?
A query came into the blog email box about how to deal with submitting a new grant based on the prior A1 that did not get funded. As you know, NIH banned any additional revisions past the A1 stage back in 2009. Recently, they have decided to stop scrutinizing "new" applications for similarity with previously reviewed and not-funded applications. This is all well and good but how should we go about constructing the "new" grant, eh? A query from a Reader:
Do you use part of your background section to address reviewer comments? You're not allowed to have an introduction to the application, but as far as I can tell there is no prohibition on using other parts of the application as a response to reviewers.
I could see the study section as viewing this a) innovative, b) a sneaky attempt to get around the rules, c) both a and b.
I am uncertain about the phrasing of the Notice where it says "must not contain an introduction to respond to the critiques from the previous review". In context I certainly read this as prohibiting the extra page that you get for an amended application. What is less clear is whether this is prohibiting anything that amounts to such introduction if you place it in the Research Strategy. I suspect you could probably get away with direct quotes of reviewer criticisms.
This seems unwise to me, however. I think you should simply take the criticisms and revise your proposal accordingly as you would in the case of an amended version. These revisions will be sprinkled throughout the application as appropriate- maybe a change in the Significance argument, maybe a new Experiment in Aim 2, maybe a more elaborated discussion of Potential Pitfalls and Alternative Approaches.
Given the comments, perhaps you might need to state some things twice or set off key points in bold type. Just so the next set of reviewers don't miss your point.
But I see no profit in directly quoting the prior review and it just wastes space.
It has recently come to my attention that not everyone views the no-cost extension (NCE) of a NIH grant the same way I do.
When the interval of support is over for many of the NIH grant mechanisms, the PI (actually the University or Institute, of course) can request a NCE. This means that while the NIH is not going to give out any additional money past the original award, the University may continue to spend any un-expended funds. My experience has been that NCE requests, particularly the first year, are approved by default.
I am pretty sure that you are supposed to follow the usual rules for rolling money from one funded interval to the next funded interval, i.e., it is supposed to be only 25% of the prior year or less. Also, if you have a great excuse for why you have slightly more than 25% left over I would think this would not be a huge problem.
Personally, I have requested a NCE essentially by default for every grant award that permits it and assuming that the competing renewal has not been approved in time to keep the funded intervals rolling.
There are my own local institutional reasons for doing so, mostly having to do with moderate red tape factors.
I thought that it was sort of required in order to submit a competing continuation (now called "renewal") application. I have one grants management assurance that this is not the case but I would still want to check up on how that works. After all, with the new A2 as A0 rules, can't we just submit a competing continuation application for a grant which has been unfunded for 5 years? 10 years? Wait....google...google....NIAID says:
Is there a window of time that a PI can submit an application as a renewal? Must the original grant still be active?
No. The grant need not be active and there is no time limit for a renewal application. However, reviewers will probably be concerned by major gaps.
If a significant amount of time has elapsed, indicate what you have done in the interim. Highlight any preliminary data you may have obtained, and show that your planned research is current with the latest science.
OK, maybe I learned something this year. I mean, I'm sure I have read that before about renewals but somehow it never really connected up.
Also, for some reason maybe I thought that reviewers would be more confident that you were actively working on the project past the end date if you could say it was in NCE.
My question for the peanut gallery today is, how seriously do you take the NCE when you see it mentioned in a Biosketch or elsewhere in the grant proposal? Is it just meaningless...as in "of course they applied for a NCE, duh" or below notice altogether? Does apply only when it is a competing continuation / renewal of the grant which is in NCE?
In a related vein, does it "count" as current research funding? Do you see a grant in NCE and mentally chalk it up under the PI or other Key Personnel's "funded grants"? In the Biosketch does it go under "completed" or "ongoing" Research Support?
Do you assume it might be pending renewal if it is not the prior interval of the grant you are reviewing now?
We've previously discussed the NIH F32 Fellowship designed to support postdoctoral trainees. Some of the structural limitations to a system designed on its fact to provide necessary support for necessary (additional) training overlap considerably with the problems of the F31 program designed to support graduate students.
Nevertheless, winning an individual NRSA training fellowship (graduate or postdoctoral) has all kinds of career benefits to the trainee and primary mentor so they remain an attractive option.
A question arose on the Twitts today about whether it was worth it for a postdoc in a new lab to submit an application.
— Dr Becca, PhD (@doc_becca) May 30, 2014
In my limited experience reviewing NRSA proposals in a fellowship-dedicated panel for the NIH, there is one issue that looms large in these situations.
Reviewer #1, #2 and #3: "There is no evidence in the application that sufficient research funds will be available to complete the work described during the proposed interval of funding."
NRSA fellowships, as you are aware, do not come with money to pay for the actual research. The fellowship applications require a good deal of discussion of the research the trainee plans to complete for the proposed interval of training. In most cases that research plan involves a fair amount of work that require a decent amount of research funding to complete.
The reviewers, nearly all of them in my experience, will be looking for signs of feasibility. That the PI is actually funded, funded to do something vaguely related* to the topic of the fellowship proposal and funded for the duration over which the fellowship will be active.
When the PI is not obviously funded through that interval, eyebrows are raised. Criticism is leveled.
So, what is a postdoc in a newer lab to do? What is the PI of a newish lab, without substantial funding to do?
One popular option is to find a co-mentor for the award. A co-mentor that is involved. Meaning the research plan needs to be written as a collaborative project between laboratories. Obviously, this co-mentor should have the grant support that the primary PI is lacking. It needs to be made clear that there will be some sort of research funds to draw upon to support the fellow doing some actual research.
The inclusion of "mentoring committees" and "letters of support from the Chair" are not sufficient. Those are needed, don't get me wrong, but they address other concerns** that people have about untried PIs supervising a postdoctoral fellow.
It is essential that you anticipate the above referenced Stock Critique and do your best*** to head it off.
*I have seen several highly regarded NRSA apps for which the research plan looks to me to be of R01-quality writing and design.
**We're in stock-critique land here. Stop raging about how you are more qualified than Professor AirMiles to actually mentor a postdoc.
***Obviously the application needs to present the primary mentor's funding in as positive a light as possible. Talk about startup funds, refer to local pilot grants, drop promising R01 scores if need be. You don't want to blow smoke, or draw too much attention to deficits, but a credible plan for acquiring funding goes a lot farther than ignoring the issue.
In my view, once it is on The News Hour then it is really news.
Nature published a commentary by NIH Director Francis S. Collins and NIH Office of Research on Women's Health Director Janine A. Clayton which warns us that the NIH will start insisting on the inclusion of more sex-difference comparisons. These are to extend from cells to animal models across many areas of pre-clinical work.
The NIH is now developing policies that require applicants to report their plans for the balance of male and female cells and animals in preclinical studies in all future applications, unless sex-specific inclusion is unwarranted, based on rigorously defined exceptions. These policies will be rolled out in phases beginning in October 2014, with parallel changes in review activities and requirements.
I cannot wait to see what the "rigorously defined exceptions" will be for several types of research in which I have an interest. Every rat self-admin study must now include both males and females? For all treatment conditions or will it be acceptable to just tack the sex-comparison on at the end?
Furthermore, the NIH will monitor compliance of sex and gender inclusion in preclinical research funded by the agency through data-mining techniques that are currently being developed and implemented. Importantly, because the NIH cannot directly control the publication of sex and gender analyses performed in NIH-funded research, we will continue to partner with publishers to promote the publication of such research results.
oooooh. "partner with publishers" eh? Of course this is because Clayton and Collins realize that higher JIF journals are entirely uninterested in things as pedestrian as sex-comparisons, particularly when the outcome of the study is "no difference". Which, btw, is one of the reasons nobody* wants to waste their precious time and grant money doing something as low-return as sex-comparisons. So somehow the NIH is going to lean on publishers to be...friendlier....to such work. I do hope they realize that this is not going to work. The contingencies are not going to change because the NIH asks. Now, if they actually went all in and dismantled GlamourMagScience culture by the judicious use of grant award, grant auditing and rules about the ratio of publications to effort expended... then we might see some progress. That will never happen and thus there will be no change in the publication contingencies that fight against sex-comparison studies.
Dr. Clayton went on The News Hour where Judy Woodruff asked her (and Phyllis Greenberger of Society for Women's Health Research) some pretty obvious questions. Woodruff wanted to know if there were any clear examples in which women were put at risk or their health suffered because of a lack of such research. She also wanted to know what the implications for research might be- would it be more difficult or more expensive. Finally, Woodruff asked if scientists would resist.
From the transcript:
JUDY WOODRUFF: But how hard is that? Does that mean — is it extra work, is it more expensive? What’s involved in making sure there’s a gender balance?
Now Greenberger snuck in a "Both" off camera but then Clayton went on to be ridiculous and fail to answer the question. The answer is indeed "both" and it is a serious one if the NIH expects to get results. It will be more expensive, progress will be slower and it will be "harder" in the sense of teasing out the right experimental designs and variables so that an interpretable result can be reached. It isn't rocket science, exactly, but it is harder.
JUDY WOODRUFF: Phyllis Greenberger, were there — were there actually individuals who were harmed or where help wasn’t delivered because the research was done only on males?
Greenberger totally walked around this one and Woodruff, to her credit, fronted Clayton with the same question a bit later. Clayton referred to heart attack warning symptoms in women that might differ from men...of course this has nothing whatever to do with preclinical research. Gaaah! So frustrating. Greenberger chimed back in with talk of drugs being removed from the market for adverse effects in women....with no indication that these were adverse effects that would have been identified in female-specific PREclinical research. C'mon NIH! If you are going to take a run at this, please prepare your argument!
JUDY WOODRUFF: And is that the reason that it wasn’t done earlier, Dr. Clayton, that there was just pushback in the scientific community?
The answer is illustrative of the problem at the NIH....
DR. JANINE CLAYTON: It’s hard to say. There are probably a lot of factors that are involved.
And what’s really important now is right now we have been able to put the focus on getting this as a priority. As Phyllis mentioned, the Society and other advocacy groups and scientists and others have talked about this in the past. In fact, we are supporting scientists who are doing this research, but it wasn’t enough of a priority. In some way, it was like a blind spot. Scientists weren’t thinking about it.
Clayton is right. The NIH does indeed support investigators doing sex-differences studies.Those scientists do not have a problem of "priority" from the perspective of their own intrinsic motivation.
With respect to whether scientists resist, I enjoin you to go over to PubMed and type in Sex Differences and see what fill-in choices are offered to you. Click on several of these searches and see what you find. You will find funded projects in many of your favorite domains of interest. If you bother to click on the papers and look at the grant attributions, you may even find that many of these investigations were completed under NIH funding!
So when Clayton (and in the Commentary she is joined by Director Collins) claims it isn't a "priority", it seems misplaced to put this on the shoulders of extramural scientists.
If the NIH wants more sex-differences studies then they need to deploy their tastiest carrot to greater effect. Put out some Funding Opportunity Announcements and see what happens! Fund a few Supplements to the people who are already doing sex-comparisons! Pick up a few grants that missed the payline...again, from the people who are already proposing sex-comparisons!
And if you want to lure in new converts that you didn't get with an RFA or a Program Announcement? This is simple. Just put out a policy that any grant application with a credible stab at a sex-comparison component gets an extra 5 percentile points credit towards the payline for funding.
Just you wait and see how many sudden converts you make!
*of the GlamourMag class investigator
Occasionally, Dear Reader, one or another of you solicits my specific advice on some NIH grant situation you are experiencing. Sometimes the issues are too specific to be of much general good but this one is at least grist for discussion of how to proceed.
Today's discussion starts with the criterion scores for an R01/equivalent proposal. As a reminder, the five criteria are ordered as Significance, Investigator, Innovation, Approach and Environment. The first round for this proposal ended up with
Reviewer #1: 1, 1, 1, 3, 1
Reviewer #2: 3, 1, 1, 3, 1
Reviewer #3: 6, 2, 1, 8, 1
Reviewer #4: 2, 1, 3, 2, 1
From this, the overall outcome was.... Not Discussed. Aka, triaged.
As you might imagine, the PI was fuming. To put it mildly. Three pretty decent looking reviews and one really, really unfavorable one. This should, in my opinion, have been pulled up for discussion to resolve the differences of opinion. It was not. That indicates that the three favorable reviewers were either somehow convinced by what Reviewer #3 wrote that they had been too lenient...or they were simply not convinced discussion would make a material difference (i.e. push it over the "fund" line). The two 3s on Approach from the first two reviewers are basically a "I'd like to see this come back, fixed" type of position. So they might have decided, screw it, let this one come back and we'll fight over it then.
This right here points to my problem with the endless queue of the revision traffic pattern and the new A2 as A0 policy that will restore it to the former glory. It should be almost obligatory to discuss significantly divergent scores, particularly when they make a categorical difference. The difference between triaged and discussed and the difference between a maybe-fundable and a clearly-not-fundable score is known to the Chair and the SRO of the study section. Thee Chair could insist on resolving these types of situations. I think they should be obliged to do so, personally. It would save some hassle and extra rounds of re-review. It seems particularly called-for when the majority of the scores are in the better direction because that should be some minor indication that the revised version would have a good chance to improve in the minds of the reviewers.
There is one interesting instructive point that reinforces one of my usual soapboxes. This PI had actually asked me before the review, when the study section roster was posted, what to do about reviewer conflicts. This person was absolutely incensed (and depressed) about the fact that a scientific competitor in highly direct competition with the proposal had been brought on board. There is very little you can do, btw, 30 days out from review. That ship has sailed.
After seeing the summary statement, the PI had to admit that going by the actual criticism comments, the only person with the directly-competing expertise was not Reviewer #3. Since the other three scores were actually pretty good, we can see that I am right on the assumption of what a reviewer will think of your application based on perceptions of competition or personal dis/like. You will often be surprised that the reviewer that you assume is out to screw your application over will be pulling for it. Or at least, will be giving it a score that is in line with the majority of the other reviewers. This appears to be what happened in this case.
Okay. So, as I may have mentioned I have been reluctantly persuading myself that revising triaged applications is a waste of time. Too few of them make it over the line to fund. And in the recently past era of A1 and out....well perhaps time was better spent on a new app. In this case, however, I think there is a strong case for revision. Three of four (and we need to wonder about why there even were four reviews instead of three) of these criterion score sets look to me like scores that would get an app discussed. The ND seems to be a bit of an unfair result, based on the one hater. The PI agreed, apparently, and resubmitted a revised application. In this case the criterion scores were:
Reviewer #1: 1, 2, 2, 5, 1
Reviewer #2: 2, 2, 2, 2, 1
Reviewer #3: 1, 1, 2, 2, 1
Reviewer #4: 2, 1, 1, 2, 1
Reviewer #5: 1, 1, 4, 7, 1
I remind you that we cannot assume any overlap in reviewers nor any identity of reviewer number in the case of re-assigned reviewers. In this case the grant was discussed at study section and ended up with a 26 voted impact score. The PI noted that a second direct competitor on the science had been included on the review panel this time in addition to the aforementioned first person in direct competition.
I assure you, Dear Reader, that I understand the pain of getting reviews like this. Three reviewers throwing 1s and 2s is not only a "surely discussed" outcome but is a "probably funded" zone, especially for a revised application. Even the one "5" from Reviewer #1 on Approach is something that perhaps the other reviewers might talk him/her down from. But to have two obviously triage numbers thrown on Approach? A maddening split decision, leading to a score that is most decidedly on the bubble for funding.
My seat of the pants estimation is that this may require Program intervention to fund. I don't know for sure, I'm not familiar with the relevant paylines and likely success rates for this IC for this fiscal year.
Now, if this doesn't end up winning funding, I think the PI most certainly has to take advantage of the new A2 as A0 policy and put this sucker right back in. To the same study section. Addressing whatever complaints were associated with Reviewer #1's and #5's criticisms of course. But you have to throw yourself on the mercy of the three "good" reviewers and anyone they happened to convince during discussion. I bet a handful of them will be sufficient to bring the next "A0" of this application to a fundable score even if the two less-favorable reviewers refuse to budge. I also bet there is a decent chance the SRO will see that last reviewer as a significant outlier and not assign the grant to that person again.
I wish this PI my best of luck in getting the award.
Somebody I normally respect is on the Twitts naming and shaming scientists who have lost their NIH funding and are, allegedly*, shuttering their laboratories.
This makes me deeply uncomfortable for the naming and shaming part, one.
More important is the implication that it is somehow a greater tragedy** that people who have enjoyed something on the order of 20 years of NIH funding are now at the end of their careers.
This is nonsense. First of all, if you have the luxury to retire at 60 with a nice fat pension, maybe an Emeritus office to visit, with your kids through college (generationally more likely), house paid off (ditto), etc, etc then retiring "early" is what we used to think of as a huge win. So you stopped publishing science a little earlier than you might have liked. So what. Get a hobby.
More importantly, it necessarily diminishes the tragedy of others who never had funding.
I am not okay with this.
As you know, I've managed to keep my head above water as an NIH funded lab head......so far. And yeah, I feel the greatest affinity for my own continued survival in this career. Sure. Hard to avoid and I don't fault anyone for feeling the same way. What I do fault people for is not realizing at some level how deeply selfish this tendency is. At least act like you understand everyone should have a fair shot?
And that is the point. It has been decades that I have watched the fate of people who might have become NIH-funded investigators of various levels of fame, fortune and pizzaz. Decades over which I have watched the career arcs of people who have enjoyed NIH grant support.
There are a lot of people who should have been PIs with generous amounts of grant support who never achieved this outcome. Lots. There are a lot of people who managed to maintain funding that are clearly no more, and often less, worthy than those who did not enjoy such success.
There has to be at least one unsuccessful young gun of your field that, were you the boss of science, you would put 5 senior investigators*** out to pasture to fund. If you don't know of any people like that, you aren't thinking very hard about it.
So sure, it is a tragedy when a luminary of your field closes shop. It is distressing even when a middle rank plodder has to pack it in. These people are salient to you, I realize. Because they are publishing. They have generated papers that are important to you.
The folks who never had a chance in the first place? All too easy to forget. All too easy to shrug off their failure to become a luminary as fault of their own (they "chose" alt career) or the system (life is hard).
But they are most assuredly a Lost Lab too.
Try not to forget that.
*I say allegedly because we have heard a tremendous amount of rumoring about labs "about to close" because of the dismal NIH grant situation. See this 2009 report in Nature News. As it turned out, one of those researchers was just fine and one was picked up at the time but didn't learn the proper lesson.
**I am also unimpressed by the recitation of these Lost Labs' publication record by Journal or by the lab in which that PI was trained. As if that tells us anything about how tragic it is or is not to have lost their labs. Please.
***Naturally even within a subfield, not everyone agrees on who the most promising young gun(s) is/are and not everyone agrees on who are the 5 dispensable peers. I am not suggesting this Lost Lab situation is easy to fix.
...need to be ended.
They represent a huge risk for bias dependent on the personal characteristics of the investigators to rule the day.
Are you an older, white-haired, heteronormative appearanced, able-bodied picture of "Scientist and Professor"? Great!
Are you overweight? Do you stutter? Express unexpected gender presentation? Nonwhite? Female? Are your language skills less than native to the reviewer's ears? Too young? Too hot? Not hawt enough? .... Not so great.
Should your grant proposal be affected strongly by the direct face to face impression of these characteristics?
Update: See PDF for site visit procedures