What fraction of the stuff proposed in funded grants actually gets done after feasibility and field movement come to play?
Archive for the 'NIH' category
Datahound has some very interesting analyses up regarding NIH-wide sex differences in the success of the K99/R00 program.
Of the 218 men with K99 awards, 201 (or 92%) went on to activate the R00 portion. Of the 142 women, 127 (or 89%) went on to these R00 phase. These differences in these percentages are not statistically different.
Of the 201 men with R00 awards, 114 (57%) have gone on to receive at least 1 R01 award to date. In contrast, of the 127 women with R00 awards, only 53 (42%) have received an R01 award. This difference is jarring and is statistically significant (P value=0.009).
So per my usual, I'm very interested in what the ICs that are closest to my lab's heart have been up to with this program. Looking at K99 awardees from 07 to 09 I find women PIs to constitute 3/3, 1/3 and 2/4 in one Institute and 1/7, 2/6 and 5/14 in the other Institute. One of these is doing better than the other and I will just note that was before the arrival of a Director who has been very vocal about sex discrimination in science and academia.
In terms of the conversion to R01 funding that is the subject of Datahound's post, the smaller Institution has decent outcomes* for K99 awardees from 07 (R01, R21, nothing), 08 (R01, R01, R01) and 09 (P20 component, U component, nothing, nothing).
In the other Institute, the single woman from 07 did not appear to convert to the R00 phase but Google suggests made Assistant Professor rank anyway. No additional NIH funding. The rest of the 07 class contains 4 with R01 and two with nothing. In 08, the women PIs are split (one R01, one nothing) similar to the men (2 R01, 2 with nothing). In 09 the women PIs have two with R01s, one R03 and two with nothing.
So from this qualitative look, nothing is out of step with Datahound's NIH-wide stats. There are 14/37 women PIs, this 38% is similar to the NIH-wide 39% Datahound quoted although there may be a difference between these two ICS (30% vs 60%) that could stand some inquiry. One of 37 K99 awardees failed to convert to R00 from the K99 (but seems to be faculty anyway). Grant conversion past the R00 is looking to be roughly half or a bit better.
I didn't do the men for the 2009 cohort in the larger Institute but otherwise the sex differences in terms of getting/not getting additional funding beyond the R00 seems pretty similar.
I do hope Datahound's stats open some eyes at the NIH, however. Sure, there are reasons to potentially excuse away a sex difference in the rates of landing additional research funding past the R00. But I am reminded of a graph Sally Rockey posted regarding the success rate on R01-equivalent awards. It showed that men and women PIs had nearly identical success rates on new (Type 1) proposals but slightly lower success on Renewal (Type 2) applications. This pastes over the rates of conversion to R00 and the acquisition of additional funding, if you squint a bit.
Are women overall less productive once they've landed some initial funding? Are they viewed negatively on the continuation of a project but not on the initiation of it? Are women too humble about what they have accomplished?
*I'm counting components of P or U mechanisms but not pilot awards.
Excellent comment from eeke:
My last NIH grant application was criticized for not including a post-doc at 100% effort. I had listed two techs, instead. Are reviewers under pressure to ding PI's for not having post-docs or some sort of trainee? WTF?
I think it is mostly because reviewers think that a postdoc will provide more intellectual horsepower than a tech and especially when you have two techs, you could have one of each.
I fully embrace this bias, I have to admit. I think a one-tech, one-PD full modular R01 is about the standardest of standard lineups. Anchors the team. Best impact for the money and all that.
A divergence from this expectation would require special circumstances to be explained (and of course there are many projects imaginable where two-tech, no-postdoc *would* be best, you just have to explain it)
What do you think, folks? What arrangement of personnel do you expect to see on a grant proposal in your field, for your agencies of closest interest? Are you a blank slate until you see what the applicant has proposed or do you have....expectations?
A query came into the blog email box about how to deal with submitting a new grant based on the prior A1 that did not get funded. As you know, NIH banned any additional revisions past the A1 stage back in 2009. Recently, they have decided to stop scrutinizing "new" applications for similarity with previously reviewed and not-funded applications. This is all well and good but how should we go about constructing the "new" grant, eh? A query from a Reader:
Do you use part of your background section to address reviewer comments? You're not allowed to have an introduction to the application, but as far as I can tell there is no prohibition on using other parts of the application as a response to reviewers.
I could see the study section as viewing this a) innovative, b) a sneaky attempt to get around the rules, c) both a and b.
I am uncertain about the phrasing of the Notice where it says "must not contain an introduction to respond to the critiques from the previous review". In context I certainly read this as prohibiting the extra page that you get for an amended application. What is less clear is whether this is prohibiting anything that amounts to such introduction if you place it in the Research Strategy. I suspect you could probably get away with direct quotes of reviewer criticisms.
This seems unwise to me, however. I think you should simply take the criticisms and revise your proposal accordingly as you would in the case of an amended version. These revisions will be sprinkled throughout the application as appropriate- maybe a change in the Significance argument, maybe a new Experiment in Aim 2, maybe a more elaborated discussion of Potential Pitfalls and Alternative Approaches.
Given the comments, perhaps you might need to state some things twice or set off key points in bold type. Just so the next set of reviewers don't miss your point.
But I see no profit in directly quoting the prior review and it just wastes space.
Where are all the outraged complaints about the quality of grant peer review and Errors Of Fact for grants that were scored within the payline?
I mean, if the problem is with bad review it should plague the top scoring applications as much as the rest of the distribution. Right?
This is a guest appearance of the bluebird of Twitter happiness known as My T Chondria. I am almost positive the bird does some sort of science at some sort of US institution of scientific research.
I’m your biased reviewer. I’ve sat on study sections for most of the years I’ve been a faculty member and I’m biased. I’m exactly who Sally Rockey and Richard Nakamura are targeting in their call for proposals to lessen bias and increase impartial reviewing of NIH applications.
Webster’s defines bias as “mental tendency or inclination” listing synonyms including “predisposition, preconception, predilection, partiality, and proclivity”. When I review a grant from an African American applicant, I have a preconception of who they are. I refine that judgment based on their training, publications and productivity.
I should share that I’m also biased in my review of applicants who have health issues, are women, are older than 30 and have children. I’ve had every one of these types of trainees in my lab and my experiences with them lead me to develop partiality and preconceptions that impact my opinions and judgments. Parts of my preconceptions arise from my experiences with these trainees in my as well those I interacted with while serving on my University’s admission committee. I was biased when I performed those duties as well.
Anyone who pretends to be utterly impartial is dangerous and hurtful to those we say we value as a scientific community. I am frankly stunned to see so many tone deaf and thoughtless comments claiming they are deeply offended at the at this ‘mindless drivel’.
Dr Marconi is just one of many scientists who claim, “I’ve never seen this, so it must not be true”. Scientist’s careers are based on things that cannot be seen, but we collect and interpret data and develop an understanding based on that which we cannot see. Data has been collected and the results are alarming and open for active debate.
Bias is far more insidious than racism. Racists reveal themselves and their ignorance and are often dismissed by ‘educated’ society for their extremist views. Bias is far subtler. Even if it results in an imperceptible change in scoring, we are in a climate where these things matter. Where razor fine decisions are being made on funding.
It's the people who are sure they have no bias that I fear. I know I have bias. We are simply incapable of being utterly impartial and anyone who says they are impartial is dangerously obtuse to these problems at best and a liar at worst.
It has recently come to my attention that not everyone views the no-cost extension (NCE) of a NIH grant the same way I do.
When the interval of support is over for many of the NIH grant mechanisms, the PI (actually the University or Institute, of course) can request a NCE. This means that while the NIH is not going to give out any additional money past the original award, the University may continue to spend any un-expended funds. My experience has been that NCE requests, particularly the first year, are approved by default.
I am pretty sure that you are supposed to follow the usual rules for rolling money from one funded interval to the next funded interval, i.e., it is supposed to be only 25% of the prior year or less. Also, if you have a great excuse for why you have slightly more than 25% left over I would think this would not be a huge problem.
Personally, I have requested a NCE essentially by default for every grant award that permits it and assuming that the competing renewal has not been approved in time to keep the funded intervals rolling.
There are my own local institutional reasons for doing so, mostly having to do with moderate red tape factors.
I thought that it was sort of required in order to submit a competing continuation (now called "renewal") application. I have one grants management assurance that this is not the case but I would still want to check up on how that works. After all, with the new A2 as A0 rules, can't we just submit a competing continuation application for a grant which has been unfunded for 5 years? 10 years? Wait....google...google....NIAID says:
Is there a window of time that a PI can submit an application as a renewal? Must the original grant still be active?
No. The grant need not be active and there is no time limit for a renewal application. However, reviewers will probably be concerned by major gaps.
If a significant amount of time has elapsed, indicate what you have done in the interim. Highlight any preliminary data you may have obtained, and show that your planned research is current with the latest science.
OK, maybe I learned something this year. I mean, I'm sure I have read that before about renewals but somehow it never really connected up.
Also, for some reason maybe I thought that reviewers would be more confident that you were actively working on the project past the end date if you could say it was in NCE.
My question for the peanut gallery today is, how seriously do you take the NCE when you see it mentioned in a Biosketch or elsewhere in the grant proposal? Is it just meaningless...as in "of course they applied for a NCE, duh" or below notice altogether? Does apply only when it is a competing continuation / renewal of the grant which is in NCE?
In a related vein, does it "count" as current research funding? Do you see a grant in NCE and mentally chalk it up under the PI or other Key Personnel's "funded grants"? In the Biosketch does it go under "completed" or "ongoing" Research Support?
Do you assume it might be pending renewal if it is not the prior interval of the grant you are reviewing now?
We've previously discussed the NIH F32 Fellowship designed to support postdoctoral trainees. Some of the structural limitations to a system designed on its fact to provide necessary support for necessary (additional) training overlap considerably with the problems of the F31 program designed to support graduate students.
Nevertheless, winning an individual NRSA training fellowship (graduate or postdoctoral) has all kinds of career benefits to the trainee and primary mentor so they remain an attractive option.
A question arose on the Twitts today about whether it was worth it for a postdoc in a new lab to submit an application.
— Dr Becca, PhD (@doc_becca) May 30, 2014
In my limited experience reviewing NRSA proposals in a fellowship-dedicated panel for the NIH, there is one issue that looms large in these situations.
Reviewer #1, #2 and #3: "There is no evidence in the application that sufficient research funds will be available to complete the work described during the proposed interval of funding."
NRSA fellowships, as you are aware, do not come with money to pay for the actual research. The fellowship applications require a good deal of discussion of the research the trainee plans to complete for the proposed interval of training. In most cases that research plan involves a fair amount of work that require a decent amount of research funding to complete.
The reviewers, nearly all of them in my experience, will be looking for signs of feasibility. That the PI is actually funded, funded to do something vaguely related* to the topic of the fellowship proposal and funded for the duration over which the fellowship will be active.
When the PI is not obviously funded through that interval, eyebrows are raised. Criticism is leveled.
So, what is a postdoc in a newer lab to do? What is the PI of a newish lab, without substantial funding to do?
One popular option is to find a co-mentor for the award. A co-mentor that is involved. Meaning the research plan needs to be written as a collaborative project between laboratories. Obviously, this co-mentor should have the grant support that the primary PI is lacking. It needs to be made clear that there will be some sort of research funds to draw upon to support the fellow doing some actual research.
The inclusion of "mentoring committees" and "letters of support from the Chair" are not sufficient. Those are needed, don't get me wrong, but they address other concerns** that people have about untried PIs supervising a postdoctoral fellow.
It is essential that you anticipate the above referenced Stock Critique and do your best*** to head it off.
*I have seen several highly regarded NRSA apps for which the research plan looks to me to be of R01-quality writing and design.
**We're in stock-critique land here. Stop raging about how you are more qualified than Professor AirMiles to actually mentor a postdoc.
***Obviously the application needs to present the primary mentor's funding in as positive a light as possible. Talk about startup funds, refer to local pilot grants, drop promising R01 scores if need be. You don't want to blow smoke, or draw too much attention to deficits, but a credible plan for acquiring funding goes a lot farther than ignoring the issue.
A guest post from @iGrrrl
Like winter, changes to the biosketch are coming
Dr. Rockey spoke about the changes to the biographical sketch at the NORDP meeting this week, and I think I can at least offer a bit more depth about the thinking behind this, both from her comments and from what I've seen over the last few years. Certainly my knee-jerk negative opinion about this change has evolved upon reflection and listening to her presentation. This may not be as bad as it sounds. Maybe.
In her talk, of which this question of biosketches was one very small part, her short-hand way of referring to the reasoning behind this was to "reduce the reliance on publishing in one-word-named journals" as a shorthand to judging the quality of an investigator's productivity. When the biosketch was changed in 2010, shortening the publication list to 15 seemed to me to be designed to reduce a senior investigator's advantage of sheer numbers of publications. The rise of metrics and h-factor means that the impact factor of the journal in which the work was published now substitutes, in many a reviewer's mind, as the quick heuristic for assessing the Investigator criterion.
The move to the shorter publication list was also borrowed from NSF's limit of 10 products for the biosketch. This sounds good on paper, but didn't account for the differences in culture. Researchers in NSF-type fields are just as conscious of h-index, but you don't find the same reliance on "glamour magazines" that cut across all NSF research. The result seems to be that many young investigators in biomedicine feel they have to wait to publish until they have a story worthy of C/N/S. I hear sometimes about young researchers failing to make tenure in part because they did not publish enough, not because they didn't have data, but because they were trying for the high-level journal, didn't simply get it out in field- or sub-field-specific journals.
And work that appears in those so-called lower-tier journals shouldn't be dismissed, but it often is effectively ignored when a reviewer's eyes are looking for the titles of the high-impact journals. If a young faculty member's list maxes to 15 and they are all solid papers in reasonable journals, that's usually fine. But sometimes they have fewer than 15, so the reviewer relies more on the impact factor of the journals in which the work appears, and that in turn leads to reliance on C/N/S (JAMA, NEJM, etc). But for the applicant, sometimes the work reflected in the papers is based on a study that simply takes a long time to run, so that one paper in that year might represent a great deal of effort and time with results highly relevant within the context of the subfield. Or a series of papers may have methods published in one journal, the study in two more, and none of them are top-tier, but the entire story is important. This new narrative gives the opportunity to give that context.
This appears to be the point of the change to the biosketch: the impact factor of the journal(s) in which the work appeared may not reflect the impact of the results. Some applicants were including a sentence after every paper on the biosketch to try to give the context and impact--the contribution to the field--but in my experience, reviewers did not like and did not read these sentences. Yet, when reviewers come from a diversity of backgrounds, they may not be able to appreciate the impact of a result on the sub-field. Many of these concerns have been vociferously expressed to Dr. Rockey through various social media, primarily comments here at Our Host's blot, but also on the RockTalking blog.
The idea behind this new approach to discussing an applicant's contributions has some reasonable foundations, but I don't expect it will work. In the short term, applicants will likely struggle to assemble a response to this new requirement. I can't imagine reviewers will enjoy reading the resulting narratives. It may be that a common rubric approach to writing these sections as a clear story will make them uniform enough for reviewers to quickly judge, but I fully expect they will still be looking for Cell, Nature, and Science.
In my view, once it is on The News Hour then it is really news.
Nature published a commentary by NIH Director Francis S. Collins and NIH Office of Research on Women's Health Director Janine A. Clayton which warns us that the NIH will start insisting on the inclusion of more sex-difference comparisons. These are to extend from cells to animal models across many areas of pre-clinical work.
The NIH is now developing policies that require applicants to report their plans for the balance of male and female cells and animals in preclinical studies in all future applications, unless sex-specific inclusion is unwarranted, based on rigorously defined exceptions. These policies will be rolled out in phases beginning in October 2014, with parallel changes in review activities and requirements.
I cannot wait to see what the "rigorously defined exceptions" will be for several types of research in which I have an interest. Every rat self-admin study must now include both males and females? For all treatment conditions or will it be acceptable to just tack the sex-comparison on at the end?
Furthermore, the NIH will monitor compliance of sex and gender inclusion in preclinical research funded by the agency through data-mining techniques that are currently being developed and implemented. Importantly, because the NIH cannot directly control the publication of sex and gender analyses performed in NIH-funded research, we will continue to partner with publishers to promote the publication of such research results.
oooooh. "partner with publishers" eh? Of course this is because Clayton and Collins realize that higher JIF journals are entirely uninterested in things as pedestrian as sex-comparisons, particularly when the outcome of the study is "no difference". Which, btw, is one of the reasons nobody* wants to waste their precious time and grant money doing something as low-return as sex-comparisons. So somehow the NIH is going to lean on publishers to be...friendlier....to such work. I do hope they realize that this is not going to work. The contingencies are not going to change because the NIH asks. Now, if they actually went all in and dismantled GlamourMagScience culture by the judicious use of grant award, grant auditing and rules about the ratio of publications to effort expended... then we might see some progress. That will never happen and thus there will be no change in the publication contingencies that fight against sex-comparison studies.
Dr. Clayton went on The News Hour where Judy Woodruff asked her (and Phyllis Greenberger of Society for Women's Health Research) some pretty obvious questions. Woodruff wanted to know if there were any clear examples in which women were put at risk or their health suffered because of a lack of such research. She also wanted to know what the implications for research might be- would it be more difficult or more expensive. Finally, Woodruff asked if scientists would resist.
From the transcript:
JUDY WOODRUFF: But how hard is that? Does that mean — is it extra work, is it more expensive? What’s involved in making sure there’s a gender balance?
Now Greenberger snuck in a "Both" off camera but then Clayton went on to be ridiculous and fail to answer the question. The answer is indeed "both" and it is a serious one if the NIH expects to get results. It will be more expensive, progress will be slower and it will be "harder" in the sense of teasing out the right experimental designs and variables so that an interpretable result can be reached. It isn't rocket science, exactly, but it is harder.
JUDY WOODRUFF: Phyllis Greenberger, were there — were there actually individuals who were harmed or where help wasn’t delivered because the research was done only on males?
Greenberger totally walked around this one and Woodruff, to her credit, fronted Clayton with the same question a bit later. Clayton referred to heart attack warning symptoms in women that might differ from men...of course this has nothing whatever to do with preclinical research. Gaaah! So frustrating. Greenberger chimed back in with talk of drugs being removed from the market for adverse effects in women....with no indication that these were adverse effects that would have been identified in female-specific PREclinical research. C'mon NIH! If you are going to take a run at this, please prepare your argument!
JUDY WOODRUFF: And is that the reason that it wasn’t done earlier, Dr. Clayton, that there was just pushback in the scientific community?
The answer is illustrative of the problem at the NIH....
DR. JANINE CLAYTON: It’s hard to say. There are probably a lot of factors that are involved.
And what’s really important now is right now we have been able to put the focus on getting this as a priority. As Phyllis mentioned, the Society and other advocacy groups and scientists and others have talked about this in the past. In fact, we are supporting scientists who are doing this research, but it wasn’t enough of a priority. In some way, it was like a blind spot. Scientists weren’t thinking about it.
Clayton is right. The NIH does indeed support investigators doing sex-differences studies.Those scientists do not have a problem of "priority" from the perspective of their own intrinsic motivation.
With respect to whether scientists resist, I enjoin you to go over to PubMed and type in Sex Differences and see what fill-in choices are offered to you. Click on several of these searches and see what you find. You will find funded projects in many of your favorite domains of interest. If you bother to click on the papers and look at the grant attributions, you may even find that many of these investigations were completed under NIH funding!
So when Clayton (and in the Commentary she is joined by Director Collins) claims it isn't a "priority", it seems misplaced to put this on the shoulders of extramural scientists.
If the NIH wants more sex-differences studies then they need to deploy their tastiest carrot to greater effect. Put out some Funding Opportunity Announcements and see what happens! Fund a few Supplements to the people who are already doing sex-comparisons! Pick up a few grants that missed the payline...again, from the people who are already proposing sex-comparisons!
And if you want to lure in new converts that you didn't get with an RFA or a Program Announcement? This is simple. Just put out a policy that any grant application with a credible stab at a sex-comparison component gets an extra 5 percentile points credit towards the payline for funding.
Just you wait and see how many sudden converts you make!
*of the GlamourMag class investigator