Archive for the 'NIH funding' category

Your Grant in Review Reminder: Research Study Sections First

Aug 22 2014 Published by under Grant Review, Grantsmanship, NIH Careerism, NIH funding

One key to determining the right study section to request is to look on RePORTER for funded grants reviewed in your study sections of interest.

Sometimes this is much more informative than the boilerplate description of the study section listed at CSR.

8 responses so far

You came so close, Nature Editorial staff.

Aug 20 2014 Published by under Fixing the NIH, NIH Careerism, NIH funding

A news bit in Nature overviews Richard Nakamura's plans to investigate the disparity in grant funding that was identified in the Ginther report in 2011. See here, here, here for blog comment on Gither et al. Nakamura is, of course, the current director (pdf) of the Center for Scientific Review, the entity at NIH that conducts the peer review of most grant applications that are submitted.

It is promising-ish. Nakamura's plans are summarized.

One basic issue that the NIH will address is whether grant reviewers are thinking about an applicant’s race at all, even unconsciously. A team will strip names, racial identification and other identifying information from some proposals before reviewers see them, and look at what happens to grant scores.

Hope they check on the degree to which the blinding works, of course. As you know, Dear Reader, I am always concerned that blinding of academic works cannot always be assumed to have functionally worked to prevent the reviewer from identifying the author or lab group.

The NIH will also study reviewers’ work in finer detail, by analysing successful applications for R01 grants, the NIH’s largest funding programme for individual investigators. The goal is to see whether researchers can spot trends in the language used by reviewers to describe proposals put forward by applicants of different races. There is precedent for detectable differences: in a paper to be published in Academic Medicine, a team led by Molly Carnes, a physician at the University of Wisconsin-Madison, used automated text analysis to show that reviewers’ critiques of R01 grant applications by women tended to include more words denoting praise, as though the writer is surprised at the quality of the work.

Very intriguing contribution to the analysis. Nice.

The NIH will also analyse text in samples of reviewers’ unedited critiques. The Center for Scientific Review typically edits the wording and grammar of these reviews before grant proposals are returned to applicants, but even the subtlest details of such raw comments might hold clues about bias. Nakamura says that reviewers will not be told whether their comments will be analysed, because that in itself would bias the sample. “We want them to be sloppy,” he says.

Hmmm. I guess this is just human factors checking on the automated analysis. Together they are stronger.

The NIH’s Study Sections, in which review groups discuss the top 50% of grant applications, might also harbour bias: the 2011 Science paper found that submissions authored by African Americans are less likely to be discussed in the meetings. But when they are, a negative comment arising from even one person’s unconscious bias could have a major impact in such a group setting, says John Dovidio, a psychologist at Yale University in New Haven, Connecticut, and a member of the NIH’s Diversity Working Group. “That one person can poison the environment,” he says.

This is not presented in a context that suggests the NIH plans to investigate this directly. Not sure how this could be done without putting a severe finger on the balance. I mean sure, most meetings have call-in lines open to Program staff and Nakamura could just record and transcribe meetings but.....reviewers won't like it and if you warn them you scare off the fishies. So to speak.

Nakamura expects that the NIH’s effort to identify and root out prejudice, which he says could cost up to $5 million over three years, might prove controversial. “People resent the implication they might be biased,” he says — an idea borne out by some responses to his 29 May blogpost on the initiative. One commenter wrote, “It is absolutely insulting to be accused of review bigotry. Please tell me why I should continue to give up my time to perform peer review?”

But Nakamura believes that the NIH — and reviewers — need to keep open minds.

He, and media covering this, need to focus on the opportunity to communicate that institutional racism does not hinge on whether individual actors are overtly biased. The piece leads with the comment that Nakamura got his butt over to the Implicit Association website and identified his own biases to himself. This is the sort of introspection that needs to happen. Heck, I'd love to see a trial where study section reviewers were told to go over there and complete a few tests prior to receiving their grant assignments*.

The Nature editorial is, for the most part, on the side of goodness and light on this.

The idea that scientists who volunteer time and energy to review NIH grants could be biased against qualified minority researchers is a tough pill to swallow. The NIH is to be commended for not sweeping this possibility under the rug: it has turned to the scientific method to investigate the suggestion.

good, good.....

It is a topic that the NIH will need to broach delicately. Few academics consciously hold any such inclinations, and fewer still would deliberately allow them to affect their grant evaluations. Some are likely to bristle at what might be seen as an accusation of racism, and the NIH plans to conduct at least some of its studies of grant reviews without the reviewers’ knowledge or consent.

But better for the NIH to offend a few people than to make snap judgements and institute blunt policies to address the problem. Fixes such as increasing scholarships and training for minority groups would no doubt be a good thing, but they could be an unhelpful use of money if they do not address the root cause of the disparity.

yes, yes, excellent.....

And policies such as grant-allocation quotas could come at the expense of other researchers.

BZZZTTTTTTT!!!!

No. Bad Nature.

Right back to victim blaming. Right back to ignoring what it means to have a BIAS identified. Right back to ignoring what the nature of privilege means.

Those "other researchers" at present enjoy a disparate benefit at the expense of AfricanAmerican PIs. That's what Ginther means. Period. The onus shifted, upon identification of the disparity, to proving that non-AfricanAmerican PIs actually deserve their awards.

Ginther, btw, went a long ways toward rejecting some of the more obvious reasons why the disparity was not in fact an unfair bias. Read it, including the supplementary materials before you start commenting with stupid. Also, review this.

But there is also this. The low numbers of AfricanAmerican scientists submitting applications to the NIH for funding means that any possible hit to the success rate of non-AfricanAmerican PIs would be well nigh undetectable. A miniscule effect size relative to all other sources of variance in the funding process.

Another way to look at this issue is to take Berg's triage numbers from above. To move to 40% triage rate for the African-AmericanPI applications, we need to shift 20% (230 applications) into the discussed pile. This represents a whopping 0.4% of the White PI apps being shifted onto the triage pile to keep the numbers discussed the same.

These are entirely trivial numbers in terms of the "hit" to the chances of White PIs and yet you could easily equalize the success rate or award probability for African-American PIs.

It is even more astounding that this could be done by picking up African-American PI applications that scored better than the White PI applications that would go unfunded to make up the difference.

And of course "grant-allocation quotas" are precisely what the special paylines and other assists for ESI investigators consist of. Affirmative Action for the young and untried.

Did we get this sort of handwringing, call for long-duration "study" of the "true causes" of the disparity?

Hell no.

The NIH just started picking up ESI grants to balance the odds of funding, even when study sections responded to news of this affirmative action by further punishing ESI scores!

So yeah, my call is for the NIH to balance the funding rates first, and then do all their fancy studies to root out the "real cause" later.

Also for editorial teams like the one at Nature to stop repeating this whinging about those who already enjoy disparate privilege who might lose (some of) it.
__
*yeah, it might backfire. that would itself be interesting.

14 responses so far

Replication costs money

I ran across a curious finding in a very Glamourous publication. Being that it was in a CNS journal, the behavior sucked. The data failed to back up the central claim about that behavior*. Which was kind of central to the actual scientific advance of the entire work.

So I contemplated an initial, very limited check on the behavior. A replication of the converging sort.

It's going to cost me about $15K to do it.

If it turns out negative, then where am I? Where am I going to publish a one figure tut-tut negative that flies in the face of a result published in CNS?

If it turns out positive, this is almost worse. It's a "yeah we already knew that from this CNS paper, dumbass" rejection waiting to happen.

Either way, if I expect to be able to publish in even a dump journal I'm gong to need to throw some more money at the topic. I'd say at least $50K.

At least.

Spent from grants that are not really related to this topic in any direct way.

If the NIH is serious about the alleged replication problem then it needs to be serious about the costs and risks involved.
__
*a typical problem with CNS pubs that involve behavioral studies.

34 responses so far

Women in the R00 phase don't apply for R01s as frequently as men

Sally Rockey:

A specific issue that recently has recently created interesting conversations in the blogosphere is whether female K99/R00 awardees were less likely to receive a subsequent R01 award compared to male K99/R00 awardees. We at NIH have also found this particular outcome among K99/R00 PIs and have noted that those differences again stem from differential rates of application. Of the 2007 cohort of K99 PIs, 86 percent of the men had applied for R01s by 2013, but only 69 percent of the women had applied.

She's referring here to a post over at DataHound ("K99-R00 Evaluation: A Striking Gender Disparity") which observed:

Of the 201 men with R00 awards, 114 (57%) have gone on to receive at least 1 R01 award to date. In contrast, of the 127 women with R00 awards, only 53 (42%) have received an R01 award. This difference is jarring and is statistically significant (P value=0.009).
...
To investigate this further, I looked at the two cohorts separately. For the FY2007 cohort, 70 of the 108 men (65%) with R00 awards have received R01 grants whereas only 31 of the 62 women (50%) have (P value = 0.07). For the FY2008 cohort, 44 of the 93 men (47%) with R00 awards have received R01s whereas only 22 of the 65 women (34%) have (P value = 0.10). The lack of statistical significance is due to the smaller sample sizes for the cohorts separately rather than any difference in the trends for the separate cohorts, which are quite similar.

And Rockey isn't even giving us the data on the vigor with which a R00 holder is seeking R01 funding. That may or may not make the explanation even stronger.

Seems to me that any mid or senior level investigators who have new R00-holding female assistant professors in their department might want to make a special effort to encourage them to submit R01 apps early and often.

13 responses so far

Thought of the Day

Jul 24 2014 Published by under Careerism, NIH, NIH Careerism, NIH funding

What fraction of the stuff proposed in funded grants actually gets done after feasibility and field movement come to play?

22 responses so far

Sex differences in K99/R00 awardees from my favorite ICs

Jul 21 2014 Published by under Grantsmanship, NIH, NIH Careerism, NIH funding, Peer Review

Datahound has some very interesting analyses up regarding NIH-wide sex differences in the success of the K99/R00 program.

Of the 218 men with K99 awards, 201 (or 92%) went on to activate the R00 portion. Of the 142 women, 127 (or 89%) went on to these R00 phase. These differences in these percentages are not statistically different.

Of the 201 men with R00 awards, 114 (57%) have gone on to receive at least 1 R01 award to date. In contrast, of the 127 women with R00 awards, only 53 (42%) have received an R01 award. This difference is jarring and is statistically significant (P value=0.009).

Yowza.

So per my usual, I'm very interested in what the ICs that are closest to my lab's heart have been up to with this program. Looking at K99 awardees from 07 to 09 I find women PIs to constitute 3/3, 1/3 and 2/4 in one Institute and 1/7, 2/6 and 5/14 in the other Institute. One of these is doing better than the other and I will just note that was before the arrival of a Director who has been very vocal about sex discrimination in science and academia.

In terms of the conversion to R01 funding that is the subject of Datahound's post, the smaller Institution has decent outcomes* for K99 awardees from 07 (R01, R21, nothing), 08 (R01, R01, R01) and 09 (P20 component, U component, nothing, nothing).

In the other Institute, the single woman from 07 did not appear to convert to the R00 phase but Google suggests made Assistant Professor rank anyway. No additional NIH funding. The rest of the 07 class contains 4 with R01 and two with nothing. In 08, the women PIs are split (one R01, one nothing) similar to the men (2 R01, 2 with nothing). In 09 the women PIs have two with R01s, one R03 and two with nothing.

So from this qualitative look, nothing is out of step with Datahound's NIH-wide stats. There are 14/37 women PIs, this 38% is similar to the NIH-wide 39% Datahound quoted although there may be a difference between these two ICS (30% vs 60%) that could stand some inquiry. One of 37 K99 awardees failed to convert to R00 from the K99 (but seems to be faculty anyway). Grant conversion past the R00 is looking to be roughly half or a bit better.

I didn't do the men for the 2009 cohort in the larger Institute but otherwise the sex differences in terms of getting/not getting additional funding beyond the R00 seems pretty similar.

I do hope Datahound's stats open some eyes at the NIH, however. Sure, there are reasons to potentially excuse away a sex difference in the rates of landing additional research funding past the R00. But I am reminded of a graph Sally Rockey posted regarding the success rate on R01-equivalent awards. It showed that men and women PIs had nearly identical success rates on new (Type 1) proposals but slightly lower success on Renewal (Type 2) applications. This pastes over the rates of conversion to R00 and the acquisition of additional funding, if you squint a bit.

Are women overall less productive once they've landed some initial funding? Are they viewed negatively on the continuation of a project but not on the initiation of it? Are women too humble about what they have accomplished?
__
*I'm counting components of P or U mechanisms but not pilot awards.

15 responses so far

Strategies for your #A2asA0 Resubmissions

Jun 30 2014 Published by under Ask DrugMonkey, NIH, NIH Careerism, NIH funding

A query came into the blog email box about how to deal with submitting a new grant based on the prior A1 that did not get funded. As you know, NIH banned any additional revisions past the A1 stage back in 2009. Recently, they have decided to stop scrutinizing "new" applications for similarity with previously reviewed and not-funded applications. This is all well and good but how should we go about constructing the "new" grant, eh? A query from a Reader:

Do you use part of your background section to address reviewer comments? You're not allowed to have an introduction to the application, but as far as I can tell there is no prohibition on using other parts of the application as a response to reviewers.

I could see the study section as viewing this a) innovative, b) a sneaky attempt to get around the rules, c) both a and b.

I am uncertain about the phrasing of the Notice where it says "must not contain an introduction to respond to the critiques from the previous review". In context I certainly read this as prohibiting the extra page that you get for an amended application. What is less clear is whether this is prohibiting anything that amounts to such introduction if you place it in the Research Strategy. I suspect you could probably get away with direct quotes of reviewer criticisms.

This seems unwise to me, however. I think you should simply take the criticisms and revise your proposal accordingly as you would in the case of an amended version. These revisions will be sprinkled throughout the application as appropriate- maybe a change in the Significance argument, maybe a new Experiment in Aim 2, maybe a more elaborated discussion of Potential Pitfalls and Alternative Approaches.

Given the comments, perhaps you might need to state some things twice or set off key points in bold type. Just so the next set of reviewers don't miss your point.

But I see no profit in directly quoting the prior review and it just wastes space.

10 responses so far

Quality of grant review

Jun 13 2014 Published by under Grant Review, NIH funding, Peer Review

Where are all the outraged complaints about the quality of grant peer review and Errors Of Fact for grants that were scored within the payline?

I mean, if the problem is with bad review it should plague the top scoring applications as much as the rest of the distribution. Right?

47 responses so far

Impressions of the "no-cost extension" of your NIH grant.

Jun 02 2014 Published by under NIH, NIH Careerism, NIH funding

It has recently come to my attention that not everyone views the no-cost extension (NCE) of a NIH grant the same way I do.

When the interval of support is over for many of the NIH grant mechanisms, the PI (actually the University or Institute, of course) can request a NCE. This means that while the NIH is not going to give out any additional money past the original award, the University may continue to spend any un-expended funds. My experience has been that NCE requests, particularly the first year, are approved by default.

I am pretty sure that you are supposed to follow the usual rules for rolling money from one funded interval to the next funded interval, i.e., it is supposed to be only 25% of the prior year or less. Also, if you have a great excuse for why you have slightly more than 25% left over I would think this would not be a huge problem.

Personally, I have requested a NCE essentially by default for every grant award that permits it and assuming that the competing renewal has not been approved in time to keep the funded intervals rolling.

There are my own local institutional reasons for doing so, mostly having to do with moderate red tape factors.

I thought that it was sort of required in order to submit a competing continuation (now called "renewal") application. I have one grants management assurance that this is not the case but I would still want to check up on how that works. After all, with the new A2 as A0 rules, can't we just submit a competing continuation application for a grant which has been unfunded for 5 years? 10 years? Wait....google...google....NIAID says:

Is there a window of time that a PI can submit an application as a renewal? Must the original grant still be active?

No. The grant need not be active and there is no time limit for a renewal application. However, reviewers will probably be concerned by major gaps.

If a significant amount of time has elapsed, indicate what you have done in the interim. Highlight any preliminary data you may have obtained, and show that your planned research is current with the latest science.

OK, maybe I learned something this year. I mean, I'm sure I have read that before about renewals but somehow it never really connected up.

Also, for some reason maybe I thought that reviewers would be more confident that you were actively working on the project past the end date if you could say it was in NCE.

My question for the peanut gallery today is, how seriously do you take the NCE when you see it mentioned in a Biosketch or elsewhere in the grant proposal? Is it just meaningless...as in "of course they applied for a NCE, duh" or below notice altogether? Does apply only when it is a competing continuation / renewal of the grant which is in NCE?

In a related vein, does it "count" as current research funding? Do you see a grant in NCE and mentally chalk it up under the PI or other Key Personnel's "funded grants"? In the Biosketch does it go under "completed" or "ongoing" Research Support?

Do you assume it might be pending renewal if it is not the prior interval of the grant you are reviewing now?

19 responses so far

NIH takes their Sex-Differences show on the road

May 21 2014 Published by under Diversity in Science, NIH, NIH Careerism, NIH funding

In my view, once it is on The News Hour then it is really news.

Nature published a commentary by NIH Director Francis S. Collins and NIH Office of Research on Women's Health Director Janine A. Clayton which warns us that the NIH will start insisting on the inclusion of more sex-difference comparisons. These are to extend from cells to animal models across many areas of pre-clinical work.

The NIH is now developing policies that require applicants to report their plans for the balance of male and female cells and animals in preclinical studies in all future applications, unless sex-specific inclusion is unwarranted, based on rigorously defined exceptions. These policies will be rolled out in phases beginning in October 2014, with parallel changes in review activities and requirements.

I cannot wait to see what the "rigorously defined exceptions" will be for several types of research in which I have an interest. Every rat self-admin study must now include both males and females? For all treatment conditions or will it be acceptable to just tack the sex-comparison on at the end?

Furthermore, the NIH will monitor compliance of sex and gender inclusion in preclinical research funded by the agency through data-mining techniques that are currently being developed and implemented. Importantly, because the NIH cannot directly control the publication of sex and gender analyses performed in NIH-funded research, we will continue to partner with publishers to promote the publication of such research results.

oooooh. "partner with publishers" eh? Of course this is because Clayton and Collins realize that higher JIF journals are entirely uninterested in things as pedestrian as sex-comparisons, particularly when the outcome of the study is "no difference". Which, btw, is one of the reasons nobody* wants to waste their precious time and grant money doing something as low-return as sex-comparisons. So somehow the NIH is going to lean on publishers to be...friendlier....to such work. I do hope they realize that this is not going to work. The contingencies are not going to change because the NIH asks. Now, if they actually went all in and dismantled GlamourMagScience culture by the judicious use of grant award, grant auditing and rules about the ratio of publications to effort expended... then we might see some progress. That will never happen and thus there will be no change in the publication contingencies that fight against sex-comparison studies.

Dr. Clayton went on The News Hour where Judy Woodruff asked her (and Phyllis Greenberger of Society for Women's Health Research) some pretty obvious questions. Woodruff wanted to know if there were any clear examples in which women were put at risk or their health suffered because of a lack of such research. She also wanted to know what the implications for research might be- would it be more difficult or more expensive. Finally, Woodruff asked if scientists would resist.

From the transcript:

JUDY WOODRUFF: But how hard is that? Does that mean — is it extra work, is it more expensive? What’s involved in making sure there’s a gender balance?

Now Greenberger snuck in a "Both" off camera but then Clayton went on to be ridiculous and fail to answer the question. The answer is indeed "both" and it is a serious one if the NIH expects to get results. It will be more expensive, progress will be slower and it will be "harder" in the sense of teasing out the right experimental designs and variables so that an interpretable result can be reached. It isn't rocket science, exactly, but it is harder.

JUDY WOODRUFF: Phyllis Greenberger, were there — were there actually individuals who were harmed or where help wasn’t delivered because the research was done only on males?

Greenberger totally walked around this one and Woodruff, to her credit, fronted Clayton with the same question a bit later. Clayton referred to heart attack warning symptoms in women that might differ from men...of course this has nothing whatever to do with preclinical research. Gaaah! So frustrating. Greenberger chimed back in with talk of drugs being removed from the market for adverse effects in women....with no indication that these were adverse effects that would have been identified in female-specific PREclinical research. C'mon NIH! If you are going to take a run at this, please prepare your argument!


JUDY WOODRUFF: And is that the reason that it wasn’t done earlier, Dr. Clayton, that there was just pushback in the scientific community?

The answer is illustrative of the problem at the NIH....

DR. JANINE CLAYTON: It’s hard to say. There are probably a lot of factors that are involved.

And what’s really important now is right now we have been able to put the focus on getting this as a priority. As Phyllis mentioned, the Society and other advocacy groups and scientists and others have talked about this in the past. In fact, we are supporting scientists who are doing this research, but it wasn’t enough of a priority. In some way, it was like a blind spot. Scientists weren’t thinking about it.

Yes, there are a lot of factors. They aren't all that complicated either, since they boil down to scientists who want to conduct sex-differences comparisons being able to win funding to do the work.

Clayton is right. The NIH does indeed support investigators doing sex-differences studies.Those scientists do not have a problem of "priority" from the perspective of their own intrinsic motivation.

PubMedSexDiffsWith respect to whether scientists resist, I enjoin you to go over to PubMed and type in Sex Differences and see what fill-in choices are offered to you. Click on several of these searches and see what you find. You will find funded projects in many of your favorite domains of interest. If you bother to click on the papers and look at the grant attributions, you may even find that many of these investigations were completed under NIH funding!

So when Clayton (and in the Commentary she is joined by Director Collins) claims it isn't a "priority", it seems misplaced to put this on the shoulders of extramural scientists.

If the NIH wants more sex-differences studies then they need to deploy their tastiest carrot to greater effect. Put out some Funding Opportunity Announcements and see what happens! Fund a few Supplements to the people who are already doing sex-comparisons! Pick up a few grants that missed the payline...again, from the people who are already proposing sex-comparisons!

And if you want to lure in new converts that you didn't get with an RFA or a Program Announcement? This is simple. Just put out a policy that any grant application with a credible stab at a sex-comparison component gets an extra 5 percentile points credit towards the payline for funding.

Just you wait and see how many sudden converts you make!

___
*of the GlamourMag class investigator

20 responses so far

Older posts »