Nobody really "knows what they are talking about" when it comes to #NIHGrants.
We all muddle along as best we can and persuade ourselves we have a partial clue.
That part is just to keep from going insane, though.
Archive for the 'NIH funding' category
NCI publishes their grant funding outcome for FY2011
This is huge. Previously the only IC that, to my knowledge, made their funding data available was the NIGMS. We grant geeks were big fans, even those of us who don't seek funding from that particular Institute of the NIH.
Well, apparently NCI has joined the party....I do hope this is a sign of things to come at other ICs. Do note that this comes in the wake of some announced policy changes from NCI head Varmus which caused some consternation. Check the comments over at writedit's pad. I concluded that this was just business as usual (i.e., as already practiced by numerous other ICs).
meh. he's actually talking normal stuff here. I read him as saying the payline is 7%ile (he says priority score but I suspect he means percentile) and then, as is totally normal business as usual for many ICs, he's talking about the gray zone wherein they violate the strict order of review for various Programmatic reasons.
Nothing to see here, save 7%ile is the lowest payline I've heard mentioned as such...
Given such practices, however, we are all intensely curious about the mysterious grey zone behavior. I have asserted in the past that I think the NIGMS data very likely stand as proxy for most, if not all, other ICs in the broad strokes. (The reason is that they dovetail nicely with the tiny bits of info that sneak out around the corners for the other ICs, if one is inclined to follow the breadcrumbs.) Importantly, the grey zone pickups are not randomly distributed. They are more likely the closer the score is to the payline. Well, now I have another data point..
First up, the Experienced Investigator graph:
yep. looks very familiar.
ok, how about the New Investigators?
hmm, notice that percentile skew? Now let's see about the ESI folks:
Yep.
Okay, so what? Well I think this should continue to motivate people to keep the heat on whatever NIH representative happens to be listening. POs or those poor, poor higher-ups that have to get up in front of a room of agitated PIs and put a happy face on things.
My point is that this skew shows that study sections are not responding to the clear intents and desire of the NIH. I.e., to treat newb investigators more fairly (or "generously" one might argue). And just like with any other initiative of the NIH with respect to review, I assume that they are serious about it. They've shown this, but changing ESI paylines, making greyzone pickups more frequently, etc. So why not fix the problem at the point of review?
1) First off, you would think that both Program and the reviewers would see that their refusal to treat ESI apps in the mix with the rest decreases their input even further. We all have the experience that the tightest discussion and the most agonized decision making as an assigned reviewer comes at the perceived payline. For the obviously top applications, all we're looking to do is to make an obvious argument. For the ones that are going to get triaged, or nearly triaged, well the tendency is to just hit the high points, slap on a few StockCritiques and assign a score. Even if the app is discussed way down in ~30-40%ile land, there isn't going to be so much argument about the exact score range. The other reviewers aren't going to be so engaged trying to decide which end of the post-discussion score range they should go with. Not like they will with applications that appear to be right around the perceived payline.
2) Next, this is a symptom of a larger problem. I.e., for the NIH trying to get the review panels on board with their broader goals. Take "Innovation". Despite a lot of hoopla in launching a new review approach, the data showed (thanks again, NIGMS) that review outcome was driven mostly by the same old, same old. I.e., Significance and Approach. The same problem applies if the ICs choose to fix this by scrutinizing the grey zone critiques for the ones that seem most "Innovative"....panels haven't discriminated the pool very well on that factor. More variance, more influence of the PO.
3) I think a lot of reviewers have no concept of these broader statistical trends. They are unaware of the data, blind to study section cultural influences and generally just haven't thought things through very well. It may be that some of these people really believe in a different type of outcome for their study section and their subfield. But they have no notion of where the problem lies, nor that it is fixable.
It is most assuredly a fixable problem. I have two themes that I've pursued on the blog. First, education of study section members with respect to what they are doing. The funding data, such as the NCI and NIGMS charts posted and linked above, is the start of this. I'd like to see these outcomes be made available to study section reviewers, right down to the level of their own review panel. Second, the solution of competing biases. Anytime there is human judgement, there is bias. Anytime. The only solution that offers high confidence of having an effect is the competition of biases. This is why the panels are explicitly representative of geography, sex, ethnicity and institution type/size. What they are not representative on is the Newb/Experienced PI axis. (Also, one might argue that the Innovative!!!1111!!/Conservative PI axis has some skew, but that's a chat for another day.)
@boehninglab was skeptical that there was any point in talking about these issues. I, naturally, am of the opinion that the surest way to prevent things you want to see happen is to remain silent. Sure, there are never any guarantees that you position will change anything at the NIH, but if you don't say something then there is a guarantee you won't be heard.
So comment on the NIH sites, in this case the NCI one. Let them know how you see their behavior and why it is good or bad for the science in your subfield.
[h/t: @salsb]
__
ps. check the NCI page for the R21 data, also interesting.
update: pps, PhysioProf noted that the score distribution for Experienced/New Investigators is much more similar for R21s than for R01s. Interesting to consider why that might be so. I would point to the "starter grant" bias....
Sample R21 grants with Summary Statements from NIAID
As I previously noted (somewhat critically) that the NIAID had posted sample R01 grants and the corresponding summary statements. Well, they've added some R21 applications to the page.
Again, I wonder how useful this really is for most applicants. First thing you notice is that it takes a perfect score to get funded. Three of the four received 10s and the fourth limped home with an 11. Remember, the study section score range starts at 1, which is then multiplied by 10 after the voting of the entire panel is averaged.
Then there's this (emphasis added):
From the Dow summary statement's resume of discussion: "Strengths of the application include the accomplished investigator and research team, strong preliminary data, the direct doable and logical set of experiments, and the likelihood of paradigm shifting insights into meliodosis"
From the resume on the Starnbach app: "Strengths of the application include the innovative use of the novel GPS strategy, compelling preliminary data, an investigator with a strong bacterial pathogenesis research track record, an excellent and appropriate set of collaborators, and a high degree of confidence that import results will emerge from these studies."
Weis, individual critique #2: "Strong and compelling preliminary data is presented that indicate a high likelihood of success"
Well, at least NIAID is telling it like it is with these examples.....
Keeping in good with the PO? or respect your institution's best interests?
There's a lot of paranoid insanity over at writedit's blog these days, which is a mixed bag. On the one hand, I'm glad so many folks are finding some place online to ask their questions about the NIH process. It can be maddeningly opaque to the newcomers at times and having been there myself, I sympathize. On the other hand....some of the questions make me really question whether these people can possibly be capable of running a lab with such cluelessness.
Then there are the questions that are initially maddening but on reflection are worth discussing. This may or may not be one of them. One JaneyC asked:
My NIH R15 grant proposal got a score of 20 recently. The payline was 24 in the September 2011 and January 2012. The PO was very positive that I will be awarded the R15 in June. The problem for me is that I have accepted a job offer from a top research one university which is not eligible for R15. My last day of employment is late August at the present institution. I met the grant officer in the current institution, and she suggested me not to inform NIH at this moment. Rather she will contact NIH after the council meeting for funding in June, and try to switch PI for this grant. Should I contact the PO in June and explain my situation? I just want to make good connection with the PO for future funding opportunities. Thanks a lot.
Now let us all remember the mantra....a grant is awarded to the institution, not to the PI. It is their award, not "your" award. And it is most certainly in the interests of the University not to queer the deal in any way. That means, "let's not rock the boat" when the NIH appears ready to fund a given award.
In this case, it would appear that the grant should issue as a July 1 start. So technically the University could accept the award with "JaneyC" as the PI, get it started and one month later say "Whoops, the PI is leaving. We propose elevating the co-PI or co-I (or some other local investigator) to the PI slot on this grant". The thinking here is that a done-deal award is much less likely to be pulled back.
I would make that assumption myself. Always better to let the grant fund and then ask to swap out personnel later, if you ask me. The R15 is a bit weird but if it were an R01, wel I've seen some serious kickback when an institution proposes swapping PIs prior to award. Enough so that if the University can credibly try to let on that they have no idea the PI was leaving when they accepted the award, it's worth doing.
but the original PI has concerns:
I just want to make good connection with the PO for future funding opportunities.
Indeed. Technicalities are all well and good but if you've read my ramblings over the years, you know good relationships with Program Officers can mean a lot. Like, the difference between "sorry" and an out-of-order pickup of your grant application that juuuuuuuust missed the payline. That's important.
The PO could be annoyed at the University under the above strategy for two reasons. One, just a generic annoyance at the lack of courtesy. It seems like better manners to inform the PO about anything major that will affect the award at the earliest possible convenience. A sort of "just in case you wanted to weigh in" kind of thing. Especially given that the R15/AREA mechanism is so focused on the University itself...the risks seem low that the PO would be bothered. The second reason could be if the PO really does have an issue related to the PI of the application. They are (I surmise) well within their rights to complain about University shenanigans expressing the full force of their "rights" with respect to the award. The PO might just block the funding of the award.
Now given that, what are the odds the PO would be annoyed with the PI in a case like this. If s/he doesn't bother to pick up the phone and inform the PO of the imminent departure. Now again, we have to reflect that this is a grant which cannot be transferred to just any other institution, unlike the R mechanisms. So it wouldn't be quite so weird, right? The application's PI might be justified in simply walking away with no further comment. Especially after being told directly by the University contracts and grants people not to call the PO. This gives the original PI a nice little excuse after the fact. After s/he's arrived at NewU and the R15 has been awarded to OldU and all that.
I'm thinking that you, the PI, keep your own counsel in a case like this.
This doesn't sound "mean" to me
Some commenter at Rock Talk complained about a recent grant review:
I just received the most terrible of reviews, where the reviewer was not only biased but highly inflammatory, prejudicial and aggressive. I must say I was totally taken aback. When you say things like “…terribly convoluted approach”,…”PI has clearly no clue…” how something works, trashes my published work by saying these pubs “are a gross exaggeration”….the list goes on and on. Even as a relatively senior investigator, I was very shocked by the mean-spirited nature of the comments. I cannot imagine how it would destroy a new investigator.
I am having trouble seeing it. I mean sure "no clue" is directed at the applicant rather than the application, but it's pretty tame stuff. If a reviewer thinks your papers exaggerate? Presumably in wild speculative interpretation that runs beyond your data? Seems okay and even obligatory to express this. The "terribly convoluted approach" comment is a pretty inoffensive way to get to the heart of this common failing of grants as well...I'm not seeing how you could put it more "nicely".
Francis Collins assures Congress that NCATS won't draw from basic research
via the Nature News Blog we learn that NIH Director Collins has been called upon by Congress to explain NCATS. This is the acronym for his pet project a Center dedicated to "Translational Science" that required axing the venerable National Center for Research Resources.
Collins noted that NIH’s support for basic research has held steady at about 54% of the agency’s budget for decades. “I do not expect that percentage to change,” he said. He added repeatedly that all but 2% of the $575 million funding the translational medicine center this year comes from preexisting NIH programs, and is not “new” money.
Some legislators, understanding quite clearly that money is fungible were keen to press Collins:
Representative Cynthia Lummis, Republican of Wyoming, interrupted Collins to insist that he explain how the $64 million increase proposed for NCATS in 2013 can’t be seen as being largely funded by a cut to the Institutional Development Award (IDEA) program. The NIH in 2013 has proposed cutting $50.5 million from the program, which funds biomedical investigators, trainees and infrastructure in 23 largely rural states that have historically experienced low application success rates for NIH grants.
“I would not want you to see a direction connection between…the IDEA program and NCATS. Those are not the same dollars that just got moved from one box to another,” Collins responded.
“Dollars are dollars,” Lummis replied.
Exactly. And similarly there are plenty of imaginable grants that would be "translational science" that Collins will get to score in the "basic" category as well. Another CongressCritter argued with Collins that a prior boost to the IDEA program was intended to be permanent, something Collins disputed. Yes, keeping track of this slippery customer down the line will be pretty hard for our intrepid Congressional heroes.
There was another bit of testimony that drew my eye because it speaks to the potential upside of NCATS rather than whinging (ahem) about the costs to other programs. NCATS is supposed to somehow do better than the pharma industry. Ok, fine, but it sort of presumes the pharma industry is full of morons*. I've seen this before from academics under various guises of "Rational Drug Design" and the like. I am, shall we say, skeptical. In this particular bit of testimony on the "we're smarter than they are, nyah, nyah" defense for NCATS a BigPharma type observed that FC is full of stuff and nonsense:
That view was challenged later in the hearing by Roy Vagelos, the former CEO of Merck, who said that the pharmaceutical industry spends about $50 billion annually, or roughly 100 times the NCATS budget, without solving the problems, like inadequate toxicology, that cause so many failures in drug development . “Does anyone in the audience believe that there is something that NCATS is going to do that the industry thinks is critical and that they are not doing? That is incredible to think that. If you believe that you believe in fairies.”
Vagelos added that, with success rates for applicants for NIH grants at historic lows, “We would be doing a lot more good for getting important new drugs on the market,” by funding more young investigators.
Word, PharmaDude, word.
__
*unless it addresses things that the profit industry is not really capable of grappling with such as their penchant for huge payoff, block buster, serves everyone type of drugs.
Summary Statement Paranoia
An interesting viewpoint popped up on writedit's blog over the past couple of days. A commenter who is apparently not yet in an Assistant Professor position has managed to get a nice little grant score:
My RO1 is scored 23 with a percentile is 10. I am now applying for an AP position.
Sweet. Nice accomplishment. Where's the problem?
An University that I just visited aksed me to email them the summary statement. I was wondering if this is appropriated?
Ummmm, well, sure. I mean, particularly if this applicant has informed the search committee / Chair / Dean that s/he has this particular score in hand. Let us be clear. Many, many hiring departments are trying to game out the potential each new hire has for funding. Funding in the short term and funding in the long term. Funding is tight all over, including internal pilot funds, and all things equal the Department wants to hire someone who can get and sustain major funding. Rapidly.
Because that is the lynchpin to scholarly research (again, for many research-oriented job categories), to productivity, to tenure and ultimately to burnishing the Department's reputation.
In some places we are even hearing rumour that search committees are not really considering anyone who doesn't already come with some sort of research funding. This can be via informal or formal (Dean diktat) rules...or merely via competition within the applicant pool.
Consequently, any evidence of a fundable score on a NIH grant is pretty meaningful. Meaningful to your chances before you make the short list and even more meaningful once they've brought you out to interview.
Correspondingly I think the applicant would be EAGER to show off a score like this and send off the summary statement post haste.
Apparently opinions vary:
My RO1 is currently pending and the council meeting will be in May. If another university, not my current institution, contacts my program officer, it will mean that I am leaving from my current institution. The score of a RO1 is based on the environment of applicant’s current institution. Will leaving this environment be considered a negative factor by the council meeting and the program officer, and thus influence their decision on whether they would like to fund my RO1 or not?
additional crosstalk went off the rails. writedit: They should only need the first page in that case.
more writedit: The request for the summary statement is not appropriate, especially so early in the application/interview stage.
the original commenter ate this up:
I probably will send them the first page of the SS, but will mask the confidential information, such as the PO contact information and my application ID. This might offend the prospective univeristy, to avoid this, I may call them first to let them know my concern.
and some other commenter did as well:
The bigger concern is perhaps that there may specific criticisms regarding you, your team, and the institution. There may be comments in SS that highlight any weaknesses that are really not the business of the perspective institution.
This is nuts. Absolute nuts. First, the PO is going to be happy that the applicant has secured a tenure track appointment. This is a good thing. It is doubtful this is going to put the PO off funding the application. Unless there is something in the way of highly unusual circumstances like a unique resource or some spendy equipment that cannot be duplicated at the new place.
Second, the prospective University is not going to contact the PO....what would they be asking anyway? Whether some other University's grant application will be funded? The PO will tell them to butt out.
Third, criticisms in a tenth percentile summary statement are going to put the hiring University off? If they are this stupid, this person doesn't want to work there, if you ask me. Seriously though, what are they going to read? "This Investigator is well qualified and promising", that's about it.
Finally, all this "masking" the grant number and "only the first page" stuff is seriously silly. All it is going to do is make the prospective University wonder what sort of paranoid nutjob they have on the line and whether they've made a serious mistake in trying to hire this wackaloon.
Perfection
The notion that there is some perfect pedigree, some perfect CV that most applicants to the NIH (or other funding body, I assume) possess is untrue.
Oh, it can be comforting, I realize. To think that if you only had picked the right doctoral or post doc lab, if only you had been able to move across the country, if only that damn PI had written you a better letter and gotten you that job at HighFalutinU....
If only.
But for those slings and arrows of outrageous fortune, you too would get every application funded. Nary a triage and seldom a revision.
Hahahahahhahahahahaaaa!!!!
Get real.
Many, many funded PI's of my acquaintance have holes in their CVs that you can drive the ParanoidApplicantTrain through. The wrong doctoral Uni/program. Dismal productivity in a crucial stop. No post doc. Too many postdoc years. A less-than appointment. Seeming lack of independence. A mid career research drought. Low IF pubs. A scientific diversion. Too narrow a scientific focus. Too diffuse.
The thing is...the extramural, NIH-funded community is diverse on this axis. It permits a lot of room for Investigators who differ from the "ideal". To the degree that the "ideal" is more of a fantasy of the unsuccessful applicant than it is a reality.
My charge to you is this: Ignore your seeming deficits.
Ignore the inner voice telling you that easy street was back there on the path not taken. Or the path that was barred to you.
Focus instead on crafting the story of you, as a scientist and investigator. How did your experiences make you the independent scientist that you are?
Remember that you are not talking to your detractors but rather to your advocates.
Alcohol research fraudster Michael W. Miller
Per ORI, one Michael W Miller, most recently Chair of Neuroscience and Physiology at SUNY Upstate Medical University, is a data faker.
ORI finds that the Respondent engaged in research misconduct by falsifying and/or fabricating data that were included in grant applications R01 AA07568-18, R01 AA07568-18A1, R01 AA006916-25, and P50 AA017823-01 and in the following:
The "following" included some paper retractions detailed over at the RetractionWatch blog.
One of the interesting things is that this guy published in decidedly normal journals. There was something in the ORI finding about a PNAS submission, but that seemed to be the high IF watermark for Miller. I make special note of this since I am one of those fond of pointing out the positive correlation between journal IF and retractions.
You will be unsurprised that my attention is drawn in this case to the grant support. That year -18 grant renewal application mentioned? It got converted into an R37 MERIT (10 yrs of non competing renewals instead of the usual 5) in the A1 version. Which means it was scored very highly, from what I deduce about the R37 process. The P50 is, of course, a Center.
Big monetary commitment for NIAAA and very prestigious for Professor Miller.
Boo, hiss, fraudster bad guy....
Except think about those folks who didn't get something because if this guy. The Chair position was an external hire. The P50 took the place of another one- and it isn't just the PI/PD. Each competing Center that didn't get funded probably also had a handful of Component PIs. Who put in a lot of hard work and had a lot of great science. The R37? Well it probably counts at least double because of the 10 year interval. And of course some other worthy mid-career NIAAA scientist didn't receive this honor for her work.
I'm irritated on behalf of anyone who applied to NIAAA for grant support and didn't get the award during the interval they were supporting this Miller fraudster.
Final note. He was supposedly ratted out by someone in his lab. Since the offense seemed to be making up bar graphs rather than the all-too-typical duplicated gel/blot/image, there really would have been no other way to nail him. So the reviewers really can't be blamed for missing anything.
Reviewing NIH Grants: Innocent until proven guilty?
Commenter Physician Scientist notes on a prior post that an individual scientist under suspicion for several dubious papers has retained his NIH funding.
OMG
This grant was RENEWED!!!!
Not quite. Or maybe the timeline is not quite what it seems. This would appear to be the most recent competing award in 2009. The budget is listed as ending in 2010 but then it continues onward under the "7R01" code in the next year (indicating a change of University) until 2014.
So the objection may not be all that direct if the news of these alleged frauds, misdeeds and/or actual retractions and corrections hadn't been known when the grant was reviewed.
However, davebridges continues from my more general query about whether PIs should be viewed as innocent* until proven guilty of fraud.
innocent until proven guilty, but im sure reviewers sure can take into account historical accuracy of a lab. Better to renew a grant of a good lab with an instance sloppy record keeping (if thats the case) than a non-retraction lab whose data is never reproducible
This brings up a related, and more pernicious, issue. In my limited experience, "lab whose data is never reproducible" tends to be the stuff of rumor. Word around the campfire. Suspicion. Widespread far beyond those who might actually have tried seriously to replicate said data.
Correct or false, rarely is it a matter of well-explicated, scientific lack-of-evidence. Which, in itself, would still be problematic. There are many Nobel prizes and other fantastic scientific discoveries with a back story of "nobody believes his data". At least at first, but that could have continued for years or decades. But if it is only suspicion? even if there are a couple of retracted papers....
Should the grant reviewers bust on an application on this basis? If there was one retracted paper would you refuse to issue a fundable score, even if the application had little to do with the topic of the retracted work?
What if you've read some internet clown detailing the "obvious" duplications of figures in papers but they've yet to be retracted or corrected? Would you mark the present application from that PI downward?
The flip side is that nobody deserves NIH funding. It is a privilege that is getting rarer all the time, going by the success rates. Seemingly the proposals that make it over the bar are held to the highest standard. As we've noted repeatedly, there are LOTS of great applications which are not going to get funded.
So why should we (the system) tolerate even a whiff of impropriety? Why not apply the one-strike and yer out principle?
As you know, we had one major ass retraction in the substance abuse fields in recent memory. Major because of the profile and public interest rather than because it had broad influence on the other scientists. I mean sure, maybe people were trying to replicate and follow up but the retraction came out within a year. Not too much damage was done**.
As far as I can tell Ricaurte kept his grants and kept getting more of them. Never paid any obvious price. Was this right? Should he have been busted out of the business for something over which we still do not know, and will never know, the extent of culpability. Should the reviewers simply moved on to a less tainted individual?
I don't know. All I know is what I would do as a reviewer which is to try to be as fair as possible and to rely on my fellow panel members to reach consensus over how retractions or more suspicions should be viewed.
What would you do?
__
*from what I can tell in the chatter, this Chu case is limited to suspicion and a few retractions so far?
**yes, if you were the one wasting a year of work it sucked.
