Archive for the 'Grant Review' category

On resubmitting unfunded A1 NIH grant applications

Apr 08 2014 Published by under Grant Review, Grantsmanship, NIH, NIH funding

Well, well, well.

The NIH limited applicants to a single revision ("amendment", hence -01A1 version) of an unfunded "new" grant submission (the -01 version, sometimes called "A0") in 2009.

This followed the action in 1997 to limit revisions to two (see RockTalk chart), which hurt PIs like Croce and Perrin. (Six revision? Wow, that is some serious persistence guys, my hat is off.)

I wasn't really paying attention to such matters in 1997 but there was some screaming in 2009, let me tell you.
Delusional Biomedical Researchers Seek Repeal Of Arithmetic
More on the new NIH policy on grant application revisions

Initial outcome of limiting NIH apps to a single revision?


NIH re-evaluating ‘two strikes’ rule – Updated

Crocodile tears from experienced NIH investigators over the discontinued A2 revision

I don't know how many people actually got stuck in the filter for submitting a A0 that was too similar to their prior, unfunded A1. I heard of a few, so it did happen. On the flip side of that, I've sure as heck been putting in more than two versions of a proposal which is designed to fund the same area of interest in my laboratory. I have not yet been flagged for it. My initial reaction that any PI who has an ounce of creativity ought to be able to come up with a credible alternative take on their project is still my current take.

Nevertheless, rumor has it that changes are in the wind.

Pinko Punko made an interesting comment on the blog:

DM, I heard the craziest thing today- the possibility of removing the "substantial revision" criterion for new A0 related to previous A1. Supposedly announcement soon- I was kind of surprised.

This was news to me but I have heard things from about five independent sources in the past few days that tend to confirm that changes are being considered.

The most consistent rumor is that new grants will no longer be checked for similarity to prior unfunded proposals. They will be called new grants, but there is no apparent reason for this. In all ways I can see, this is going to be a return to the days prior to 1997 where you could just endlessly revise until the study section relented.

The supposed benefit of reduced "time to award from original proposal" is now going totally by the wayside. I mean, the NIH will still be able to lie and say "look it was an A0!" if they want to but this is even less credible.

More dangerously, the will of study sections to endlessly queue applications will be released from whatever tepid effect the A1 limit has produced.

This is a very BadThing.

__
whoa. I found three A7 projects. All three are competing continuations. I can't EVEN....five and six year apparent funding gaps for two of them. The other I can't work out why there is no apparent gap in funding.

31 responses so far

Sharing (in science) with people you don't particularly like

Mar 24 2014 Published by under Academics, Grant Review, Grantsmanship

The Twitt @tehbride raised an interesting mentoring question:

 

As you are likely aware Dear Reader, due to the accident and intent of where I tend to sit on the scientific spectrum, the scooping type of competition is not a huge part of my professional life. That is, I have managed to get by to this point by not being terribly afraid of people knowing what I am working on or what I plan to work on. Part of this has to do with playing at a level of publication that is not obsessed with the very first person to demonstrate something. Part of it is selecting research questions that are not densely populated with dozens or scores of other laboratories trying to scratch the same flea. Part of it is my overweening and misplaced self-confidence that we did it better, dammit, so who cares who published first.

 

Part of it is pure wrongheadedness on my part, no doubt.

When it comes to grants, specifically, I was always around people who were reflexively generous with sharing their applications when I was a late-postdoc and an early-career faculty member. As time has gone on and more people are asking me for my proposals than I feel the need to ask, I have given mine out to anyone who requests them. (Usually with a little lecture about how my "successful" apps are no more informative than my triaged ones, of course.)

So take that into account.

On a purely tactical level, it is possible for the postdoc in this situation to simply refuse. We can extend this to PIs who are asked for their successful grant applications. You can just say no.

It seems to me to be unwise to do so, particularly when it comes to an application that has been successful. Even if you cannot stand the person who is asking. It just seem churlish when the cost to you is so low.

Is it going to give this person ScienceEnemy little boost ahead? Sure. But remember, the odds of funding are still very steep. So it isn't like you are handing them an award. They still have to write a credible application. And get lucky. So why not*? It costs you essentially nothing to email over your application.

On a strategic level, this person could be your colleague in science for a long time. They could very well be in a position to review you and your work, particularly if they are in a related area of science. And even if they annoy you, it isn't necessarily the case that they have so much as noticed. Lots of annoying people are kind of unaware... So why make an enemy?

And there is one more thing to consider. If you act within a professional capacity on personal whim and dislike, what does this say about your behavior as an objective peer reviewer? Shouldn't you be able to set aside personal dislike to effectively review the scientific content of a paper or grant proposal? Yes, yes you should.

 

__

*Now, if you think the person is a data fraud or something...well that is entirely different.

13 responses so far

Ask DrugMonkey: How do we focus the reviewer on 'Innovation'?

Mar 18 2014 Published by under Fixing the NIH, Grant Review, NIH, NIH funding

As you are aware, Dear Reader, despite attempts by the NIH to focus the grant reviewer on the "Innovation" criterion, the available data show that the overall Impact score for a NIH Grant application correlates best with Significance and Approach.

Jeremy Berg first posted data from NIGMS showing that Innovation was a distant third behind Significance and Approach. See Berg's blogposts for the correlations with NIGMS grants alone and a followup post on NIH-wide data broken out for each IC. The latter emphasized how Approach is much more of a driver than any other of the criterion scores.

This brings me to a query recently directed to the blog which wanted to know if the commentariat here had any brilliant ideas on how to effectively focus reviewer attention on the Innovation criterion.

There is a discussion to be had about novel approaches supporting innovative research. I can see that the Overall Impact score is correlated better with the Approach and not very well with the Innovation criterion score. This is the case even for funding mechanisms which are supposed to be targeting innovative research, including specific RFAs (i.e., not only the R21).

From one side, it is understandable because reviewers' concerns over the high risk associated with innovative research and lack of solid preliminary data. But on the other side, risk is the very nature of innovative research and the application should not be criticized heavily for this supposed weakness. From my view, for innovative research, the overall score should be correlated well with Innovation score.

So, I am wondering whether the language for these existing review criteria should be revised, whether additional review criterion instructing reviewers to appropriately evaluate innovation should be added and how this might be accomplished. (N.b. heavily edited for anonymity and other reasons. Apologies to the original questioner for any inaccuracies this introduced -DM)

My take on NIH grant reviewer instruction is that the NIH should do a lot more of it, instead of issuing ill-considered platitudes and then wringing their hands about a lack of result. My experience suggests that reviewers are actually really good (on average) about trying to do a fair job of the task set in front of them. The variability and frustration that we see applicants express about significantly divergent reviews of their proposals reflects, I believe, differential reviewer interpretation about what the job is supposed to be. This is a direct reflection of the uncertainty of instruction, and the degree to which the instruction cannot possibly fit the task.

With respect to the first point, Significance is an excellent example. What is "Significant" to a given reviewer? Well, there is wide latitude.

Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

Well? What is the reviewer to do with this? Is the ultimate pizza combo of "all of the above" the best? Is the reviewer's pet "important problem" far more important than any sort of attempt to look at the field as a whole? For that matter, why should the field as a whole trump the Small Town Grocer interest...after all, the very diversity of research interests is what protects us from group-think harms. Is technical capability sufficient? Is health advance sufficient? Does the one trump the other? How the hell does anyone know what will prove to be a "critical" barrier and what will be a false summit?

To come back to my correspondent's question, I don't particularly want the NIH to get more focused on this criterion. I think any and all of the above CAN represent a highly significant aspect of a grant proposal. Reviewers (and applicants) should be allowed to wrangle over this. Perhaps even more important for today's topic, the Significance recommendations from NIH seem to me to capture almost everything that a peer scientist might be looking for as "Significance". It captures the natural distribution of what the extramural scientists feel is important in a grant proposal.

You may have noticed over the years that for me, "Significance" is the most important criterion. In particular, I would like to see Approach de-emphasized because I think this is the most kabuki-theatre-like aspect of review. (The short version is that I think nitpicking well-experienced* investigators' description of what they plan to do is useless in affecting the eventual conduct of the science.)

Where I might improve reviewer instruction on this area is trying to get them to be clear about which of these suggested aspects of Significance are being addressed. Then to encourage reviewers to state more clearly why/why not these sub-criteria should be viewed as strengths or lack thereof.

With respect to second point raised by the correspondent, the Innovation criterion is a clear problem. One NIH site says this about the judgment of Innovation:

Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

The trouble is not a lack of reviewer instruction, however. The fact is that many of us extramural scientists simply do not buy into the idea that every valuable NIH Grant application has to be innovative. Nor do we think that mere Innovation (as reflected in the above questions) is the most important thing. This makes it a different problem when this is co-equal with criteria for which the very existence as a major criterion is not in debate.

I think a recognition of this disconnect would go a long way to addressing the NIH's apparent goal of increasing innovation. The most effective thing that they could do, in my view, is to remove Innovation as one of the five general review criteria. This move could then be coupled to increased emphasis on FOA criteria and an issuance of Program Announcements and RFAs that were highly targeted to Innovation.

For an SEP convened in response to an RFA or PAR that emphasizes innovation....well, this should be relatively easy. The SRO simply needs to hammer relentlessly on the idea that the panel should prioritize Innovation as defined by...whatever. Use the existing verbiage quoted above, change it around a little....doesn't really matter.

As I said above, I believe that reviewers are indeed capable of setting aside their own derived criteria** and using the criteria they are given. NIH just has to be willing to give very specific guidance. If the SRO / Chair of a study section make it clear that Innovation is to be prioritized over Approach then it is easy during discussion to hammer down an "Approach" fan. Sure, it will not be perfect. But it would help a lot. I predict.

I'll leave you with the key question though. If you were to try to get reviewers to focus on Innovation, how would you accomplish this goal?

___
*Asst Professor and above. By the time someone lands a professorial job in biomedicine they know how to conduct a dang research project. Furthermore, most of the objections to Approach in grant review are the proper province of manuscript review.

**When it comes to training a reviewer how to behave on study section, the first point of attack is the way that s/he has perceived the treatment of their own grant applications in the past***. The second bit of training is the first round or two of study section service. Every section has a cultural tone. It can even be explicit during discussion such as "Well, yes it is Significant and Innovative but we would never give a good score to such a crappy Approach section". A comment like that makes it pretty clear to a new-ish reviewer on the panel that everything takes a back seat to Approach. Another panel might be positively obsessed with Innovation and care very little for the point-by-point detailing of experimental hypotheses and interpretations of various predicted outcomes.

***It is my belief that this is a significant root cause of "All those Assistant Professors on study section don't know how to review! They are too nitpicky! They do not respect my awesome track record! What do you mean they question my productivity because I list three grants on each paper?" complaining.

12 responses so far

Your Grant in Review: I Can't Work With This

Mar 14 2014 Published by under Grant Review, Grantsmanship, NIH, NIH Careerism

Reminder. You are going to have advocates and detractor reviewing your grant proposals. Your goal is to give the advocate what she needs to promote your proposal.

No matter how much the advocate might love the essential ideas in your application, nothing good is going to happen if you violate every rule of basic grantsmithing.

At the least you should be able to put together a proposal that gets it mostly right. Credible. Serious. Without huge gaping holes or obvious piles of StockCritique bait lying around everywhere.

It should not be hard to give the advocating reviewer something they can work with.

17 responses so far

A NIH Grant Applicant's Calming Mantra

Mar 10 2014 Published by under Grant Review, Grantsmanship, NIH Careerism

"No matter how ridiculous the summary statement comment, it helps you to write better proposals in the future."

11 responses so far

Potnia Theron on Stock Criticism of NIH Grants

Dec 13 2013 Published by under Grant Review, Grantsmanship

Apparently Potnia is going to do a series over at Mistress of the Animals blog. This statement is one of those mnemonic gems you should paste on your monitor edge.

Aims should be general enough to require a project (1-2 papers per aim), but specific enough that they are a project.

One response so far

STAY TUNED: CSR / NIH is reconsidering their options for study sections

Elevating a comment from Chris:

IMPORTANT UPDATE: My study section (originally scheduled for Oct 1-2, then cancelled until Feb) is now back on! I just heard from our chair that we will be holding an online meeting in the next few days (thanks for the advance notice). According to our chair, the "significant pressure" put on the NIH from the extramural community has led them to reconsider their decision to cancel everything and bring things back online. I have no authority on this, but would assume that this will happen across the board. So take heart, all may not be lost for this round...

I am likewise hearing rumor that the CSR is reconsidering what they are going to do.

Stay tuned folks, this ride ain't over yet.

Updated: ps, your comments at Rock Talk can't help but be viewed as part of the "significant pressure". Go to it.

Update 2:

16 responses so far

Government shutdown NIH Grant strategery

Oct 17 2013 Published by under Grant Review, Grantsmanship, NIH

There's at least one early indicator of what is going to happen with the study section rounds that were cancelled because of the government shutdown.

This has all sorts of implications, one of which was brought up by Professor Jentsch in a subsequent tweet. It is related to the NOTice just issued which says that all October deadlines will be pushed forward into the "November timeframe".

Let's say your submitted a new proposal in June or perhaps a revised or competing renewal proposal in July. And like a busy little beaver you've continued to work on the project. Perhaps you have some excellent new data that further supports your awesome ideas and the killer experiments that you've proposed.

There is only one thing to do. Pull the grant from consideration and resubmit it, with the new data, once the NIH picks some November deadlines.

44 responses so far

You know how you like to complain about your NIH grant reviewers?

Oct 02 2013 Published by under Grant Review, NIH

Remember, o ye NIH grant seeking Readers, that your peers are supposed to be reviewing the grants you submitted in the June/July interval right about now. And thanks to the House of GOP shuttering the Federal government, the study sections are being cancelled.

Maybe.

You see, maybe a Continuing Resolution will be passed....tonight? or tomorrow morning? or Friday at 2?

And then the study section meetings for next week will be back on.

So the reviewers have to struggle along and finish up their jobs as best they can, not knowing if it will be for the meeting that is scheduled....or if it will be some sort of replacement meeting later in the month or year.

From what I am hearing, your friendly peers are stepping up to the damn plate and getting their grants reviewed.

Even without access to eRA Commons (where all the grants are stored, hardly anyone bothers to get a CD anymore). So the "read phase" that is supposed to take place the week before a study section meeting is going to be difficult. Hard to read the other reviewers' comments on the grants you are assigned because you don't know who they are! All that is supposed to be automatic on Commons you see. Well, from what I hear around the campfire, the sections are doing what they can, no doubt with heroic work from the Chairs and a little illegal subterranean rogue work from the government employee SROs.

I thank you all.

72 responses so far

Dodging Vaporware: In Preparation, In Submission, Under Review, Accepted Pending

As we all know, much of the evaluation of scientists for various important career purposes involves the record of published work.

More is better.

We also know that, at any given point in time, one might have work that will eventually be published that is not, quiiiiiite, actually published. And one would like to gain credit for such work.

This is most important when you have relatively few papers of "X" quality and this next bit of work will satisfy the "X" demand.

This can mean first-author papers, papers from a given training stint (like a 3-5 yr postdoc) or the first paper(s) from a new Asst Professor's lab. It may mean papers associated with a particular grant award or papers conducted in collaboration with a specific set of co-authors. It could mean the first paper(s) associated with a new research direction for the author.

Consequently, we wish to list items that are not-yet-papers in a way that implies they are inevitably going to be real papers. Published papers.

The problem is that of vaporware. Listing paper titles and authors with an indication that it is "in preparation" is the easiest thing in the world. I must have a half-dozen (10?) projects at various stages of completion that are in preparation for publication. Not all of these are going to be published papers and so it would be wrong for me to pretend that they were.

Hardliners, and the NIH biosketch rules, insist that published is published and all other manuscripts do not exist.

In this case, "published" is generally the threshold of receiving the decision letter from the journal Editor that the paper is accepted for publication. In this case the manuscript may be listed as "in press". Yes, this is a holdover term from the old days. Some people, and institutions requiring you to submit a CV, insist that this is the minimum threshold.

But there are other situations in which there are no rules and you can get away with whatever you like.

I'd suggest two rules of thumb. Try to follow the community standards for whatever the purpose and avoid looking like a big steaming hosepipe of vapor.

"In preparation" is the slipperiest of terms and is to be generally avoided. I'd say if you are anything beyond the very newest of authors with very few publications then skip this term as much as possible.

I'd suggest that "in submission" and "under review" are fine and it looks really good if that is backed up with the journal's ID number that it assigned to your submission.

Obviously, I suggest this for manuscripts that actually have been submitted somewhere and/or are out for review.

It is a really bad idea to lie. A bad idea to make up endless manuscripts in preparation, unless you have a draft of a manuscript, with figures, that you can show on demand.

Where it gets tricky is what you do after a manuscript comes back from the journal with a decision.

What if it has been rejected? Then it is right back to the in preparation category, right? But on the other hand, whatever perception of it being a real manuscript is conferred by "in submission" is still true. A manuscript good enough that you would submit it for consideration. Right? So personally I wouldn't get to fussed if it is still described as in submission, particularly if you know you are going to send it right back out essentially as-is. If it's been hammered so hard in review that you need to do a lot more work then perhaps you'd better stick it back in the in preparation stack.

What if it comes back from a journal with an invitation to revise and resubmit it? Well, I think it is totally kosher to describe it as under review, even if it is currently on your desk. This is part of the review process, right?

Next we come to a slightly less kosher thing which I see pretty frequently in the context of grant and fellowship review. Occasionally from postdoctoral applicants. It is when the manuscript is listed as "accepted, pending (minor) revision".

Oh, I do not like this Sam I Am.

The paper is not accepted for publication until it is accepted. Period. I am not familiar with any journals which have accepted pending revision as a formal decision category and even if such exist that little word pending makes my eyebrow raise. I'd rather just see "Interim decision: minor revisions" but for some reason I never see this phrasing. Weird. It would be even better to just list it as under review.

Final note is that the acceptability of listing less-than-published stuff on your CV or biosketch or Progress Report varies with your career tenure, in my view. In a fellowship application where the poor postdoc has only one middle author pub from grad school and the two first author works are just being submitted...well I have some sympathy. A senior type with several pages of PubMed results? Hmmmm, what are you trying to pull here. As I said above, maybe if there is a clear reason to have to fluff the record. Maybe it is only the third paper from a 5 yr grant and you really need to know about this to review their continuation proposal. I can see that. I have sympathies. But a list of 8 manuscripts from disparate projects in the lab that are all in preparation? Boooo-gus.

32 responses so far

« Newer posts Older posts »