Anyone out there ever reviewed SBIR or STTR grants for the NIH? Any thoughts on what seems to be most important, common pitfalls and the like? Any thoughts on how the review discussion tends to differ from standard R01 review?
I have essentially zero experience with these mechanisms and some reader was asking.....
A question arrived about publication expectations for trainees at the blog mailbox recently.
I was wondering if you would consider a blog post and perhaps encouraging discussion on a related topic, on how do you evaluate your student/postdoc performance and how common is the 1 paper/yr "rule"?
At the outset I was skeptical that much use would come of trying to answer this because the real answer is "It depends very much on subfield and ultimate career aspirations, therefore broad sweeping pronouncements are of little value.". And this is true. But what the heck? I'll give you my thoughts from my point of view, no doubt some others will go shitnutz about how it is clearly different and maybe we can hash out the space of useful answers.
Some detailed stuff that I thought about, but often are not discussed thoroughly include:
- I always assumed that when people talk about 1 paper/yr it refers to 1 first-author paper but not in a top-tier journal (usually "best in the sub-field" journal, e.g. Org. Lett., J. Med. Chem., etc.)
Yeah. I think one paper per year is a pretty good general starting point. Emphasis on general. For trainees, I think this average will be lower, ditto if you only count first-author papers. But it is a pretty good target expectation for the central tendency. One first author per year in a "top tier" journal is a ridiculously absurd expectation for postdocs. Even one per year in a "top tier" journal as senior author is only possible for the very top laboratories and is therefore not the expectation for everyone. If you can do it, good on you, but it ain't typical. So if you are in a place where you think this is the standard for postdocs? please. I'm familiar with a lab that has probably one of the highest CNS counts ever and the postdocs do not hit one CNS pub per year as first author. They have not done so over the ~15 years I've been watching the lab's production. So anyone who does this out there in the whole postdoc population is the rare exception.
- How do you factor in non-1st author papers? Ignoring the effects of journal IF, would one 1st-author paper = two 2nd-author paper?
There is no direct relationship, I would argue. Non-substitutable quantities. No amount of non-first author papers makes up for not having any first-author papers. They are just that important in the minds of many people, including me. Conversely, the existence of some 2nd-Xth author papers is better than not having any, because more is better when it comes to publications on the CV. I suppose at some point there would be a balance point in which too many Nth author papers starts to subtract from the credit generated by the first-author list. It would be related to the thought of "why doesn't this trainee have more firsts if she is this experimentally productive?".
- Do people even consider anything greater than 2nd-authorship (i.e. having 3rd authorship is basically useless or not counted)? If so, does the level of the prestige of the journal change this perception (i.e. having 3rd authorship in PNAS is equivalent to a 1st-author in some 2nd-tier journal like Biochemistry)?
In my view, no, the Nth author on an article in a higher IF journal doesn't trump first-author in a lesser journal. See above, the Nth authorships count but I would say they are independent of the first-author credits. So within the sphere of Nth authorships, sure, the higher IF is better.
- How do you factor in the prestige or IF of the journal? Does publishing in Science/Nature/Cell count as having 2-3 1st-author papers in 2nd-tier journals?
Indubitably the CNS first-author counts more than several first-authors in lesser journals. One might even suggest that CNS first-author as a postdoc trumps infinity non-CNS first-authors. For some situations. There are those that assert that the presence/absence of very specific journals on the CV is the difference between round-filing an application for an Assistant Professor position and placing it on the long-list for consideration. I credit these assertions but would also point out that there are many perfectly acceptable jobs that would not have this absurd criterion.
- Do people take a time-average (i.e. as long as you get 5 papers in 5 years it's fine), or is having a regular output more important (i.e. would prefer to have 1 paper every year as opposed to 2 papers in 1st year and 3 papers in 5th year but nothing in between)?
I would say that it is only once one becomes a PI that it is ever reasonable to look at consistency of output. This particular example would not even be noticed, I would say. And even then it sort of depends on the type of work you do. I know of multiple types of work in my areas of interest (particularly human studies) that have years of data collection followed by a flurry of papers.
When I have recommended shooting for consistent output, being concerned with whether a manuscript submitted to Journal X at this point in the year will have a pub date from this year, etc it has to do mostly with motivation. Most of the time the pace of submission for a postdoc is not going to be easily controlled. The experiments have their own timeline. Things come up. New things need to be done to wrap up the paper. Then there are the many sources of delay in the review process. There is no reason to obsess about 2 in first year / 3 in fifth over meeting a strict rate of 1 per year for 5 years.
The clock is ever ticking, however and since one cannot go back and fill in missing publication-years, one is best keeping one's eye on the prize. If you haven't had a paper in a two year span, well maybe it is better to dump out a quick one, give up on hitting the highest possible IF, etc. You have to make this judgement thinkingly, of course. And no, there are no formulaic answers such as my correspondent seems to be seeking.
Balance. That is my best suggestion.
The notion that tenure provisions in the contracts of University Professors or school teachers means that they are uniquely protected against firing for incompetency is a scurrilous lie.
Many, many job places that do not have anything called "tenure" do in fact have procedures in place, formal or otherwise, that respect the duration of service to the place of employment.
The notion that those more senior than oneself are undeserving of the job titles, promotions, payrate and/or privileges that you do not currently enjoy is a nice comforting meme. But if you are going to advance a strong accusation against the seniority system and argue about who "deserves" to have a job and who does not, you better bring some ammunition.
Or, you know, run for office as a Republican where that kind of unverified ranting convinces someone useful.
The models indicate that as competition increases, many people can be taken out of the career pathway by... blind, stinking, clueless, doo-da luck.
Just as one can seemingly succeed through alignment of circumstances with a normal level of talent and effort, one can wash out through no fault of one's own too.
But the competition turns out to be very important in this model; and that relates to tenure. Many people want to see tenure replaced with a series of recurring short-term contracts. The authors imply that the short-term model could be harmful for the development of science. A failure in one short-term contract could derail a productive researcher, since early career shocks can ripple throughout a scientist’s career.
And this is why we're in the state of "Do it to Julia, not me, Julia" backstabbing panic about the NIH budget situation. The immense fear on the part of all many of us that the next grant rejection means the end of our career is palpable. Visceral. The anger of the young that they are "better than" half of the existing faculty and therefore deserve that person's job is clear. Very clear.
but be careful about that to which you aspire. Our history of pure Darwinian tooth-and-nail employment is not a pretty one. The dawning of the industrial age showed us how that goes down.
Job protections are there because on average they make all workers' experiences better. Not there to protect the lazy and incompetent. That effect is an unintended consequence.
So when you are ranting your rants about the deadwood tenured fucks, please, do your homework. Show us how dismantling Professorial tenure is not going to rapidly devolve us to the level of the itinerant Adjunct Professor.
The Proceedings of the National Academy of Sciences has an interesting manuscript submission process.
Apart from allowing NAS members to "contribute" a paper from their own lab that they've gotten peer-reviewed themselves, there is a curious distinction for more normal submissions.
The pre-arranged editor track permits you to find a PNAS editor before you submit it. Presumably a friendly editor.
In the best case it is similar to a pre-submission inquiry practiced formally or informally at the GlamourMags. In the worst case, an end run around "pure" peer-review via the Insider's Club.
(The end run being as benign as simply avoiding the desk-reject and as pernicious as getting a gamed peer-review.)
But is this any different from other journals? GlamourEditors require some buttering up. They brag in unguarded moments about how much they've "worked with" the authors to make the paper awesome. So many of those papers end up functionally identical to having a pre-arranged editor who has agreed to handle the manuscript.
In pedestrian-journal land, one can easily go Editor hunting. If a host of journals sort-of fit, and the IFs are indistinguishable, then it behooves the authors to seek a journal with a friendly Associate Editor. And to ask for that person in the many submission systems that permit such requests.
So really, how does the PNAS system really differ?
In fact, you might see that as being more honest and transparent.
As the Impact Factor discussion has been percolating along (Stephen Curry, Björn Brembs, YHN) it has touched briefly on the core valuation of a scientific paper: Citations!
Coincidentally, a couple of twitter remarks today also reinforced the idea that what we are all really after is other people who cite our work.
Dr24hrs:
More people should cite my papers.
I totally agree. More people should cite my papers. Often.
was a bit discouraged when a few papers were pub'ed recently that conceivably could have cited mine
Yep. I've had that feeling on occasion and it stings. Especially early in the career when you have relatively few publications to your name, it can feel like you haven't really arrived yet until people are citing your work.
Before we get too far into this discussion, let us all pause and remember that all of the specifics of citation numbers, citation speed and citation practices are going to be very subfield dependent. Sometimes our best discussions are enhanced by dissecting these differences but let's try not to act like nobody recognizes this, even though I'm going to do so for the balance of the post....
So, why might you not be getting cited and what can you do about it? (in no particular order)
1) Time. I dealt with this in a prior post on gaming the impact factor by having a lengthy pre-publication queue. The fact of the matter is that it takes a long time for a study that is primarily motivated by your paper to reach publication. As in, several years of time. So be patient.
2) Time (b). As pointed out by Odyssey, sometimes a paper that just appeared reached final draft status 1, 2 or more years ago and the authors have been fighting the publication process ever since. Sure, occasionally they'll slip in a few new references when revising for yet the umpteenth time but this is limited.
3) Your paper doesn't hit the sweet spot. Speaking for myself, my citation practices lean this way for any given point I'm trying to make. The first, best and most recent. Rationale's vary and I would assume most of us can agree that the best, most comprehensive, most elegant and all around most scientifically awesome study is the primary citation. Opinions might vary on primacy but there is a profound sub-current that we must respect the first person to publish something. The most-recent is a nebulous concept because it is a moving target and might have little to do with scientific quality. But all else equal, the more recent citations should give the reader access to the front of the citation thread for the whole body of work. These three concerns are not etched in stone but they inform my citation practices substantially.
4) Journal identity. I don't need to belabor this but suffice it to say some people cite based on the journal identity. This includes Impact Factor, citing papers on the journal to which one is submitting, citing journals thought important to the field, etc. If you didn't happen to publish there but someone else did, you might be passed over.
5) Your paper actually sucks. Look, if you continually fail to get cited when you think you should have been mentioned, maybe your paper(s) just sucks. It is worth considering this. Not to contribute to Imposter Syndrome but if the field is telling you to up your game...up your game.
6) The other authors think your paper sucks (but it doesn't). Water off a duck's back, my friends. We all have our opinions about what makes for a good paper. What is interesting and what is not. That's just the way it goes sometimes. Keep publishing.
7) Nobody knows you, your lab, etc. I know I talk about how anyone can find any paper in PubMed but we all need to remember this is a social business. Scientists cite people they know well, people they've just been chatting with at a poster session and people who have just visited for Departmental seminar. Your work is going to be cited more by people for whom you/it/your lab are most salient. Obviously, you can do something about this factor...get more visible!
8) Shenanigans (a): Sometimes the findings in your paper are, shall we say, inconvenient to the story the authors wish to tell about their data. Either they find it hard to fit it in (even though it is obvious to you) or they realize it compromises the story they wish to advance. Obviously this spans the spectrum from essentially benign to active misrepresentation. Can you really tell which it is? Worth getting angsty about? Rarely.....
9) Shenanigans (b): Sometimes people are motivated to screw you or your lab in some way. They may feel in competition with you and, nothing personal but they don't want to extend any more credit to you than they have to. It happens, it is real. If you cite someone, then the person reading your paper might cite them. If you don't, hey, maybe that person will miss it. Over time, this all contributes to reputation. Other times, you may be on the butt end of disagreements that took place years before. Maybe two people trained in a lab together 30 years ago and still hate each other. Maybe someone scooped someone back in the 80s. Maybe they perceived that a recent paper from your laboratory should have cited them and this is payback time.
10) Nobody knows you, your lab, etc II, electric boogaloo. Cite your own papers. Liberally. The natural way papers come to the attention of the right people is by pulling the threads. Read one paper and then collect all the cited works of interest. Read them and collect the works cited in that paper. Repeat. This is the essence of graduate school if you ask me. And it is a staple behavior of any decent scientist. You pull the threads. So consequently, you need to include all the thread-ends in as many of your own papers as possible. If you don't, why should anyone else? Who else is most motivated to cite your work? Who is most likely to be working on related studies? And if you can't find a place for a citation....
Thanks to a query from a reader off the blog and a resulting request from me, our blog-friend microfool pointed us to some data. Since I don't like Tables, and the figure on the excel file stinks, here is a different graphical depiction:
The red trace depicts success rates from 1962 to 2008 for R01 equivalents (R01, R23, R29, R37). Note that they are not broken down by experienced/new investigators status, nor are new applications distinguished from competing continuation applications. The blue line shows total number of applications reviewed...which may or may not be of interest to you. [update 7/12/12: I forgot to mention that the data in the 60s are listed as "estimated" success rates.]
The bottom line here is that looking at the actual numbers can be handy when playing the latest round of "We had it tougher than you did" at the w(h)ine and cheese hour after departmental seminar. Success rates end at an unusually low point...and these numbers stop in 2008. We're seeing 15% for R01s (only) in FY2011.
Things are worse than they've ever been and these dismal patterns have bee sustained for much longer. If we look at the ~30% success rates that ruled the day from 1980-2003, the divergence from the trend from about 1989 to 1996 was interrupted in the middle and, of course, saw steady improvement in the latter half. The badness that started in FY2004 has been 8 unrelieved Fiscal Years and shows no sign of abatement. Plus, the nadir (to date) is much lower.
Anyone who tries to tell you they had it as hard or harder at any time in the past versus now is high as a kite. Period.
Now, of course, it IS true that someone may have had it more difficult in the past than they do now, simply because it has always been harder for the inexperienced PIs to win their funding.
sourceAs we know from prior posts, career-stage differences matter a LOT. In the 80s when the overall success rate was 30%, you can see that newcomers were at about 20% and established investigators were enjoying at least a 17%age point advantage (I think these data also conflate competing continuation with new applications so there's another important factor buried in the "Experienced" trace.) Nevertheless, since the Experienced/New gap was similar from 1980 to 2006, we can probably assume it held true prior to that interval as well.
Per NOT-DA-12-016,
NIDA and NIAAA will not accept any K05 applications beyond the November 2012/ January 2013 receipt dates.
For those unfamiliar, the latest PA-12-148 FOA for the K05 is here and the purpose is listed as:
The purpose of the Senior Scientist Research (K05) is intended to provide protected time for outstanding senior scientists who have demonstrated a sustained high level of productivity conducting biomedical research relevant to the scientific mission of the appropriate institute to focus on their research and to provide mentoring of new investigators.
The DM Executive Summary is, "Salary support for BigSchwangingTypes to relieve the burden on their research awards, covered by a faked-up need for buyout from local Institutional responsibilities with regard to teaching and service".
The junior faculty summary is, "What another total scam by which the GoodOldBoyes/Girlz extract yet MORE money out of the NIH grant system!"
Note: If it were ever possible, you are damn right I would have applied for one of these in a heartbeat and I'm bitter that it was never possible for me to get one of these schweeeeet deals 
.
So when I first saw this notice I just thought "Good!" and moved on.
However.
A reader of the blog wrote to ask for my take on this and further observed:
I am curious how this will impact these senior "NIDA" investigators and their current/future grant applications? If these K05 scientist now have to put more of their effort on their NIDA grants, what impact will that have on their grants, their trainees who are funded off their grants, etc.
I guess I am a little shocked, and happy, that they are finally sticking it to the senior scientists a little. This may indirectly help out us younger scientists?
First of all, I should make clear I have no insight into why NIDA and NIAAA have chosen to dismantle this program at this time. Haven't heard any rumours about it at all.
Charging on over to RePORTER I find that there are only 33 K05s funded between the two institutes (13 NIAAA) at present. This is not a tremendously large number. The total costs seem to run from about $120K per year up to about $250K...over a R03 equivalent on the low end and we're talking R21 territory for the bigger ones. I guess 30-50 more of the smaller R grants funded every year would be a good improvement.
Alternately, they could use these to offset one-module reductions for a larger number of R01 apps, or get perhaps 15 more R01s funded entirely. NIDA has about 90 new R01s funded in FY2012 (to date) and they funded about 122 new R01s in FY2011. Therefore, adding 10-15 more is a significant improvement. This could very well be the only reason. They are searching under the couch cushions for a way, any way, to keep the R-mech success rates up.
Getting back to the emailer's suggestion that this is a way to stick it to the senior investigators....well, that isn't quite clear. The senior investigator salaries have to come from somewhere. And as I alluded to, I think the whole idea of buying out local institutional service time was a bit of a sham. The K05 serves, in part, to relieve the burden on the PIs' R grants. If they didn't have the K05 buyout, they'd be looking to land another R grant to get the same percentage of their salary covered.
They will return to doing so (and here we have a nebulous, fluid population "they" which is made up of the current awardees and the next set of potential future K05 awardees). Putting in more R or P mechanism proposals that will compete with, you guessed it, everyone else. Including the junior investigators.
So it isn't the case that this magically frees up money for the younger set to obtain. It throws the money, yes, but also the Greybearded and Bluehaired Professors back into the R-mech pool.
In yesterday's discussion, I finally got a partial glimpse of the issue when NatC observed:
Discussions about how to manage and plan protected pockets of time OUTSIDE work to do whatever - walk the bulldog, play music, train for a triathlon, watch baseball, play with your kids or nieces/nephews ir travel - would be extremely valuable work/life balance discussions to have early in this sometimes crazy career.
In full disclosure this has rarely been a problem for me. I've managed to get to where I am today (such as it is) with what I think is a healthy balance of work-to-life. Obviously some, including my spouse, might disagree but the important thing is that I think this is the case. We're talking personal, subjective "balance" here and nobody can define it for you. If you have reached it, you are going to be relatively happier and if you feel imbalanced you are going to feel sad* about it.
Yes, I for damn sure wish for more hours in the day. Yes. Of course. And at each and every major stage there were things being neglected so that I could pursue some other thing. Either in the proximal, days to weeks, or in the long-haul, years to decades(!), perspective. But I have never been an obsessive and any fair read would fail to find any major imbalance.
How did I do it?
I think the most useful and general approach is that you have to be willing to fail.
Let me say it again: YOU HAVE TO BE WILLING TO FAIL!!!!!
I was not, I think, willing to fail at getting the PhD. This was a defined, obtainable target for which the steps were mostly clear to me. Do the research, write that shit up into a dissertation and bob's your uncle.
After that? Well, yes, of course I wanted to succeed career-wise. In one of the professorial paths preferably. But I was willing to...not. To fail.
There have been several defined choice points at which I did the considerably sub-optimal career move for the sake of issues that we shall encompass under "life". (Also career moves which might have in the long run been suboptimal but looked great** at the time. Some of this initial appearance was influenced by "life".) Sometimes I did this out of unthinking ignorance, I will admit. I didn't perhaps realize the magnitude of the risk I was running. But I for damn sure knew there was risk. Risk of not making it in some way. Of not getting on the independent research track. Of not getting funding...or not keeping it. Of letting the lab and research program crash down to nonviability.
This hasn't stopped and it continues to this day.
Is my virtue untested? Some might observe that. From the perspective of some it looks like I have a pretty schweet gig***. From above the waterline it looks okay. Something a disgruntled postdoc or Year 3 faculty member might think is pretty much IT. As in "career accomplished"...all it takes now is running it out like you always wanted to. No risk.
I don't see it that way. I still risk failures of various sorts. Mostly the big axe is the grant funding....and it is a big one, hanging over my head more often than it is not.
So much like the disgruntled postdoc and the terrified junior faculty member...I could always work harder. More. Put in more grants. Squeeze out more papers. Refine my lab efficiency to maximize the data. Chase small project funds. Woo more trainees. Hit the seminar circuit harder. Go to more meetings.
All of this would probably benefit my career. It would make things go better professionally. We'd be more productive, no doubt.
I choose not to. That's it. There's no secret. There's no special case of insulation from the risks of choosing not to work harder than the next person. You risk paying a price.
Balance implies tradeoffs. I've certainly found it to be so. There are costs to go with every benefit. Costs that may be "just" stress, may be health issues (mental or otherwise), may be definable career failures. Having "life" balance makes this inevitable. There will be tradeoffs****, people.
This is my answer to NatC's question. Choose. Choose to take the time. Make room for what is important to you. Realize that by doing so you might fail. You might.
But you know what? These St Kern and Poo types?
I know for damn sure they've failed at life.
And that I was never willing to risk.
__
*don't get a puppy to cheer yourself up.
**so we won't count these, at the time they seemed really pro-career.
***and I do, I do.
****of course it goes both ways. you may be choosing a career path that really isn't compatible with your desire to tour Europe with an opera group every summer. You may have to give up some of the "life" stuff
The demonstrably worst drivers are 1) Beemer pilots, 2) Volvo drivers of the mom demographic, 3) jacked up pickemup trucks with trucknutz idiots and 4) minivan drivers. Lately the ecophreaks driving Prii are making a *strong* showing. But I suspect that is either the dope or the conversion of #2s to Prii.
Discuss.
Christ.
The notion that I have to be all #getoffylawn about the concept of fartlek pains me.
It is not intervals, you do not pluralize the word and you most certainly should not be throwing up at the end of the workout.
