I love it when the data support my position

Sep 27 2010 Published by under Careerism, NIH, NIH Budgets and Economics, NIH funding

We occasionally lapse into discussions around here on various career and grant related topics. One that emerges now and again is what constitutes an acceptable Impact Factor of a journal. Yes, higher is better but out in the real world there are a lot of us publishing most of our work in journals with impact factors that are an order of magnitude lower than Glamour.

I make a big point out of field specific expectations- this is a well known, well discussed..... and generally ignored property of the IF. Nevertheless if we limit ourselves to NIH-funded biomedical research and excuse ourselves from discussing outlier scores....."society level" journals are generally going to be below 10 IF, most frequently in the 2-6 range and with a gut-feeling point of deflection around IF 4 or 5. As in the number of journals really thins out past this point. Whether that reflects some objective or arbitrary standard of quality...that's a debatable point. At any rate, there are those that sneer at journals of almost any IF below Glamour level (i.e., north of 20). Some, such as my coblogger PhysioProf, are known to make comments suggesting that the expected norm is something other than what I describe it to be.

Something else we talk about now and again has to do with the desired target lab size as expressed in Direct Costs value of extramural funding. This is the cash value you have to spend on supplies, personnel, equipment, etc in a given year. The R01 from the NIH is theoretically unlimited however the cap for writing a modular budget (instead of full itemization of expenses) is $250,000 / yr and you have to get special permission to even submit one for over $500,000 per year. Under the presumption that most newbies try to stay under the modular cap (and they should in most cases) my recommendation is that even a starting-out lab should be trying to land 2 R01s. I.e., I suggest that you need $500,000 in direct costs just to give yourself a fighting chance at scientific survival and (modest) success. There are those, likely those that are still fighting just to get one-R01 level funding, that argue that this is wild excess and the source of all that is wrong with success rates at the NIH grant getting game.

Well, wouldn't you know, Director Berg of NIGMS has yet MORE data for us to geek over and it is relevant to these two discussion points. He has a new post up on his blog reviewing the scientific output of NIGMS grants first funded in 2006. Out of nearly 3,000 investigators

...the median annual total direct cost was $220,000, the median number of grant-linked publications was six and the median journal average impact factor was 5.5.

I was particularly struck by the second figure.

A plot of the median number of grant-linked publications from 2007 to mid-2010 (red circles) and median average impact factor for journals in which these papers were published (blue squares) for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006. The shared bars show the interquartile ranges for the number of grant-linked publications (longer red bars) and journal average impact factors (shorter blue bars). The medians are for bins, with the number of investigators in each bin shown below the bars.

Look at that interquartile range for the IF of the journals in which the papers were published. Up to the total laboratory R01/P01 funding level of $500K in direct costs per year, the 75th percentile is still only around an IF of 7.5-8. And the 25th percentile is below 5.0. So even if you do not account for subfield differences already there is plenty of evidence about what is a "normal" IF level for NIGMS awardees. And that norm reads one heck of a lot closer to my concept of normal than that of the GlamourAdvocates around these parts.

The second point is the obvious support for my position that around $500K / year in direct costs is a good target because the number of publications increases pretty steadily up to this point and then flattens out from ~$600K-$1M in direct costs.

There are a couple of interesting questions that naturally occur to one upon reviewing these data. First, the relationship between number of publications linked to a grant may be increasing because you can list more than one grant for a given paper. So if you have two or three grants, it is possible that you have a sort of convenient synergy. Sometimes one project is going better and leads to an extra publication- if you list both of your grants, this might be slightly overcounting the impact of the project which isn't going so well and would not have produced that paper in isolation. Second, it could be that as you get into the zone well north of $500K, you start reaching a different type of laboratory. One that focuses more heavily on GlamourPubs is going to (inevitably?) trade IF for number of publications.

25 responses so far

  • Interesting data. It's interesting how much this varies across field, which itself illustrates the danger of having university-wide standards about publication. Since I'm a psychologist, there aren't many relevant glamour mags, just Science, Nature and PNAS. The short format of Science isn't really appropriate for the field, so its rare that a paper is even plausible for Science, and it's not really where the more interesting papers are published.

    There are a handful of journals in the 10-20 IF range, but they are mostly journals that specialize in review papers (e.g., Behavioral and Brain Sciences, IF 19, which only publishes one long paper per issues, plus commentary) or neuroscience journals, or neuroscience review journals (Nature Reviews Neuroscience, IF 26 -- ok, that's not actually in the 10-20 range). It's great to publish in those, but one does have to occasionally publish empirical research. And if you don't do fMRI or do patient work, the neuroscience journals aren't an option.

    Within the field, I actually think of Psychological Science as a kind of glamor mag, though it's IF is only 5. The papers are short (by our standards) and cute. Despite the fact that Psych Science (in theory) covers the entire field, an informal poll suggests that most of my colleagues think that Cognition (IF 3.6) publishes the best work in our subfield (cognitive psychology), and the 90% rejection rate certainly suggests that lots of people would like to publish in it.

    Why are psych IFs so low? A lot of it sociological. Our field is pretty small. It's rare to cite across subfields. Our reference lists are often short (that is, we don't cite as many papers per paper). Etc. No formal data here -- just what I'm told.

    And a lab with $500k in directs ... there are MRI labs that have that much, but I'm not even sure what a lab like mine would do with that money. Go on a lot of lab retreats, I imagine. It would be hard to spend it on research!

  • Dario Ringach says:

    ...but the data also show that as you move from 200k to 800k the number of publications merely doubled while NIH investment quadrupled. The $/paper increases with increased funding.

    The increase in average IF with increased funding is not impressive at all... but yes, those publishing in Glamour tend to have more funding... but probably also more experience writing good papers.

  • drugmonkey says:

    DR, I would think that i t s obvious that there is a nonlinear relationship between IF and grant $$ expended. And given that big labs (defined in Berg's post only on the basis of NIH R01/P01 awards are going to have a boatload of fellowships, endowed chairs, philanthropy, smaller foundation awards and HHMI, even these data underrepresent the relationship.

    gww, indeed. It is totally BS that everyone "knows" the limits of the IF and then goes right on to use it "well, it is all we have so....". I am not so fussed about idiots that imagine that the emperor's fine clothes fit them too as I am about actual scientists getting hosed by these field-discriminatory University-wide policies that are based on nonsense.

    Now, with respect to my psych friends, no sympathies. Yes, you can do "neuroscience"..y0u've just abandoned it (back in the 70s) and refused to get with the Insel/NIMH times. Just from my narrow little perspective alone, you have a HUGE natural experiment going on with all those drug-using undergrads that you have pressing RT buttons. Data that you accept as variance is throwing the pharmacological baby out with the ivory tower bathwater.

  • @DM -- we abandoned neuroscience in the 70s? I'm not sure what you're referring to. At least half the people in my department do MRI work (I think it's probably a fair claim that nearly *all* "cognitive neuroscientists" are in psych departments). MRI and other neuro methods simply aren't yet sophisticated enough to be applied to most of the questions I work on.

    Using the natural experiment that is undergraduate life is an interesting idea, though I'm not sure how it's going to constrain, say, theories of verb argument structure or pronoun resolution. But if might be useful for some researchers. Though you're assuming there are no relevant preexisting differences between populations that do or do not choose particular recreational drugs -- not an especially likely assumption.

  • And I wasn't complaining about funding. I've written entire papers without spending *any* money on supplies or materials. A typical study (not counting occasional purchase of computers and staff salaries) that I'd do runs about $100-$200. So $500,000 in directs would go a long, long way.

  • drugmonkey says:

    I’m not sure how it’s going to constrain, say, theories of verb argument structure or pronoun resolution.

    This is my EXACT problem with you people. You aren't curious if dope smoking alters speech generation or perception? Have you spoken to a pothead? How about caffeine and nicotine...does being abstinent from the usual dose alter speech generation or perception? How about those people who have ADHD med prescriptions?

    C'mon now Psych Departments, this is not difficult to dream up....and these factors are already contaminating your studies since if you are anything like the people I know, you don't so much as ask about your college sophomore subjects' drug taking.

  • Namnezia says:

    Even in neuroscience there are no journals with IFs over 20. Usually only Nature and Science and Cell have such high impact factors. Other, what I would consider top-tier, neuroscience journals have IFs of around 14. Review journals don't really count.

    On another note, it would be interesting to see correlation between number of people funded and number of publications. I bet it's not linear either.

  • drugmonkey says:

    correlation between number of people funded and number of publications

    huh?

  • Namnezia says:

    Sorry - I meant the number of people in each lab vs the number of publications.

  • Dude, potheads talk totally normally.

  • Dario Ringach says:

    @DM

    Homework:

    Find the optimal funding level that maximizes:

    (#papers * average IF where they are published) / (funding $)

    Due on Oct 1st at noon PST.

  • @DM I'll turn it back to you. Let's say dope smoking does affect language comprehension. So what? Being clubbed with a jackhammer will, too. And if you are right next to the speakers during a Megadeath concert, that'll affect your ability to discriminate phonemes.

    Life is short, and experimental ideas are many. I have a couple reservations about yours. One is that, as posed, it's primarily descriptive. Will the results tell us something about the way the world is? Sure. But that's not really the business we're in. We're trying to *explain* the world and to build predictive models of behavior.

    Your next move is probably to point out that The Great Model in the Sky would have predictions about stoners' language. Sure. But that doesn't mean we're in the position to make or test those predictions. Language is super, super complicated, and getting meaningful data out of well-designed studies is hard enough. The behavior is (literally) infinitely complex, so you can't just test "everything." Throwing shit at the wall and seeing what sticks isn't a recommended strategy.

  • drugmonkey says:

    Have you heard of endocannabinoids? For that matter have you heard of neurotransmission?

  • [...] This post was mentioned on Twitter by Drug Monkey and Drug Monkey, ScientopiaBlogs. ScientopiaBlogs said: I love it when the data support my position http://dlvr.it/62gf6 [...]

  • bob says:

    What I find interesting about this plot is the big increase in the range of 'productivity' with funding. If the spread is due mainly to lab-to-lab differences and not year-to-year variation for each lab, let the groups that maintain a high productivity/dollar keep their multiple grants and make the others scale back.

    You seem to be saying that the levelling-off supports the idea that concentration of resources is inefficient. That may be true, but I think the broader range suggests that a smarter selection of grantees could be more effective than a (soft) cap of 750k in maximising efficiency.

  • @DM -- Heard of it. What of it? Sounds like shit being thrown against the wall.

    You come across a lot of studies like this: studies of how Hungarian-Aztec bilingual children learn scalar implicature. You ask the researcher, "Is there any reason Hungarian-Aztec bilingual children should learn scalar implicature differently?" And the researcher says, "Well, Hungarian-Aztec bilingualism is weird. Do you have any fuckin' idea how rare it is?"

    Usually it turns out that Hungarian-Aztec bilinguals learn scalar implicature same as everyone else. Good thing, since the researcher wouldn't know how to interpret any differences.

    I guess when I do research, I'm usually testing a hypothesis. And "pot-smoking Hungarian-Aztec bilingual children are weird" isn't really much of a hypothesis.

    I did once want to do a study of morphological inflection by drunk people, but there I had an actual hypothesis: given the effect of alcohol on the motor system, and given theories that link skilled motor behavior to rule-based linguistic processing, you might predict alcohol should impair rule-based linguistic processing while leaving other processes (relatively) spared. Never ran the study though, due to time constraints.

  • drugmonkey says:

    Dude, the fact that I am not writing out made-for-R01 hypotheses in this discussion doesn't mean I couldn't write them if I chose. My point is only to point out that you could be a "neuroscientist" in a heartbeat if you wanted to. No skin off my behind if your field wants to continue on with grand handwaving that tells us little about how the brain actually works.

  • Every psych study is a study of how the brain works. Unless you don't think the brain is what gives rise to behavior.

    Even if you define neuroscience methodologically -- as you seem to be doing -- I still dont' know where you get the idea that psychologists don't do neuroscience. Take a look at the faculty at any major psych department and try to count up how many *don't* use neuroscience methods. Even with my own deep skepticism about the (near-term) usefulness of brain data, I've been known to run EEG studies.

  • drugmonkey says:

    I still dont’ know where you get the idea that psychologists don’t do neuroscience.

    I got the idea from some comment I read on a blog somewhere that said

    And if you don’t do fMRI or do patient work, the neuroscience journals aren’t an option.

  • I meant for a specific project. If I ever actually finish this interminable EEG project, I'll probably submit it to Journal of Cognitive Neuroscience or some place like that. But there are a lot of behaviors one might wish to study for which the neuroscience-y methods simply aren't sophisticated enough (yet). Plus, if you don't have a good understanding of the behavior, it's not even clear how you'd study its implementation in neural structure.

  • drugmonkey says:

    Plus, if you don’t have a good understanding of the behavior, it’s not even clear how you’d study its implementation in neural structure.

    Quoth the die-hard black box experimental psychologist....

    There is absolutely no reason whatsoever that you cannot pursue "a good understanding of the behavior" in combination with testing hypotheses about the neuroanatomy, neuropharmacology, neurophysiology.....or neuroconnectomeELEVEN. As it happens I have reviewed quite a number of grants which propose to do exactly this in my stints on NIH study section. There are also quite a few people publishing papers in what I think of as my most relevant subfields that do exactly this.

    But there are a lot of behaviors one might wish to study for which the neuroscience-y methods simply aren’t sophisticated enough (yet).

    If the methods are sophisticated enough, build them. But even if that isn't an option, what part of incremental advance of science is unclear to you? Do studies that move closer to being able to study behaviors in a sufficiently "neuroscience-y" way. Create bits of the puzzle, even if negative (see your AustroFrench linguistic example). Contribute. Why keep doing your narrow little head-of-pin experiments whilst sneering about those mere experimentalists who are building knowledge? What exactly is it about the Grande Theorye of Everything that is going to be nailed down by a bunch of navel inspecting BunnyHoppers all working within a carefully constructed mutually-agreeable workspace?

  • Physician Scientist says:

    you guys are so off-topic on this. Dr. Berg is good enough to provide data and all you have is a lover's quarrel about psychology v. neuroscience. All I know about psychology is that it was one of the easiest intro classes my freshman year in college.

  • And you're a typical black-box neuroscientist who hasn't yet realized that it all reduces to quantum mechanics.

    No, seriously -- I have a passing curiosity in how the mental computations I study are implemented in the brain in the same way I sometimes think I should probably brush up my knowledge of General Relativity (took part of a class on it in college but the professor was boring and I decided to do something else instead). What I really care about is predicting behavior: writing out a formal theory that will predict, given a particular situation and a particular person, what that person will do.

    Will understanding the underlying neural implementation help? Maybe. Some. But in the way that understanding quantum mechanics helps inform chemistry. Most of the time the chemist can get by blissfully ignorant of all those virtual particles popping in and out of existence. And if you're goal is to understand chemistry better, you're probably better off doing more chemistry experiments rather than building yourself a backyard particle accelerator.

    Why there are a lot of psychologists nonetheless using such methods is an interesting question but takes a lot more words. I mention in passing only that a lot of the people I talk to feel that interest in such methods is starting to wane. It's not clear if that's true, but the pattern of junior hires at, say, Stanford, MIT & Harvard in the last few years seems to bear this out.

  • By coincidence, I had a post on a related topic scheduled to run today.

  • I really like your writing style, great info, appreciate it for putting up :D.

Leave a Reply