Replication costs money

(by drugmonkey) Aug 12 2014

I ran across a curious finding in a very Glamourous publication. Being that it was in a CNS journal, the behavior sucked. The data failed to back up the central claim about that behavior*. Which was kind of central to the actual scientific advance of the entire work.

So I contemplated an initial, very limited check on the behavior. A replication of the converging sort.

It's going to cost me about $15K to do it.

If it turns out negative, then where am I? Where am I going to publish a one figure tut-tut negative that flies in the face of a result published in CNS?

If it turns out positive, this is almost worse. It's a "yeah we already knew that from this CNS paper, dumbass" rejection waiting to happen.

Either way, if I expect to be able to publish in even a dump journal I'm gong to need to throw some more money at the topic. I'd say at least $50K.

At least.

Spent from grants that are not really related to this topic in any direct way.

If the NIH is serious about the alleged replication problem then it needs to be serious about the costs and risks involved.
*a typical problem with CNS pubs that involve behavioral studies.

34 responses so far

Women in the R00 phase don't apply for R01s as frequently as men

(by drugmonkey) Aug 11 2014

Sally Rockey:

A specific issue that recently has recently created interesting conversations in the blogosphere is whether female K99/R00 awardees were less likely to receive a subsequent R01 award compared to male K99/R00 awardees. We at NIH have also found this particular outcome among K99/R00 PIs and have noted that those differences again stem from differential rates of application. Of the 2007 cohort of K99 PIs, 86 percent of the men had applied for R01s by 2013, but only 69 percent of the women had applied.

She's referring here to a post over at DataHound ("K99-R00 Evaluation: A Striking Gender Disparity") which observed:

Of the 201 men with R00 awards, 114 (57%) have gone on to receive at least 1 R01 award to date. In contrast, of the 127 women with R00 awards, only 53 (42%) have received an R01 award. This difference is jarring and is statistically significant (P value=0.009).
To investigate this further, I looked at the two cohorts separately. For the FY2007 cohort, 70 of the 108 men (65%) with R00 awards have received R01 grants whereas only 31 of the 62 women (50%) have (P value = 0.07). For the FY2008 cohort, 44 of the 93 men (47%) with R00 awards have received R01s whereas only 22 of the 65 women (34%) have (P value = 0.10). The lack of statistical significance is due to the smaller sample sizes for the cohorts separately rather than any difference in the trends for the separate cohorts, which are quite similar.

And Rockey isn't even giving us the data on the vigor with which a R00 holder is seeking R01 funding. That may or may not make the explanation even stronger.

Seems to me that any mid or senior level investigators who have new R00-holding female assistant professors in their department might want to make a special effort to encourage them to submit R01 apps early and often.

13 responses so far

On making progress

(by drugmonkey) Aug 11 2014

90% of the progress on my manuscripts and grants takes place during 20% of the time I am ostensibly working on them.

7 responses so far

Simple, can't-miss strategy to get a grant from the NIH

(by drugmonkey) Aug 08 2014

Work at it.
Continue Reading »

79 responses so far

Ebola and ZMapp...A scientist explains

(by drugmonkey) Aug 07 2014

Erica Ollman Saphire (lab website, PubMed, RePORTER) was interviewed on KPBS in San Diego about the use of highly experimental antibody therapy for the US health workers infected with Ebola virus.

It's a pretty interesting viewpoint on basic science, translation to humans and what we do when an emergency situation like an infectious disease outbreak happens. I have been struck in past days about the huge international discussion this ZMapp treatment has been sparking. As you might expect, we have dark thoughts being expressed along the lines of "Why does this apparently miraculous treatment emerge all of a sudden when Americans are infected but it hasn't been given to suffering Africans, hmmmm?". There are all kinds of ethical issues to think about.

The television version linked below is 5 minutes but be sure to click on the link to the "midday edition" which is a longer voice interview. It gives a much fuller discussion.

Additional Reading:

CDC: Questions and Answers on experimental treatments and vaccines for Ebola

Experimental Ebola drug based on research from Canada’s national lab

Ebola experimental drug, ZMapp sparks ethical controversy

David Kroll on ZMapp

David Kroll on two other Ebola therapies

26 responses so far

There is no "filter problem" in science

(by drugmonkey) Aug 07 2014


It is your job as a scientist to read the literature, keep abreast of findings of interest and integrate this knowledge with your own work.

We have amazing tools for doing so that were not available in times past, everything gets fantastically better all the time.

If you are a PI you even have minions to help you! And colleagues! And manuscripts and grants to review which catch you up.

So I ask you, people who spout off about the "filter" problem.....

What IS the nature of this problem? How does it affect your working day?

Since most of you deploy this in the context of wanting fewer papers to be published in fewer is that better? What is supposed to disappear from your view?

The stuff that you happen not to be interested in?

32 responses so far

It isn't the fraud witchhunt, it's the Glamour culture of science

(by drugmonkey) Aug 05 2014

The Sesai suicide has been deemed the result of an anti-fraud witch hunt by well respected biomedical ethics / conduct of science / publishing / open science dude Michael Eisen.

I disagree that this is the proper frame for what happened.

First, while I am no fan of the sort of lynch mob behavior that tends to emerge in the comments at the retractionwatch blog these days, scientific fraud still needs to be rooted out and exposed wherever it occurs. Blaming a suicide on the "witch hunt", as if rooting out fraud is not a valid or significant concern, is a problem to me. We already have enough enabling behavior in the Academy. Enough excusing, enough looking the other way and enough failing to convict a pattern of behavior because we can't lay beyond-reasonable-doubt gloves on the perpetrator. Dismissing all vigorous attempts to get to the bottom of a paper fraud situation as if they are baseless (i.e., witches do not exist) is counterproductive.

Second, data fabrication and fraud has victims. And all too often we frame scientific fraud ONLY through the lens of scientific understanding. Which, let us be honest, is fairly robust against claims that turn out to be wrong. Sure, time and money are lost, but the scientific understanding wins out in the end. Peoples' careers, however, often suffer irretrievable harm. When a job is won by a data faker like Marc Hauser or Michael Miller then someone else lost that job. When a research grant is awarded based on faked publications or preliminary data, another investigator doesn't get those funds [even the grants themselves are rarely pulled from the University, a new PI is frequently substituted]. These are serious harms, there are victims and turning a blind eye to scientific misconduct ensures more harms in the future.

Third, this was a Japanese investigator who decided to take ultimate responsibility by killing himself. I've been around a few decades and have noticed that middle and top level managers in Japan occasionally commit suicide over work-related matters that are inexplicably strange and unjustified to most Western (and certainly USian) eyes. It strikes me that there are cultural factors at play here that explain this event far more truthily than some analysis of the effects of a "witch hunt" about data fraud.

Nevertheless, if you absolutely insist that there is some thing about the current culture of science that resulted in this suicide of a research scientist, rxnm has some thoughts which seem much more related to me.

And what about everyone else? Journals, colleagues, scientists, journalists? Do we really need hero narratives? The splashy results that will “change everything”? The hype machine it is out of fucking control. We are adopting the language of biz-speak bullshit and starting to buy into these empty non-values about techno-utopian revolutionaries and lone geniuses. We all participate in the culture of valuing glam, prestige, prizes. Who gets the 8-figure grants while everyone else struggles to stay afloat? Who can I get a selfie with at SfN? Who gets to stamp their name all over the culmination of decades of work by hundreds or thousands? We’ve become cultish: around people, journals, technologies, institutions. As if these are things that matter more than the colleagues around us, or our own integrity. It’s pathetic, and we can be better.

Without the need for Glamourous results, there is less need to fake data. Without the hero and lone-genius narrative, PIs would feel less desire to appear always-correct and fear the overturning of their pet story or hypothesis much less. Without this intensely competitive fight to publish in the right limited subset of journals.... etc.

ps. Graduate students suicide occasionally too. Guess which culture change would have the greater effect- anti-fraud alleged witch hunts or dismantling the hypercompetitive, Glamour-humping prestige-seeking?

40 responses so far

Thought of the Day

(by drugmonkey) Jul 24 2014

What fraction of the stuff proposed in funded grants actually gets done after feasibility and field movement come to play?

22 responses so far

A can't-miss inquiry to Editor following the initial review of your paper

(by drugmonkey) Jul 23 2014

Dear Editor Whitehare,

Do you really expect us to complete the additional experiments that Reviewer #3 insisted were necessary? You DO realize that if we did those experiments the paper would be upgraded enough that we sure as hell would be submitting it upstream of your raggedy ass publication, right?

The Authors

22 responses so far

Sex differences in K99/R00 awardees from my favorite ICs

(by drugmonkey) Jul 21 2014

Datahound has some very interesting analyses up regarding NIH-wide sex differences in the success of the K99/R00 program.

Of the 218 men with K99 awards, 201 (or 92%) went on to activate the R00 portion. Of the 142 women, 127 (or 89%) went on to these R00 phase. These differences in these percentages are not statistically different.

Of the 201 men with R00 awards, 114 (57%) have gone on to receive at least 1 R01 award to date. In contrast, of the 127 women with R00 awards, only 53 (42%) have received an R01 award. This difference is jarring and is statistically significant (P value=0.009).


So per my usual, I'm very interested in what the ICs that are closest to my lab's heart have been up to with this program. Looking at K99 awardees from 07 to 09 I find women PIs to constitute 3/3, 1/3 and 2/4 in one Institute and 1/7, 2/6 and 5/14 in the other Institute. One of these is doing better than the other and I will just note that was before the arrival of a Director who has been very vocal about sex discrimination in science and academia.

In terms of the conversion to R01 funding that is the subject of Datahound's post, the smaller Institution has decent outcomes* for K99 awardees from 07 (R01, R21, nothing), 08 (R01, R01, R01) and 09 (P20 component, U component, nothing, nothing).

In the other Institute, the single woman from 07 did not appear to convert to the R00 phase but Google suggests made Assistant Professor rank anyway. No additional NIH funding. The rest of the 07 class contains 4 with R01 and two with nothing. In 08, the women PIs are split (one R01, one nothing) similar to the men (2 R01, 2 with nothing). In 09 the women PIs have two with R01s, one R03 and two with nothing.

So from this qualitative look, nothing is out of step with Datahound's NIH-wide stats. There are 14/37 women PIs, this 38% is similar to the NIH-wide 39% Datahound quoted although there may be a difference between these two ICS (30% vs 60%) that could stand some inquiry. One of 37 K99 awardees failed to convert to R00 from the K99 (but seems to be faculty anyway). Grant conversion past the R00 is looking to be roughly half or a bit better.

I didn't do the men for the 2009 cohort in the larger Institute but otherwise the sex differences in terms of getting/not getting additional funding beyond the R00 seems pretty similar.

I do hope Datahound's stats open some eyes at the NIH, however. Sure, there are reasons to potentially excuse away a sex difference in the rates of landing additional research funding past the R00. But I am reminded of a graph Sally Rockey posted regarding the success rate on R01-equivalent awards. It showed that men and women PIs had nearly identical success rates on new (Type 1) proposals but slightly lower success on Renewal (Type 2) applications. This pastes over the rates of conversion to R00 and the acquisition of additional funding, if you squint a bit.

Are women overall less productive once they've landed some initial funding? Are they viewed negatively on the continuation of a project but not on the initiation of it? Are women too humble about what they have accomplished?
*I'm counting components of P or U mechanisms but not pilot awards.

15 responses so far

« Newer posts Older posts »