Archive for the 'Conduct of Science' category

Replication costs money

I ran across a curious finding in a very Glamourous publication. Being that it was in a CNS journal, the behavior sucked. The data failed to back up the central claim about that behavior*. Which was kind of central to the actual scientific advance of the entire work.

So I contemplated an initial, very limited check on the behavior. A replication of the converging sort.

It's going to cost me about $15K to do it.

If it turns out negative, then where am I? Where am I going to publish a one figure tut-tut negative that flies in the face of a result published in CNS?

If it turns out positive, this is almost worse. It's a "yeah we already knew that from this CNS paper, dumbass" rejection waiting to happen.

Either way, if I expect to be able to publish in even a dump journal I'm gong to need to throw some more money at the topic. I'd say at least $50K.

At least.

Spent from grants that are not really related to this topic in any direct way.

If the NIH is serious about the alleged replication problem then it needs to be serious about the costs and risks involved.
__
*a typical problem with CNS pubs that involve behavioral studies.

34 responses so far

Ebola and ZMapp...A scientist explains

Aug 07 2014 Published by under Conduct of Science, Public Health

Erica Ollman Saphire (lab website, PubMed, RePORTER) was interviewed on KPBS in San Diego about the use of highly experimental antibody therapy for the US health workers infected with Ebola virus.

It's a pretty interesting viewpoint on basic science, translation to humans and what we do when an emergency situation like an infectious disease outbreak happens. I have been struck in past days about the huge international discussion this ZMapp treatment has been sparking. As you might expect, we have dark thoughts being expressed along the lines of "Why does this apparently miraculous treatment emerge all of a sudden when Americans are infected but it hasn't been given to suffering Africans, hmmmm?". There are all kinds of ethical issues to think about.

The television version linked below is 5 minutes but be sure to click on the link to the "midday edition" which is a longer voice interview. It gives a much fuller discussion.

__
Additional Reading:

CDC: Questions and Answers on experimental treatments and vaccines for Ebola

Experimental Ebola drug based on research from Canada’s national lab

Ebola experimental drug, ZMapp sparks ethical controversy

UPDATE:
David Kroll on ZMapp

David Kroll on two other Ebola therapies

26 responses so far

There is no "filter problem" in science

Seriously.

It is your job as a scientist to read the literature, keep abreast of findings of interest and integrate this knowledge with your own work.

We have amazing tools for doing so that were not available in times past, everything gets fantastically better all the time.

If you are a PI you even have minions to help you! And colleagues! And manuscripts and grants to review which catch you up.

So I ask you, people who spout off about the "filter" problem.....

What IS the nature of this problem? How does it affect your working day?

Since most of you deploy this in the context of wanting fewer papers to be published in fewer journals...how is that better? What is supposed to disappear from your view?

The stuff that you happen not to be interested in?

32 responses so far

A can't-miss inquiry to Editor following the initial review of your paper

Jul 23 2014 Published by under Careerism, Conduct of Science, Peer Review

Dear Editor Whitehare,

Do you really expect us to complete the additional experiments that Reviewer #3 insisted were necessary? You DO realize that if we did those experiments the paper would be upgraded enough that we sure as hell would be submitting it upstream of your raggedy ass publication, right?

Collegially,
The Authors

22 responses so far

The most replicated finding in drug abuse science

Ok, ok, I have no actual data on this. But if I had to pick one thing in substance abuse science that has been most replicated it is this.

If you surgically implant a group of rats with intravenous catheters, hook them up to a pump which can deliver small infusions of saline adulterated with cocaine HCl and make these infusions contingent upon the rat pressing a lever...

Rats will intravenously self-administer (IVSA) cocaine.

This has been replicated ad nauseum.

If you want to pass a fairly low bar to demonstrate you can do a behavioral study with accepted relevance to drug abuse, you conduct a cocaine IVSA study [Wikipedia] in rats. Period.

And yet. There are sooooo many ways to screw it up and fail to replicate the expected finding.

Note that I say "expected finding" because we must include significant quantitative changes along with the qualitative ones.

Off the top of my head, the types of factors that can reduce your "effect" to a null effect, change the outcome to the extent even a statistically significant result isn't really the effect you are looking for, etc

  • Catheter diameter or length
  • Cocaine dose available in each infusion
  • Rate of infusion/concentration of drug
  • Sex of the rats
  • Age of rats
  • Strain of the rats
  • Vendor source (of the same nominal strain)
  • Time of day in which rats are run (not just light/dark* either)
  • Food restriction status
  • Time of last food availability
  • Pair vs single housing
  • "Enrichment" that is called-for in default guidelines for laboratory animal care and needs special exception under protocol to prevent.
  • Experimenter choice of smelly personal care products
  • Dirty/clean labcoat (I kid you not)
  • Handling of the rats on arrival from vendor
  • Fire-alarm
  • Cage-change day
  • Minor rat illness
  • Location of operant box in the room (floor vs ceiling, near door or away)
  • Ambient temperature of vivarium or test room
  • Schedule- weekends off? seven days a week?
  • Schedule- 1 hr? 2hr? 6 hr? access sessions
  • Schedule- are reinforcer deliveries contingent upon one lever press? five? does the requirement progressively increase with each successive infusion?
  • Animal loss from the study for various reasons

As you might expect, these factors interact with each other in the real world of conducting science. Some factors you can eliminate, some you have to work around and some you just have to accept as contributions to variability. Your choices depend, in many ways, on your scientific goals beyond merely establishing the IVSA of cocaine.

Up to this point I'm in seeming agreement with that anti-replication yahoo, am I not? Jason Mitchell definitely agrees with me that there are a multitude of ways to come up with a null result.

I am not agreeing with his larger point. In fact, quite the contrary.

The point I am making is that we only know this stuff because of attempts to replicate! Many of these attempts were null and/or might be viewed as a failure to replicate some study that existed prior to the discovery that Factor X was actually pretty important.

Replication attempts taught the field more about the model, which allowed investigators of diverse interests to learn more about cocaine abuse and, indeed, drug abuse generally.

The heavy lifting in discovering the variables and outcomes related to rat IVSA of cocaine took place long before I entered graduate school. Consequently, I really can't speak to whether investigators felt that their integrity was impugned when another study seemed to question their own work. I can't speak to how many "failure to replicate" studies were discussed at conferences and less formal interactions. But given what I do know about science, I am confident that there was a little bit of everything. Probably some accusations of faking data popped up now and again. Some investigators no doubt were considered generally incompetent and others were revered (sometimes unjustifiably). No doubt. Some failures to replicate were based on ignorance or incompetence...and some were valid findings which altered the way the field looked upon prior results.

Ultimately the result was a good one. The rat IVSA model of cocaine use has proved useful to understand the neurobiology of addiction.

The incremental, halting, back and forth methodological steps along the path of scientific exploration were necessary for lasting advance. Such processes continue to be necessary in many, many other aspects of science.

Replication is not an insult. It is not worthless or a-scientific.

Replication is the very lifeblood of science.

__
*rats are nocturnal. check out how many studies**, including behavioral ones, are run in the light cycle of the animal.

**yes to this very day, although they are certainly less common now

22 responses so far

Being as wrong as can be on the so-called replication crisis of science

Jul 07 2014 Published by under Conduct of Science

I am no fan of the hysterical hand wringing about some alleged "crisis" of science whereby the small minded and Glam-blinded insist that most science is not replicable.

Oh, don't get me wrong. I think replication of a prior result is the only way we really know what is most likely to be what. I am a huge fan of the incremental advance of knowledge built on prior work.

The thing is, I believe that this occurs down in the trenches where real science is conducted.

Most of the specific complaining that I hear about failures to replicate studies is focused on 1) Pharma companies trying to cherry pick intellectual property off the latest Science, Nature or Cell paper and 2) experimental psychology stuff that is super truthy.

With regard to the former, cry me a river. Publication in the highest echelons of journals, publication of a "first" discovery/demonstration of some phenomenon is, by design, very likely not easily replicated. It is likely to be a false alarm (and therefore wrong) and it is likely to be much less generalizable than hoped (and therefore not "wrong" but definitely not of use to Pharma vultures). I am not bothered by Pharma execs who wish that public funded labs would do more advancing of intellectual property and serve it up to them part way down the traditional company pipeline. Screw them.

Psych studies. Aaah, yes. They have a strong tradition of replication to rely upon. Perhaps they have fallen by the wayside in recent decades? Become seduced to the dark side? No matter. Let us return to our past, eh? Where papers in the most revered Experimental Psychology journals required several replications within a single paper. Each "Experiment" constituting a minor tweak on the other ones. Each paper firmly grounded in the extant literature with no excuses for shitty scholarship and ignoring inconvenient papers. If there is a problem in Psych, there is no excuse because they have an older tradition. Or possibly some of the lesser Experimental Psychology sects (like Cognitive and Social) need to talk to the Second Sect (aka Behaviorism).

In either of these situations, we must admit that replication is hard. It may take some work. It may take some experimental tweaking. Heck, you might spend years trying to figure out what is replicable / generalizable, what relies upon very ....specific experimental conditions and what is likely to have been a false alarm. And let us admit that in the competitive arena of academic science, we are often more motivated by productivity than we are solving some esoteric problem that is nagging the back of our minds. So we give up.

So yeah, sometimes practicalities (like grant money. You didn't seriously think I'd write a post without mentioning that, did you?) prevent a thorough run at a replication. One try simply isn't enough. And that is not a GoodThing, even if it is current reality. I get this.

But....

Some guy has written a screed against the replication fervor that is actually against replication itself. It is breathtaking.

All you need to hook your attention is conveniently placed as a bullet point pre-amble:

· Recent hand-wringing over failed replications in social psychology is largely pointless, because unsuccessful experiments have no meaningful scientific value.
· Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way. Unless direct replications are conducted by flawless experimenters, nothing interesting can be learned from them.
· Three standard rejoinders to this critique are considered and rejected. Despite claims to the contrary, failed replications do not provide meaningful information if they closely follow original methodology; they do not necessarily identify effects that may be too small or flimsy to be worth studying; and they cannot contribute to a cumulative understanding of scientific phenomena.
· Replication efforts appear to reflect strong prior expectations that published findings are not reliable, and as such, do not constitute scientific output.

· The field of social psychology can be improved, but not by the publication of negative findings. Experimenters should be encouraged to restrict their "degrees of freedom," for example, by specifying designs in advance.

· Whether they mean to or not, authors and editors of failed replications are publicly impugning the scientific integrity of their colleagues. Targets of failed replications are justifiably upset, particularly given the inadequate basis for replicators’ extraordinary claims.

Seriously, go read this dog.

This part seals it for me.

So we should take note when the targets of replication efforts complain about how they are being treated. These are people who have thrived in a profession that alternates between quiet rejection and blistering criticism, and who have held up admirably under the weight of earlier scientific challenges. They are not crybabies. What they are is justifiably upset at having their integrity questioned.

This is just so dang wrong. Trying to replicate another paper's effects is a compliment! Failing to do so is not an attack on the authors' "integrity". It is how science advances. And, I dunno, maybe this guy is revealing something about how he thinks about other scientists? If so, it is totally foreign to me. I left behind the stupid game of who is "brilliant" and who is "stupid" long ago. You know, when I was leaving my adolescent arrogance (of which I had plenty) behind. Particularly in the experimental sciences, what matters is designing good studies, generating data, interpreting data and communicating that finding as best one can. One will stumble during this process...if it were easy it wouldn't be science. We are wrong on a near-weekly basis. Given this day to day reality, we're going to be spectacularly wrong on the scale of an entire paper every once in awhile.

This is no knock on someone's "integrity".

Trying to prevent* anyone from replicating your work, however, IS a knock on integrity.

On the scientific integrity of that person who does not wish anyone to try to replicate his or her work, that is.

__
*whether this be by blocking publication via reviewer or editorial power/influence, torpedoing a grant proposal, interfering with hiring and promotion or by squelching intrepid grad students and postdoctoral trainees in your own lab who can't replicate "The Effect".

26 responses so far

Clarity

Jun 26 2014 Published by under Conduct of Science, Mentoring, Tribe of Science

This sentence gave me cold chills.

Younger scientists need protection from the ambitions of their elders.

22 responses so far

How do you unpoison a sub-field?

Jun 12 2014 Published by under Conduct of Science, Tribe of Science

We all know the signs.

A small group of successful, egotistical scientists who view themselves as being in mortal combat with each other.

A few trainees or hangers-on who have signed on under each warring camp.

A scientific topic of mutual interest.

But then there are the good people. The ones who want to advance the topic and care much less about who gets Ultimate Credit. The ones who think the field should, would and could be much farther along if the Big Folks would just stop their petty infighting and paranoid ravings.

We need the Big Cheeses...they do decent work and they have the ear of Program. They have the reputational chops to pull off ambitious projects. If only they would do so.

So side stepping them altogether is not an option.

But we cannot continue letting them clog things up with their egos and competitions.

So far, all I have is a vague idea of an alliance of the like minded, lower-down decent people in the subfield. But I don't know how to make this work yet.

How do you draw the poison?

27 responses so far

Question of the Day

Are your papers reporting "discoveries"?

Or "demonstrating" something?

Discuss.

37 responses so far

Revision strategy, with an eye to the new A2 as A0 policy at NIH

Occasionally, Dear Reader, one or another of you solicits my specific advice on some NIH grant situation you are experiencing. Sometimes the issues are too specific to be of much general good but this one is at least grist for discussion of how to proceed.

Today's discussion starts with the criterion scores for an R01/equivalent proposal. As a reminder, the five criteria are ordered as Significance, Investigator, Innovation, Approach and Environment. The first round for this proposal ended up with

Reviewer #1: 1, 1, 1, 3, 1
Reviewer #2: 3, 1, 1, 3, 1
Reviewer #3: 6, 2, 1, 8, 1
Reviewer #4: 2, 1, 3, 2, 1

From this, the overall outcome was.... Not Discussed. Aka, triaged.

As you might imagine, the PI was fuming. To put it mildly. Three pretty decent looking reviews and one really, really unfavorable one. This should, in my opinion, have been pulled up for discussion to resolve the differences of opinion. It was not. That indicates that the three favorable reviewers were either somehow convinced by what Reviewer #3 wrote that they had been too lenient...or they were simply not convinced discussion would make a material difference (i.e. push it over the "fund" line). The two 3s on Approach from the first two reviewers are basically a "I'd like to see this come back, fixed" type of position. So they might have decided, screw it, let this one come back and we'll fight over it then.

This right here points to my problem with the endless queue of the revision traffic pattern and the new A2 as A0 policy that will restore it to the former glory. It should be almost obligatory to discuss significantly divergent scores, particularly when they make a categorical difference. The difference between triaged and discussed and the difference between a maybe-fundable and a clearly-not-fundable score is known to the Chair and the SRO of the study section. Thee Chair could insist on resolving these types of situations. I think they should be obliged to do so, personally. It would save some hassle and extra rounds of re-review. It seems particularly called-for when the majority of the scores are in the better direction because that should be some minor indication that the revised version would have a good chance to improve in the minds of the reviewers.

There is one interesting instructive point that reinforces one of my usual soapboxes. This PI had actually asked me before the review, when the study section roster was posted, what to do about reviewer conflicts. This person was absolutely incensed (and depressed) about the fact that a scientific competitor in highly direct competition with the proposal had been brought on board. There is very little you can do, btw, 30 days out from review. That ship has sailed.

After seeing the summary statement, the PI had to admit that going by the actual criticism comments, the only person with the directly-competing expertise was not Reviewer #3. Since the other three scores were actually pretty good, we can see that I am right on the assumption of what a reviewer will think of your application based on perceptions of competition or personal dis/like. You will often be surprised that the reviewer that you assume is out to screw your application over will be pulling for it. Or at least, will be giving it a score that is in line with the majority of the other reviewers. This appears to be what happened in this case.

Okay. So, as I may have mentioned I have been reluctantly persuading myself that revising triaged applications is a waste of time. Too few of them make it over the line to fund. And in the recently past era of A1 and out....well perhaps time was better spent on a new app. In this case, however, I think there is a strong case for revision. Three of four (and we need to wonder about why there even were four reviews instead of three) of these criterion score sets look to me like scores that would get an app discussed. The ND seems to be a bit of an unfair result, based on the one hater. The PI agreed, apparently, and resubmitted a revised application. In this case the criterion scores were:

Reviewer #1: 1, 2, 2, 5, 1
Reviewer #2: 2, 2, 2, 2, 1
Reviewer #3: 1, 1, 2, 2, 1
Reviewer #4: 2, 1, 1, 2, 1
Reviewer #5: 1, 1, 4, 7, 1

I remind you that we cannot assume any overlap in reviewers nor any identity of reviewer number in the case of re-assigned reviewers. In this case the grant was discussed at study section and ended up with a 26 voted impact score. The PI noted that a second direct competitor on the science had been included on the review panel this time in addition to the aforementioned first person in direct competition.

Oh Brother.

I assure you, Dear Reader, that I understand the pain of getting reviews like this. Three reviewers throwing 1s and 2s is not only a "surely discussed" outcome but is a "probably funded" zone, especially for a revised application. Even the one "5" from Reviewer #1 on Approach is something that perhaps the other reviewers might talk him/her down from. But to have two obviously triage numbers thrown on Approach? A maddening split decision, leading to a score that is most decidedly on the bubble for funding.

My seat of the pants estimation is that this may require Program intervention to fund. I don't know for sure, I'm not familiar with the relevant paylines and likely success rates for this IC for this fiscal year.

Now, if this doesn't end up winning funding, I think the PI most certainly has to take advantage of the new A2 as A0 policy and put this sucker right back in. To the same study section. Addressing whatever complaints were associated with Reviewer #1's and #5's criticisms of course. But you have to throw yourself on the mercy of the three "good" reviewers and anyone they happened to convince during discussion. I bet a handful of them will be sufficient to bring the next "A0" of this application to a fundable score even if the two less-favorable reviewers refuse to budge. I also bet there is a decent chance the SRO will see that last reviewer as a significant outlier and not assign the grant to that person again.

I wish this PI my best of luck in getting the award.

29 responses so far

Older posts »