The data are what the data are

Feb 14 2013 Published by under Uncategorized

p < 0.05.

How many of you have seen these 6 little characters and felt The Rush? Admit it--it's better than Christmas morning, your first kiss*, and good bourbon all rolled into one! You want to run through the halls of your Classy Institution screaming, "I HAVE DATAAAAAAAA!!!!" - and you very well might do that.

But here's the thing we often forget: p > 0.05 is data, too. And having more significant differences doesn't necessarily mean your paper's going to a fancier journal. Earlier this evening, I caught whiff of a conversation on twitter, started by Sciencegurl, and it made me a little sad. She tweeted:


And that just about broke my heart. PIs, do you understand that negative data might not be the result of  your trainees being bad at science, but instead perhaps there simply are no differences?

Here in the Laboratory of Neuroscience and Awesomeness, we are getting to the point where we've collected what can only be described as a shit ton of data, and it is my job as PI to help my trainees make sense of it all. Their natural inclination is to hope that experimental and control groups are different in every measure possible, but that is of course not how things pan out, ever. For example, we recently had some interesting behavioral data, and so I asked one of my grad students to process tissue from the brains of the animals from that study (to me, running a behavioral experiment without looking at the brains in some capacity is like roasting a chicken and then throwing out the bones without making a stock. So much potential goodness to squeeze out!) So grad student worked extremely hard to do this --she worked long hours, all through winter break, and well into this semester. She trouble shot, she took gorgeous images, and she handed me a spreadsheet  full of raw data that I analyzed in every way I could think of, from every angle imaginable. But no matter how much I squinted or turned my head sideways, there were simply no significant differences between groups.

Would it have been fun if we had found differences? Of course. Does the fact that this data set happened to turn out negative mean we won't include it when we write it up? Fuck no. Like it or not, these data are part of our narrative, and it's our job as scientists to think hard about what both positive AND negative data mean for the story we're trying to tell. Not everything may fall perfectly into place the way we'd originally imagined it, but half the fun of being a scientist is trying to wrap your brain around the data you have, and coming up with a new interpretation of what's going on. Don't let your data make you sad, and for fuck's sake don't take it out on your trainees--make your data work for you. Because after all, the data are what the data are.

* truth be told, my first kiss was not all that pleasant, because braces.

16 responses so far

  • physioprof says:

    [R]unning a behavioral experiment without looking at the brains in some capacity is like roasting a chicken and then throwing out the bones without making a stock.

    lolz

    PIs who express "disappointment" (or worse, which obviously occurs all too frequently) because the results of an experiment are not as they hoped/expected/fantasized are absolutely begging to get their labs embroiled in scientific misconduct. This is in large part why labs built around proving some grandiose theory or hypothesis are so dysfunctional.

    • Exactly!!! As a grad student, I find myself working AGAINST people more senior than me to maintain scientific integrity (no I will not just ignore that condition because it shows the opposite to what we wanted). I never expected that!

  • igrrrl says:

    First, what physioprof said. Fraud comes of pressure, regardless of the source of the pressure, but it's worse when it comes from the PI in a highly competitive lab in a competitive field.

    And as for negative data, I had an entire year of my life boil down to one not-very-significant figure in my dissertation. (Of course, that was better than the two years that resulted in... no figures in anything.) The trick in training is learning to recognize when the rabbit holes lead nowhere, or when to stop beating your head against a particular wall. Just being 'disappointed' at your trainees doesn't accomplish either of those things.

  • Jim Thomerson says:

    One of the introductory labs I devised involved going out and measuring the diameters of pine trees in two groups by the library. Then the students did a Mann-Whitney-U test to see if the two groups were significantly different. They were not. At the end of the session, I showed them an aerial photograph of both groups being planted at the same time. I had a couple of other labs where differences were significant, so I did this one to be fair and balanced, so to speak.

  • AEI says:

    There's disappointment, and then there's disappointment.

    As a PI, if I thought something was going, and the student comes back to me and shows that the method doesn't work as I'd expect it to—and that there are no other methodological issues to resolve—then I'd be reasonably disappointed that my hunch wasn't correct.

    However, the goal is not to become disappointed in the student, but rather to figure out what can be done to resolve the situation. Finding the root cause is instructive for everyone; assigning blame under such circumstances is not.

  • DrLizzyMoore says:

    Shouldn't we also be teaching our students how to handle it when their hypothesis is wrong?

    Save the disappointment for the lack of following a protocol for the uptenth time or for tossing picric acid down the sink....ya know!?

  • qaz says:

    This is why one needs positive controls to show that the method is working, even when the results show no difference.

  • anthea says:

    Ahhh....this reminds me of when I did a ton of work with a bunch of computing science and I had negative results. I was happy, they were happy, my supervisor was happy but some of the other academics on the team were furious (they weren't computer scientists) freaked out since they didn't the results that they had wanted since I didn't prove that they were right. It was argued that the data was wrong, the results flawed and something was wrong regardless of the fact that the method had worked, the data was ok, the tests had run properly, everything had gone perfectly in fact. ..It was a case where you just hit your head against the wall.

  • Jan says:

    I'm not quite sure about your definition of negative data. Data itself is, as you rightly point out, just as it is.

    If you get inconclusive results and it is not your fault (because your experiment was well planned and executed), then follow-up research is required. This is the basis of scientific inquiry, asking "why is that?" and you can still publish in the Journal of Unsolved Questions (that is totally a thing).

    Just looking at the p-value, no matter what it is, is a mistake. If you get results with p > .05, it may mean that there is no difference in some value for two distinct groups, which might be news in itself. But p-values are nothing more than the deviation from the expectation divided by the standard error. If one does not look at the effect size, much else is meaningless. If you get a large difference between two groups but a large standard error, it may show to be statistically insignificant at 5%, but is really the most important find you make.

    But I do not think you do not know this. My question is which of these result types do you consider to be negative?

    The most negative thing about this situation is definitely the attitude of the PI, who might not be the right person for the job if he or she is only concerned with getting statistically significant results.

  • Ilovepigenetics says:

    As an undergrad, my PI was furious with me because my data didn't come out the same each time. We were doing nuclear run-off assays (yes, I did one of the most complicated assays as an undergrad) and measuring CMYC expression in HL-60 cells (by dot blot--one of the least sensitive measuring methods). This was before we knew anything about epigenetics, before I understood about cell-cycle genes, etc. If I had to do that experiment over again, I would have designed the experiment differently. However, to yell and scream at an undergrad (who did not have much help in experimental design) is not appropriate. I have never forgotten that, and I do my best not to be disappointed when my students hand me their data. The data is what the data is. I can be disappointed in the experimental design, and I share the blame on that, but never in the data themselves. And, I agree that focusing on the data will increase the likelihood of scientific misconduct, etc. If Linus Pauling can be wrong about how antibodies form, I can be wrong about my hypothesis.

  • namnezia says:

    I agree that negative data are data, and data are good. I think the problem is when you can't be sure if your negative data are due to the fact that the experiment was done incorrectly and thus you ended up with an artifact, or whether your experiment had sufficient statistical power to show that a p>0.05 is real. If you get a significant result then you can be fairly secure that in fact you have significant results, but if you have a lot of variability then you might need way more experiments before you can see significance or lack of it. For example in Sciencegurl's tweet, she said her RNA was crap, so a negative result could be real or due to an artifact of bad RNA isolation. So that's the worse situation to be in.

    That being said, it's no reason for her PI to be mad at her and make her feel bad. The times my trainees disappoint me is not when they bring me negative or inconclusive data, but rather when the data are inconclusive due to neglect in being careful or willful lack of productivity and thus failure to obtain sufficient to reach an answer, either positive or negative.

  • Tess says:

    I do agree that "making" someone feel responsible for data not coming out a certain way is not right and creates an environment that is unpleasant at best, and one that can lead to fraud at worst.

    However, not every single time someone shows disappointment needs to be taken personally. Sometimes you have a hunch or even a desire to find an effect and when you don't you humanly show disappointment. It may not be directed disappointment but some other people assume they are the one responsible for the disappointment. If we behaved like robots and never showed disappointment (or any other "weakness") to our trainees when things did not work out, that would be a dishonest representation of our work and a discredit to our selves as human scientists with human emotions.

    I don't know what happened in this case. Perhaps the PI really did act in a way that blamed the student, which is not OK. But we should separate the expression of the disappointment from targeting someone specific with that disappointment.

    And of course, when we encounter something like this, the data are what they are. One will eventually find a way to accept this and take it as it is. Hopefully sooner thna later. But it's not a horrid thing if one has a brief moment of emotion when something you hoped would go a certain way, does not.

  • [...] this doesn’t even rise to the level of ‘math’, but is arithmatic illiteracy) The data are what the data are Bigfoot genome paper “conclusively proves” that Sasquatch is real Let’s make 2013 the year of [...]

  • Dr Becca says:

    Jan, by "negative data" I mean simply when you compare two groups and there is no significant difference between them in the measure you're targeting. As you say, there can be a number of different reasons why this could be the case, and I always like to look at the raw data and see if it's due to large variability within each group, or some sort of inconsistency across cohorts. A couple of times now, this has allowed me to catch irregular data that was due to problems in the animal facility.

  • AJD says:

    p > 0.05 is data too?

    I always tell my students "p > 0.05 means we *don't necessarily have enough data* to draw a conclusion from"—p > 0.05 alone is not a "negative result" (i.e., showing that our hypothesis was false), but a *lack* of a result.

  • Holly says:

    My PI would have yelled. And if it was a bad day, said i was stupid. If I look mutinous enough, an apology might follow.

Leave a Reply