On Supplemental Data

Oct 21 2010 Published by under Academia, Uncategorized

Sci isn't feeling very well today folks. She hit the wall during one of her training sessions VERY hard yesterday, and is still trying to convince her weary legs to carry her about the lab with her normal alacrity. So I'm gonna go on a brief rant on the subject of supplemental data. You were warned.

Drugmonkey pointed out a bit ago that one of the bigger journals in our field, the Journal of Neuroscience will be getting rid of supplemental data. I have to admit, Sci may have cheered a little at this prospect. In fact, a LOT of scientists are cheering. But I've had some of my non-science friends ask me WHY.

And so, Sci will tell you.

An example of supplemental data. Sci wishes she could make her own flowchart of supplemental data. It might look like this:

Does the journal have ridiculously short page/word limits compared to the ridiculously huge amount of data they require for publication? Yes.
Are your methods way too numerous and complicated due to the ridiculously huge amount of data required for publication? Yes.
Did a reviewer demand extra experiments which in no way help your paper's cause and which no one but that particular reviewer cares about? Yes.
Do you feel the need to publish ALL OF YOUR FIGURES. ALL OF THEM? Yes.

Then you need SUPPLEMENTAL DATA!!!

Now, are any of the above items NECESSARY to the publication of decent science?

Nope.

I feel like this is a trend toward several things:

1) Needless amounts of reviewer requested experiments that don't add anything. Now don't get me wrong, often reviewers request experiments that NEED to be done, and that really do enhance the paper and add a lot. But not always, and you can always tell when you see something looking kind of unrelated and stuck in the supplemental data.

2) The pressure to publish a huge complete story. This is not necessarily bad, but it's getting to the point where every time I try to publish a paper in a decent journal, I feel like I need to publish the freakin Silmarilion! Journals are very interested in "impact" and in how high priority the manuscripts going into them are, and this means that they really only want to publish complete stories. While I understand that (and a good complete story is MUCH more satisfying to read), it often results in huge amounts of supplemental data in an effort to show how much the scientists covered their bases. Perhaps this indicates, not a need for less of a story, but a need for designing a more elegantly crafted series of experiments.

So recently the Journal of Neuroscience decided to get rid of the supplemental data. Their reasons for doing so are as follows:

1) Large amounts of supplemental materials means that reviewers will often overlook that material or not review it at all, meaning that the supplemental data is not peer reviewed (icky!).

2) Supplemental data encourages excessive demands from reviewers.

3) Stuff that would be important to the article is being forced into the supplemental data, which undermines the concept of a self-contained research report.

w00t. I hear all that and raise you eleventy.

Of course, the final result of HAVING supplemental data in high end journals is that you end up with these tiny, tight little stories with ridiculously small word limits in Science, and with the "oh, yeah, here's what we REALLY did" in the fifty kabillion supplemental pages. It also means that you can get away with not particularly well crafted stories, as your many little side experiments get relegated to the supplemental data. So I'm glad that J Neuroscience is going to be getting rid of Supplemental data, but I also hope that they, and their REVIEWERS, keep this in mind. It's going to mean, maybe demanding tighter stories, and maybe ALSO not demanding stories that are as sweeping and huge as the big journals like to go for. And it's going to mean thinking carefully before reviewers suggest that you do fifty billion extra experiments to tie in to their little corner of the field. It may also mean, that if Journals really want to publish well-done, carefully crafted, high end experiments, with difficult methods, they may need to expand their sometimes rather paltry word limits (seriously, Nature? 3000 words INCLUDING references and captions?! You gotta be kidding. I 'm all for being succinct, but that's really rather ridiculous. You want well-cited, detailed methods, plenty of well-done figures, a complete story, and perspective on the field in 4500 words including the references? In the dreadfully butchered words of Elizabeth Bennett "I am no longer surprised at you knowing only *six* accomplished neuroscientists, Mr. Darcy. I rather wonder at your knowing *any*.").

So I say down with Supplemental Data, and with excessive reviewer demands. Down with teensy word limits which cannot carefully encompass what it is we actually DID in our experiments. Taking down the Supplemental data is the first step. I think this is a good thing, but hey, what do I know.

12 responses so far

  • Samantha Alsbury says:

    As John Wilbanks said at a TalkScience event on publishing, but other people I'm sure have said it too, papers are now often just adverts.

  • drugmonkey says:

    there is another factor that benefits GlamourMags demanding that another 2-4 papers worth (from times past) of lower-IF journal pubs be buried as "supplemental data". It further enhances their citations at the expense of "lesser" journals. In a prior time, the sexy hawt figures would be in Nature / Science and then the fleshing out of the "full story" would be in subsequent papers. plural.

    this is a monopolization strategy, pure and simple business and nothing to do with the communication of science. in fact it is ruining science.

    major kudos to JNeuro for putting down their foot. Now we'll see if they have the stones to put that foot in the arse of GlamourWannabe reviewers...

  • DrugMonkey says:

    Oh and Sci? The idea of a "complete story" is utter bollocks. Science has neither a beginning nor an end...it exists on a continuous (curvilinear) line. Have you ever read a paper and had no remaining experimental questions that contribute to that story? I sure have not.

    • scicurious says:

      Oh yes, I know the "complete story" is silly, I'm not really referring to that, but to the concept of the "story" that we often use when talking about published papers (that experiment doesn't fit with the story), and how to get in a GlamourMag you need some combination of behavior, circuitry, mechanism of action of specific thingy you're testing, and further behavior. I don't argue that this makes for a nice paper, but often it's gone about in a very confusing and burdensome way.

  • Pharm Sci Grad says:

    Delurking to say... AMEN!!!

    Apart from what DrugMonkey said about having to shove multiple papers' worth of data into one GlamourMag pub, there's the reading of these tombs - when every other sentence references the supplemental information, there is no introduction to speak of (can we say confusing for those outside the subfield?!) because those words count against the total, and the methods in the paper are laughable (the real, detailed methodology is in the supplemental info). So you thought you were reading a 6-10 page paper, when you (or at least I) actually need to read a 50 page paper plus a couple of the references to really understand everything they're saying.

    I really, really dislike having to read papers I can't reasonably digest in one sitting because of issues such as these. I hope others follow J Neuro's lead!

  • Janne says:

    Journal of Neuroscience doesn't have a page limit, though? The introduction is interestingly terse (good luck cramming all background into 500 words) but the main sections are as long as you want them to be.

    Supplemental material in general is bad, but supplemental data is a good idea and should perhaps be required. Specifically, a policy that requires the posting of the raw data and of the source code of any model or homegrown analysis tool would be a really good idea. If not, then that material is more likely than not completely unavailable within a few years.

  • Scott says:

    I can't remember the last time I saw a random Silmarillion reference. You made my not-very-well-concealed inner nerd happy today.

  • FiSH says:

    To add my two cents on the other side, kind of; although I agree with Drug Monkey's summary of the state of publication demands and the reasons for those demands, and also their overall impact on science, one other issue is that we are amassing data at ridculous rates now, and many types of data sets require someplace to put them as supplemental data because only the terse summary can be included in a paper, such as the results of whole genome assocaition studies, massive microarrays for gene expression, and so forth. I am, at my heart, a behaviorist, but even I have problems with this sort of thing - I have often had comments from reviewers saying "Do you really want to publish all of those behavioral measures? Can't you just cut it down to one or two?" I could obviously never respond to such comments from a reviewer in the way I would have wanted, but found some polite way to say "No, you idiot, I can't".

  • I consider the "story" metaphor offensive and disturbing, and do my best to avoid it.

  • drugmonkey says:

    FiSH,

    One of the reasons for publishing more, rather than less, of your work is that given it is an interesting question somebody else is going to be working on it eventually. You did those other experiments for a reason- so they ended up being boring controls or negative outcomes and the reviewers want you to drop them. But here's the thing- if you don't publish those, some poor chump or twelve are going to be repeating them in the future to confirm the exact same "boring" result you did. Huge waste of time and money.

    Also, you never know how the study you did for reason A is going to be fascinating for a whole 'nother set of reasons to people who come at it from a different approach. So the Bunny Hoppers say "hey, why are you including that stupid bunny strolling assay, we're only interested in Hopping dagnabit!" then along come the Badger Strollers and OMG, you have an amazing nuance that they didn't necessarily run across in their Badgerish ways...

  • The diverse results in a scientific publication will have different levels of value depending on the reader's personal research interests. The availability of supplemental data allows for the key discoveries in a comprehensive study to standout in the main paper without the potential loss of useful information to specialists who can benefit from the other results. This is particularly important for systems biology type investigation, which are data-rich and could be further investigated by alternative strategies.

    Access to supplemental data can also instill greater confidence in the rigor undertaken in a published study. Ph.D. thesis examiners expect a high degree of detail in the descriptions of methodologies, results and discussion from a Ph.D. candidate. Should we expect less from experienced investigators? For pragmatic reasons, published scientific papers should obviously not be so detailed. However, I for one would like to have the choice of seeing more data from a study if it is available. I would not want to penalize those journals that provide such an option.

  • [...] I’ve railed against the rise of supplemental data and methods before, but, having just reviewed a paper where I spent more time reading the supplemental sections versus the actual fucking paper, what Scicurious wrote struck a chord with me: [...]