Archive for the 'Peer Review' category

Put up or shut up time, all ye OpenSciencePostPubReview Waccaloons!

Oct 24 2013 Published by under Academics, Conduct of Science, Open Access, Peer Review

PubMed Commons has finally incorporated a comment feature.

NCBI has released a pilot version of a new service in PubMed that allows researchers to post comments on individual PubMed abstracts. Called PubMed Commons, this service is an initiative of the NIH leadership in response to repeated requests by the scientific community for such a forum to be part of PubMed. We hope that PubMed Commons will leverage the social power of the internet to encourage constructive criticism and high quality discussions of scientific issues that will both enhance understanding and provide new avenues of collaboration within the community.

This is described as being in beta test version and for now is only open to authors of articles already listed in PubMed, so far as I can tell.

Perhaps not as Open as some would wish but it is a pretty good start.

I cannot WAIT to see how this shakes out.

The Open-Everything, RetractionWatch, ReplicationEleventy, PeerReviewFailz, etc acolytes of various strains would have us believe that this is the way to save all of science.

This step of PubMed brings the online commenting to the best place, i.e., where everyone searches out the papers, instead of the commercially beneficial place. It will link, I presume, the commentary to the openly-available PMC version once the 12 month embargo elapses for each paper. All in all, a good place for this to occur.

I will be eager to see if there is any adoption of commenting, to see the type of comments that are offered and to assess whether certain kinds of papers get more commentary than do others. All and all this is going to be a neat little experiment for the conduct-of-science geeks to observe.

I recommend you sign up as soon as possible. I'm sure the devout and TrueBelievers would beg you to make a comment on a paper yourself so, sure, go and comment on some paper.

You can search out commented papers with this string, apparently.

In case you are interested in seeing what sorts of comments are being made.

25 responses so far

Post-publication peer review and preprint fans

Anyone who thinks this is a good idea for the biomedical sciences has to have served as an Associate Editor for at least 50 submitted manuscripts or there is no reason to listen to their opinion.

27 responses so far

Repost: Study Section, Act I

I think it has been some time since I last reposted this. This originally appeared Jun 11, 2008.

Time: February, June or October
Setting: The Washington Triangle National Hotel, Washington DC

    Dramatis Personæ:

  • Assistant Professor Yun Gun (ad hoc)
  • Associate Professor Rap I.D. Squirrel (standing member)
  • Professor H. Ed Badger (standing member, second term)
  • Dr. Cat Herder (Scientific Review Officer)
  • The Chorus (assorted members of the Panel)
  • Lurkers (various Program Officers, off in the shadows)

Continue Reading »

No responses yet

If you are going to talk about "tiers", then you'd better own that

SevenTierCakeOccasionally during the review of careers or grant applications you will see dismissive comments on the journals in which someone has published their work. This is not news to you. Terms like "low-impact journals" are wonderfully imprecise and yet deliciously mean. Yes, it reflects the fact that the reviewer himself couldn't be bothered to actually review the science IN those paper, nor to acquaint himself with the notorious skew of real world impact that exists within and across journals.

More hilarious to me is the use of the word "tier". As in "The work from the prior interval of support was mostly published in second tier journals...".

It is almost always second tier that is used.

But this is never correct in my experience.

If we're talking Impact Factor (and these people are, believe it) then there is a "first" tier of journals populated by Cell, Nature and Science.

In the Neurosciences, the next tier is a place (IF in the teens) in which Nature Neuroscience and Neuron dominate. No question. THIS is the "second tier".

A jump down to the IF 12 or so of PNAS most definitely represents a different "tier" if you are going to talk about meaningful differences/similarities in IF.

Then we step down to the circa IF 7-8 range populated by J Neuroscience, Neuropsychopharmacology and Biological Psychiatry. Demonstrably fourth tier.

So for the most part when people are talking about "second tier journals" they are probably down at the FIFTH tier- 4-6 IF in my estimation.

I also argue that the run of the mill society level journals extend below this fifth tier to a "the rest of the pack" zone in which there is a meaningful perception difference from the fifth tier. So.... Six tiers.

Then we have the paper-bagger dump journals. Demonstrably a seventh tier. (And seven is such a nice number isn't it?)

So there you have it. If you* are going to use "tier" to sneer at the journals in which someone publishes, for goodness sake do it right, will ya?

*Of course it is people** who publish frequently in the third and fourth tier and only rarely in second tier, that use "second tier journal" to refer to what is in the fifth or sixth tier of IFs. Always.

**For those rare few that publish extensively in the first tier, hey, you feel free to describe all the rest as "second tier". Go nuts.

22 responses so far

GrantRant XI

Jan 31 2013 Published by under Grant Review, Grantsmanship, NIH, NIH funding, Peer Review

Combative responses to prior review are an exceptionally stupid thing to write. Even if you are right on the merits.

Your grant has been sunk in one page you poor, poor fool.

12 responses so far

The glass is half full

Jan 31 2013 Published by under Grant Review, NIH, NIH funding, Peer Review

There is nothing like a round of study section to make you wish you were the Boss of ALL the Science.

There is just soooo much incredible science being proposed. From noob to grey beard the PIs are coming up with really interesting and highly significant proposals. We'd learn a lot from all of them.

Obviously, it is the stuff that interests me that should fund. That stuff those other reviewers liked we can do without!

Sometimes I just want to blast the good ones with the NGA gun and be done.

Notice of Grant Award

2 responses so far

GrantRant VIII

Do not EVER spend so much time geeking away about the amazingly swell trees that you will be characterizing that you forget to convince the reviewer that the forest itself holds any interest. And I mean ANY interest.....Seriously dudes, I'm trying to help you out here but you are giving me absolutely nothing to work with. There is barely any point in me even reading your experimental manipulations....I can tell already there is no overall justification for doing them in the first place!

12 responses so far

Damaged Goods

Jan 14 2013 Published by under Peer Review, Science Publication

Have you ever had a manuscript severely damaged by the process of peer review?

by way of example, I can recall one time where the Editor demanded I chop off two experiments..and I did so*.

Otherwise, I'm generally of the opinion that peer review has a positive impact on the manuscript.

*Those figures have yet to see the light of day and may never get published. A shame, but then, we got the paper published and the main point was one of the other figures.

13 responses so far

GrantRant VII

Everyone is going to hate you, pretty much.

Think about it. You have 7-10 grants assigned in your pile on a typical study section these days. Odds are good that at best one or two of these is going to be good enough to be in the hunt for funding. The rest of the panel is in the same boat, so it really doesn't matter that the applicants don't know precisely which of you* on the panel reviewed his or her proposal.

80-90 % of the applicants are going to be mad at you.

Since you have been selected for expertise in the relevant field...these are people who you know. You know their work and you probably like and cite it. They know you. They know your work.

And for at least a while after they see their disappointing score, and for another while after the pink sheets are posted, they cannot help but hate you a little.

Maybe even a lot.


*If you were triaged you do know for absolute sure that every member listed on that panel roster stood by and refused to pull your application up for discussion.

46 responses so far

Mutually Assured Destruction theory of NIH Grant Review

Jan 09 2013 Published by under NIH, NIH Careerism, Peer Review

Back in the distant past, younguns, the US was involved in a struggle with the Soviet Union that many felt was an existential threat to our continuation on this planet. Among other features, this Cold War (perhaps better termed Ongoing Proxy War) featured the buildup of ecosphere destroying megaweapon bombs.

The fuzzy blankie we used to keep from going insane was the thought that since both sides could destroy huge amounts of the other side's population, render much of its territory uninhabitable, and could do so should the other side move first, we were safe.

Since we were mutually assured to destroy each other, the logic of starting some serious beef was an insane one. Nobody in their right mind would actually do such a thing. So this kept certain behaviors (like the hilariously NewSpeak "pre-emptive counter-strike" with nuclear weapons) off the table.

In discussions of NIH Grant review, there is often a certain paranoia voiced that members of the review panel use this position of tremendous power to screw over their scientific rivals. Sounds plausible, does it not? After all, this grant stuff is a zero-sum game and the "peers" of peer review are after the same pool of money that each applicant is eying. These days it is a good bet that the reviewer has her own application under review elsewhere in the CSR...or has one pending funding in this self-same Fiscal Year.

That's before we get to scientific competition to publish papers in some research area first. We all know that first is best and all others might as well go home, right? And any rational grant funding agency (don't laugh) like the NIH should diversify their portfolio such that if they fund grant on a topic, the chances of another one on nearly the same topic should be lesser.

Naturally, the closer the reviewer expertise is to the grant in question, the closer this reviewer is to being in direct conflict of interest at some level.

My first approach to comforting the distraught Assistant Professor is to emphasize that our peers are professionals, with some degree of ethical centeredness who are for the most part attempting to do the job as asked.

This doesn't comfort everyone. So today I offer the Mutually Assured Destruction theory for your consideration.

One of the most surprising things I found about study section service is the rapidity and surety with which payback opportunity was provided. During the early days of my study section service it was the appearance of many grants in my piles to review that were submitted by PIs who had previously appeared on study section panels reviewing my own proposals. After I'd been reviewing for a little bit, it was remarkable how quickly people who's grants had appeared in study sections that I was on (and in some cases apps to which I had been assigned) were now in a position on panels reviewing other grants of mine.

I came away from all of this with the understanding that what goes around comes around VERY quickly in NIH grant review.

So for the paranoid consider this additional source of pressure on the reviewer. If you don't trust their professionalism, trust in their self-interest. This Mutual Assurance tends to suggest that reviewers would be crazy to screw with applicants out of pure self-interested bias.

16 responses so far

Older posts »