Archive for the 'Peer Review' category

Post-publication peer review and preprint fans

Anyone who thinks this is a good idea for the biomedical sciences has to have served as an Associate Editor for at least 50 submitted manuscripts or there is no reason to listen to their opinion.

Share

27 responses so far

Repost: Study Section, Act I

I think it has been some time since I last reposted this. This originally appeared Jun 11, 2008.


Time: February, June or October
Setting: The Washington Triangle National Hotel, Washington DC

    Dramatis Personæ:

  • Assistant Professor Yun Gun (ad hoc)
  • Associate Professor Rap I.D. Squirrel (standing member)
  • Professor H. Ed Badger (standing member, second term)
  • Dr. Cat Herder (Scientific Review Officer)
  • The Chorus (assorted members of the Panel)
  • Lurkers (various Program Officers, off in the shadows)

Continue Reading »

Share

No responses yet

If you are going to talk about "tiers", then you'd better own that

SevenTierCakeOccasionally during the review of careers or grant applications you will see dismissive comments on the journals in which someone has published their work. This is not news to you. Terms like "low-impact journals" are wonderfully imprecise and yet deliciously mean. Yes, it reflects the fact that the reviewer himself couldn't be bothered to actually review the science IN those paper, nor to acquaint himself with the notorious skew of real world impact that exists within and across journals.

More hilarious to me is the use of the word "tier". As in "The work from the prior interval of support was mostly published in second tier journals...".

It is almost always second tier that is used.

But this is never correct in my experience.

If we're talking Impact Factor (and these people are, believe it) then there is a "first" tier of journals populated by Cell, Nature and Science.

In the Neurosciences, the next tier is a place (IF in the teens) in which Nature Neuroscience and Neuron dominate. No question. THIS is the "second tier".

A jump down to the IF 12 or so of PNAS most definitely represents a different "tier" if you are going to talk about meaningful differences/similarities in IF.

Then we step down to the circa IF 7-8 range populated by J Neuroscience, Neuropsychopharmacology and Biological Psychiatry. Demonstrably fourth tier.

So for the most part when people are talking about "second tier journals" they are probably down at the FIFTH tier- 4-6 IF in my estimation.

I also argue that the run of the mill society level journals extend below this fifth tier to a "the rest of the pack" zone in which there is a meaningful perception difference from the fifth tier. So.... Six tiers.

Then we have the paper-bagger dump journals. Demonstrably a seventh tier. (And seven is such a nice number isn't it?)

So there you have it. If you* are going to use "tier" to sneer at the journals in which someone publishes, for goodness sake do it right, will ya?

___
*Of course it is people** who publish frequently in the third and fourth tier and only rarely in second tier, that use "second tier journal" to refer to what is in the fifth or sixth tier of IFs. Always.

**For those rare few that publish extensively in the first tier, hey, you feel free to describe all the rest as "second tier". Go nuts.

Share

22 responses so far

GrantRant XI

Jan 31 2013 Published by under Grant Review, Grantsmanship, NIH, NIH funding, Peer Review

Combative responses to prior review are an exceptionally stupid thing to write. Even if you are right on the merits.

Your grant has been sunk in one page you poor, poor fool.

Share

12 responses so far

The glass is half full

Jan 31 2013 Published by under Grant Review, NIH, NIH funding, Peer Review

There is nothing like a round of study section to make you wish you were the Boss of ALL the Science.

There is just soooo much incredible science being proposed. From noob to grey beard the PIs are coming up with really interesting and highly significant proposals. We'd learn a lot from all of them.

Obviously, it is the stuff that interests me that should fund. That stuff those other reviewers liked we can do without!

Sometimes I just want to blast the good ones with the NGA gun and be done.

--
Notice of Grant Award

Share

2 responses so far

GrantRant VIII

Do not EVER spend so much time geeking away about the amazingly swell trees that you will be characterizing that you forget to convince the reviewer that the forest itself holds any interest. And I mean ANY interest.....Seriously dudes, I'm trying to help you out here but you are giving me absolutely nothing to work with. There is barely any point in me even reading your experimental manipulations....I can tell already there is no overall justification for doing them in the first place!

Share

12 responses so far

Damaged Goods

Jan 14 2013 Published by under Peer Review, Science Publication

Have you ever had a manuscript severely damaged by the process of peer review?

by way of example, I can recall one time where the Editor demanded I chop off two experiments..and I did so*.

Otherwise, I'm generally of the opinion that peer review has a positive impact on the manuscript.

___
*Those figures have yet to see the light of day and may never get published. A shame, but then, we got the paper published and the main point was one of the other figures.

Share

13 responses so far

GrantRant VII

Everyone is going to hate you, pretty much.

Think about it. You have 7-10 grants assigned in your pile on a typical study section these days. Odds are good that at best one or two of these is going to be good enough to be in the hunt for funding. The rest of the panel is in the same boat, so it really doesn't matter that the applicants don't know precisely which of you* on the panel reviewed his or her proposal.

80-90 % of the applicants are going to be mad at you.

Since you have been selected for expertise in the relevant field...these are people who you know. You know their work and you probably like and cite it. They know you. They know your work.

And for at least a while after they see their disappointing score, and for another while after the pink sheets are posted, they cannot help but hate you a little.

Maybe even a lot.

Joyous.

__
*If you were triaged you do know for absolute sure that every member listed on that panel roster stood by and refused to pull your application up for discussion.

Share

46 responses so far

Mutually Assured Destruction theory of NIH Grant Review

Jan 09 2013 Published by under NIH, NIH Careerism, Peer Review

Back in the distant past, younguns, the US was involved in a struggle with the Soviet Union that many felt was an existential threat to our continuation on this planet. Among other features, this Cold War (perhaps better termed Ongoing Proxy War) featured the buildup of ecosphere destroying megaweapon bombs.

The fuzzy blankie we used to keep from going insane was the thought that since both sides could destroy huge amounts of the other side's population, render much of its territory uninhabitable, and could do so should the other side move first, we were safe.

Since we were mutually assured to destroy each other, the logic of starting some serious beef was an insane one. Nobody in their right mind would actually do such a thing. So this kept certain behaviors (like the hilariously NewSpeak "pre-emptive counter-strike" with nuclear weapons) off the table.

In discussions of NIH Grant review, there is often a certain paranoia voiced that members of the review panel use this position of tremendous power to screw over their scientific rivals. Sounds plausible, does it not? After all, this grant stuff is a zero-sum game and the "peers" of peer review are after the same pool of money that each applicant is eying. These days it is a good bet that the reviewer has her own application under review elsewhere in the CSR...or has one pending funding in this self-same Fiscal Year.

That's before we get to scientific competition to publish papers in some research area first. We all know that first is best and all others might as well go home, right? And any rational grant funding agency (don't laugh) like the NIH should diversify their portfolio such that if they fund grant on a topic, the chances of another one on nearly the same topic should be lesser.

Naturally, the closer the reviewer expertise is to the grant in question, the closer this reviewer is to being in direct conflict of interest at some level.

My first approach to comforting the distraught Assistant Professor is to emphasize that our peers are professionals, with some degree of ethical centeredness who are for the most part attempting to do the job as asked.

This doesn't comfort everyone. So today I offer the Mutually Assured Destruction theory for your consideration.

One of the most surprising things I found about study section service is the rapidity and surety with which payback opportunity was provided. During the early days of my study section service it was the appearance of many grants in my piles to review that were submitted by PIs who had previously appeared on study section panels reviewing my own proposals. After I'd been reviewing for a little bit, it was remarkable how quickly people who's grants had appeared in study sections that I was on (and in some cases apps to which I had been assigned) were now in a position on panels reviewing other grants of mine.

I came away from all of this with the understanding that what goes around comes around VERY quickly in NIH grant review.

So for the paranoid types...do consider this additional source of pressure on the reviewer. If you don't trust their professionalism, trust in their self-interest. This Mutual Assurance tends to suggest that reviewers would be crazy to screw with applicants out of pure self-interested bias.

Share

16 responses so far

GrantRant V

Jan 08 2013 Published by under NIH Careerism, Peer Review

A grant review subculture that has been established from sub fields in which not much happens between grant submission and review has difficulty dealing with an exploding topic.

In the general case, it seems slightly unfair to kill a proposal over the four papers that have appeared after the poor sucka PI submitted the application.

Share

8 responses so far

Older posts »