Faculty of 1000

Post-publication peer review

I’m a believer

Posted by rpg on 3 February, 2010

I took my daughters round the new Darwin Centre at the Natural History Museum last year. Because we happen to be friends, I managed to persuade the incomparable Karen James, of The Beagle Project fame, to show us behind the scenes.

Calamari at the NHM After that I took the girls into the ‘Cocoon‘, a huge butterfly egg-type structure that contains 20 million or more specimens, with superbly-done exhibits and displays. The Cocoon lets visitors see into the workings of the Museum; quite literally, because one side of the egg cuts away into the research labs (where Karen works). As we walked around we came across a video display of Karen herself, talking about the process of publishing science, how you write and revise a manuscript and send it off to the dreaded ‘peer review’.

Karen did a splendid job explaining the process: how other scientists in your field look at your work and—in an ideal world—check that you’ve done the work right and that you’ve cited all the relevant literature; that the manuscript is sound. She also conveyed, far too convincingly, the heartbreak of having a treasured manuscript rejected!

The process of science for the last three hundred and more years is based on peer review. Other scientists check your work and say yup, that looks OK or no, you need to do this other experiment or read these papers. Unfortunately some reviewers (and I stress, these people are peers, that is they are your equals; not some shadowy cabal curating or judging Science from on high) do seem to hold personal grudges, or have strange agendas, or simply not be very good. (And yes, sometimes your own work is pants and should be rejected. Deal with it.)

This leads to people making wide-ranging and inflammatory statements such as “peer review is broken.” Some of them even write letters about it (as reported by the Beeb, six months later). This leads to calls for making the peer review process ‘open‘; i.e. publishing the correspondence between the reviewers and the editors, and maybe even removing the anonymity part.

We’ve been here before (haven’t we always?):

All editors have seen curt, abusive, destructive reviews and assumed that the reviewer would not have written in that way if he or she were identifiable. Openness also links accountability with credit. One important defect of closed review is that reviewers don’t receive academic credit. Finally, openness should eliminate some of the worst abuses of peer review, where reviewersunder the cloak of anonymitysteal ideas or procrastinate

BMJ, 1999

but it’s not clear to me, and indeed the results of those BMJ studies tend to bear me out, whether this is really a problem, or whether the perception is far worse than the reality. And I’ve had a paper that took nearly two years to get published.

I remain to be convinced that peer review is broken. The idea of some sort of clique quashing acceptance of manuscripts isn’t that far-fetched, pace Philip Campbell. Strong editors will get round it, but in some fields it’s quite possible for one or two individuals to make it really difficult to get anything published. This tends to be self-limiting though: in the extreme case, the field simply dies. In my experience of this, the community know who those people were, so open peer review probably wouldn’t improve matters. Double-blind will not work because it’s going to be reasonably trivial to figure out is the author. People will still sit on manuscripts, and we already know who does this when it happens.

Making the reviewers’ comments, signed or not, public might not ‘improve’ peer review: however I do believe it has value. There is virtually no training in reviewing papers, and if young post-docs and grad students could see a wide range of reviews of many different papers, surely that can only improve their reviewing skills? Maybe it would even serve to make the first submission better if nascent authors were to look at reviews in their field, and discover the common mistakes?

Having said that, I am keen to see greater accountability. On a personal level, I wouldn’t write anything I wouldn’t put my name next to, and I don’t actually see why peer review should be any different. At f1000, for example, we already put our Faculty Members’ names on their evaluations, and we call this ‘post-publication peer review’. (Our motives are slightly different of course: we’re saying that you should take notice because of who’s writing them.)
Interacting
The EMBO Journal has been experimenting with publishing reviews of accepted manuscripts for a year now. I was amused to find that when I clicked on one of the reviews at random, it was a paper about my old friend talin. And it’s from Mark Ginsberg and Iain Campbell, FRS, with both of whom have I coauthored papers.

Small world.

Advertisements

6 Responses to “I’m a believer”

  1. Da Wintle said

    Nicely put. I am of the opinion that publishing peer reviews will water the whole process down, as reviewers will find themselves needing to be *more* polite and politically correct than previously. Remember, scientists need to interact in many ways other than peer review – on promotion or search committees, grant panels, and the like. If your (justifiably) scathing comments about someone’s crappy research are identifiable, I think it’s more likely that personal vendettas could occur.

    I also think it’s likely that manuscript reviews by junior faculty would be made even more toothless than those by others, since these are the people most susceptible to future harm from colleagues who’ve been upset with their reviews.

    P.S. Nice picture of your daughter, RPG – she looks very thoughtful.

  2. Matthew McClure said

    I do fear that if reviewers were required to sign their peer reviews you would see a “watering down” of the review process. While scientists should never be afraid to state their opinions many don’t like to publicly say “you manuscript/science is a piece of crap because of X,Y,Z.”

    That said it would be nice to see reviews on good and bad papers to help train and guide new scientists. While some advisers take time to help train their graduate students in the peer review process, not all do.

  3. rpg said

    Thanks DW. Your arguments are familiar. I have some sympathy with them.

    Matthew, thanks for stating that argument so succinctly! And yes, your point about training is well made. Better training might (might) reduce the overall problem—perceived or otherwise.

  4. Brittany said

    Coming from the editorial side of peer review, I have to say that my one concern about revealing reviewers’ identities to the authors, even after a paper is accepted and especially if it is rejected, is that the authors will harrass the reviewers endlessly…like some harass the editorial board.

  5. […] I’m a believer […]

Sorry, the comment form is closed at this time.

 
%d bloggers like this: