Faculty of 1000

Post-publication peer review

Posts Tagged ‘peer review’

Publish or perish – a question of ethics

Posted by Callum Anderson on 15 March, 2010

I got a very strong sense of deja vu when leafing through PLoS Biol recently. I was sure I had seen something very similar to Jeffrey Shaman’s paper Absolute Humidity and the Seasonal Onset of Influenza in the Continental United States before.

A quick check on PubMed proved me right. I found the following, published two months earlier, in PLoS Curr Influenz:

Absolute Humidity and the Seasonal Onset of Influenza in the Continental US
Jeffrey Shaman,* Virginia Pitzer,† Cecile Viboud,‡ Marc Lipsitch,§ and Bryan Grenfell

PubMed ID 20066155

Because this was PLoS, I was also able to print the full paper and compare. I couldn’t find any differences whatsoever between the two papers. In fact they were exactly the same except for a reshuffling of author order and an abbreviation in the title.

A quick check back on PLoS Biol and I notice that someone else has seen the discrepancy. A comment attached to the article begins with the following

Compare, published in PLoS Currents influenza (dec 18th)
Absolute Humidity and the Seasonal Onset of Influenza in the Continental US
Jeffrey Shaman,* Virginia Pitzer,† Cecile Viboud,‡ Marc Lipsitch,§ and Bryan Grenfell

PubMed ID 20066155

with (and not cited, if I am not mistaken)

Absolute Humidity and the Seasonal Onset of Influenza in the Continental United States (23 february 2010)

Jeffrey Shaman1*, Virginia E. Pitzer2,3,4, Cécile Viboud2, Bryan T. Grenfell2,4,5, Marc Lipsitch6,7,8

When this poster commented, only one of the articles was listed in PubMed. A search for “Absolute humidity” on PubMed today however yielded the following results [click it to get full size]

A PLoS spokesperson had answered the comment in less than 3 hours (perhaps they anticipated something being said). Their official line was as follows

PLoS Biology is fully aware of the authors’ submission to PLoS Currents referenced above. PLoS Currents is a website for immediate, open communication and discussion of new scientific data, analyses, and ideas in a critical research area. The work is screened by experts, but is not subject to in-depth peer review…

Our policy until now (February, 2010) has been to allow resubmission of PLoS Currents content to another PLoS journal. However, the decision to include Currents in PubMed (and PubMed Central) has caused us to reconsider the status of content communicated via Currents, relative to other journals.

I am certainly not convinced by this argument. Having personal experience of getting journals into PubMed, it is not something that happens immediately; the typical process is eight to twelve weeks and PLoS Curr Influenz was already accepted by PubMed in 2009. The accepted date on the re-submitted paper in PLoS Biol was January 20, 2010.

And even worse still, the received date of the paper by PLoS Biol was September 10, 2009. PLoS Curr Influenz did not even accept the duplicate paper until December 18, 2009.

The dates simply don’t add up, a journal doesn’t just email PubMed and expect to show content the next day, and feigning innocence just makes PLoS look at worst deceitful and at very best incompetent. If PLoS was aware that the paper had been submitted to both journals, and was aware that PLoS Curr Influenz would be listed on PubMed, they should have made a full disclosure on the paper subsequently published in PLoS Biol.

Now, I am very much in favour of rapid communication journals, I think they represent an excellent platform to publish cutting edge research, but a distinction between these and traditionally peer-reviewed journals must be drawn somewhere. Should a publication like this really be submitting content to PubMed when their editorial policy allows re-submission in other PLoS journals? PLoS have been having their cake and eating it for a long time now. In a world where publication stats are frequently used as a method of judging the worth of a researcher, are the authors here benefiting twice from the same paper? And PubMed has a very clear policy on duplicate articles, which PLoS should know about.

So why didn’t they do it? Why didn’t they tell PubMed that they would be knowingly supplying duplicate articles? Well I do have a theory [snip-snip – F1000 Lawyers]… But it would be much better to see what you think.

Posted in Journalism, Literature | Tagged: , , , , , , , , , | 24 Comments »

On a new publishing model-the winner!

Posted by rpg on 16 February, 2010

Ladles and gentlespoons, the results are in. We had an amazing response, and after sifting through a mass of #sci140-tagged tweets, discarding all the retweets and publicity (and a huge thank-you to everyone who spread the word), we had 197 unique entries (grep saved my life).

Many of you posted very witty ‘historical’ paper summaries, but there were several who managed to squeeze their own papers into 140 133 characters too. This, I think, was far more difficult, even if it did not lend itself so readily to humour.

It turned out to be quite an interesting social experiment, too. There were a number of themes, possibly the most popular being the structure of DNA (not surprising seeing as most of my twitfriends are at least vaguely biochemical). This from @SelectAgent was one of the best:

Salt of DNA structure= double helix. Strands anti-parallel; has implications. (PS Rosie didn’t help)

Physics, especially quantum mechanics, also featured heavily, and @pssalgado deserves a special mention for

Where are you, Heis? “Don’t know exactly, but I can tell you how fast I’m going!”

Galileo was another favourite among physicists, @sciencebase almost scooping the prize with

Dropped heavy and light ball at Pisa; saw landed at same time. Peer review problems now, especially after telescope incident.

Many entries had fun with Mendel; here’s @marymulv:

Peas for tea. Again! (Damn that gardener.) Smooth, wrinkly, smooth… Is that a pattern? Hm. Should I tell that Darwin fellow??

Stanley Milgram was the subject of a couple of tweets, @sciencebase again making me laugh with

Stanley, is this circuit really 450 Volts, those people look like they’re in real pain? Shut up and just push the lever

Some of you are obviously budding behavioural psychologists, as Pavlov’s famous experiment also attracted a lot of attention? My favourite? @enniscath‘s

Rang bell, fed dogs. Rang bell again, dogs drooled. NO FOOD FOR YOU! BAD DOG! (heh heh. Stoopid dogs).

Honorable mentions should also go to silentypewriter and yokofakun for sheer wit and volume dedication.

But there can be only one winner, I’ve decided. For his poetic take on Watson & Crick’s structure of DNA, and for a smart paper of his own, the winner of the #sci140 competition is…

@CameronNeylon (Cameron Neylon)

The winning entries from Cameron are

Take bacterial cell wall chemistry. Replace proteins + wall with any prot + beads. Easy protein labelled beads! (link)


2 interwound helices, with handedness right, and a 3.4 pitch, and hydration just right + keto not enol or they don’t zip up right (link)

A bag full of f1000 goodies will be winging its way to him very shortly.

Mad props to Cameron, and a big thank you to everyone who participated. It was such good fun, I rather think we might run something like this again. Keep an eye on @f1000 on Twitter for the next one. The full list of #sci140 entries are below the fold.
Read the rest of this entry »

Posted in Competition, Friday afternoon | Tagged: , , , , , , , , , | 6 Comments »

On a new publishing model-update 2

Posted by rpg on 11 February, 2010

And… here’s the next batch of #sci140 entries, since 10.40 today. If you think yours should be on the list, then please let me know (with the twitter URL if possible).

Keep ’em coming…

Read the rest of this entry »

Posted in Competition | Tagged: , , , , , | Comments Off on On a new publishing model-update 2

On a new publishing model-update

Posted by rpg on 10 February, 2010


I created a twitter storm yesterday, as people leapt on the #sci140 meme like kangaroos. Thanks to everyone who picked up on it, RTed and entered. Some of you made me laugh out loud. Below the fold you’ll find all the entries as at 10.24 UTC today (I’ve spent much of the morning stripping out extraneous links, styling, RTs and dupes).

Meantime, David Bradley has extended the meme, using the hashtag #histsci140. That looks like fun too. He’s offering a book prize. I can’t compete with that I’m afraid, but I’m thinking of offering two swag bags for entries tagged with #sci140: one for the best ‘own research’ paper and one for the best summary of someone else’s, whether ‘historical’ or recent. The bag contains an F1000 mug, mints, light saber pen, magnifying glass, compass and the amazing F1000 combo laser pointer/pen/torch/stylus. And yes, enter in your native tongue, too. If I can translate it, it’ll be considered a valid entry!

There’ll be more updates during the week, and I’ll announce a winner on Monday. Good luck!
Read the rest of this entry »

Posted in Competition | Tagged: , , , , , | Comments Off on On a new publishing model-update

On a new publishing model

Posted by rpg on 9 February, 2010

UPDATE: Entries so far

Twitter, what is it good for? Hunh.

There’s been rather an interesting couple of posts over at the Scholarly Kitchen, recently. What am I saying? They’re all interesting. Anyway, Kent Anderson says that blogs are for fogies and David Crotty talks about ‘talking’ vs ‘doing’. Elsewhere on Nature Network we’re re-visiting the meme of why do we blog anyway? (to which I’m not going to contribute, myself having decided to do rather than talk about). You can look up the links yourself if you can be bothered.

Anyway, in the middle of a rather long and involved conversation, someone made a throwaway comment on David Crotty’s post. Then I thought it might be fun to see if I could write a scientific paper in 140 characters.

“Clned gene _cancer_. KO in Ms. Ms dead. Cure cancer.”

But why stop there? Here’s a challenge for you.

Your task is to re-write a scientific paper, a real, peer-reviewed and published one, in 140 characters. Twitter it with the hashtag #sci140 so we can track them (OK, so that’s 7 characters you’ve just lost but no one said it would be easy). You can do this as many times as you like, as many papers as you like, and it would be nice if they were your own, but they don’t have to be. I’ll see if I can get some f1000 swag for what I deem to be the best entry.

Go for it.
Read the rest of this entry »

Posted in Friday afternoon, Literature | Tagged: , , , , , , | 2 Comments »

On the run-04Feb10

Posted by rpg on 5 February, 2010

What was that? I think it was the sound of a week flashing past.

I keep saying things like “We’ve got a brand new website… but you can’t see it yet.” This must be quite frustrating. Truth is, the dev team are working very hard (and specs have changed and changed again—but let’s not go there now) and a lot of stuff has to come together all at once. There’s actually no point showing you what we have at the moment because it’d all “ignore that, we’re changing it” and “the design is going to be different than this” and “oh, yeah, we know about that bug”.

But I can tell you that the new search is very funky and we all like it, and that the new design is very spiffy (hang on, I did that already). On Monday we’re going to work out once and for all what we can deliver and work to that. So far, the ‘definite’ list contains the new design (both what it looks like and functionally), the improved search, comments on evaluated articles and RSS. There are a heap of other behind-the-scenes changes too. Then after we go live we can add on all the other things that are on the backlog, so you will see new things appear as we keep building and tweaking and rolling out new features.

I spent some more time on our journal rankings this week. The critical thing appears to be the timing: as I’ve said before, most of our evaluations are published quite quickly after the original article appears. We get around 90% of all evaluations within about three months of the publication date. So what we want to do, for yearly journal league tables, is capture as many as possible while making our stats as up-to-date and relevant as possible. The issue is that if we took April, say, as the cut-off for the previous year we’d end up giving the journals that publish more stuff towards the beginning of the year an unfair advantage. So we’re going to implement a rolling cut-off, with a provisional ‘current’ ranking and publish the official f1000 stats somewhere around May each year, which gives us four months to collect evaluations for each original article.

However, the big news this week is that we welcomed Sarah Greene into the office. This is part of the move to bring f1000 and The Scientist closer together: f1000 is going to start seeding The Scientist‘s scientific content, and use it to help build a community around the service.

As part of this, my own role is changing. I’m going to move away from web development (although I’ll still have input into the design and user experience), which will free me up to be more editorial/journalistic. I’ll still be running the Twitter feed and Facebook page and wittering about things that catch my eye in f1000 (perhaps even more so). There’ll also be the ‘special projects’, such as the rankings, federated comments and various research collaborations. I guess Eva will still be wanting me to make logos for her too.

And finally

The late pick-up of the disenchantment of a small number of researchers with the peer review process is still making waves this week. Cameron Neylon gives his own take on the matter at his blog. I’m not at all sure that I agree with his analysis, having had my own manuscripts subject to both what I might call ‘good’ and ‘bad’ review. I think that too many people view peer review as a stamp on the ‘rightness’ of the paper, rather than a technical check that the experiments and controls are done correctly and that the literature has been read.

Cameron has also been having a go at Nature Communications. The argument hinges on the Creative Commons licences they ask authors to choose. You can sign up and join the conversation at Nature Network.

With that, have a great weekend. And sorry, no cytoskeletal porn this week. Maybe next time.


Posted in Friday afternoon, Website | Tagged: , , , , , , , | Comments Off on On the run-04Feb10

I’m a believer

Posted by rpg on 3 February, 2010

I took my daughters round the new Darwin Centre at the Natural History Museum last year. Because we happen to be friends, I managed to persuade the incomparable Karen James, of The Beagle Project fame, to show us behind the scenes.

Calamari at the NHM After that I took the girls into the ‘Cocoon‘, a huge butterfly egg-type structure that contains 20 million or more specimens, with superbly-done exhibits and displays. The Cocoon lets visitors see into the workings of the Museum; quite literally, because one side of the egg cuts away into the research labs (where Karen works). As we walked around we came across a video display of Karen herself, talking about the process of publishing science, how you write and revise a manuscript and send it off to the dreaded ‘peer review’.

Karen did a splendid job explaining the process: how other scientists in your field look at your work and—in an ideal world—check that you’ve done the work right and that you’ve cited all the relevant literature; that the manuscript is sound. She also conveyed, far too convincingly, the heartbreak of having a treasured manuscript rejected!

The process of science for the last three hundred and more years is based on peer review. Other scientists check your work and say yup, that looks OK or no, you need to do this other experiment or read these papers. Unfortunately some reviewers (and I stress, these people are peers, that is they are your equals; not some shadowy cabal curating or judging Science from on high) do seem to hold personal grudges, or have strange agendas, or simply not be very good. (And yes, sometimes your own work is pants and should be rejected. Deal with it.)

This leads to people making wide-ranging and inflammatory statements such as “peer review is broken.” Some of them even write letters about it (as reported by the Beeb, six months later). This leads to calls for making the peer review process ‘open‘; i.e. publishing the correspondence between the reviewers and the editors, and maybe even removing the anonymity part.

We’ve been here before (haven’t we always?):

All editors have seen curt, abusive, destructive reviews and assumed that the reviewer would not have written in that way if he or she were identifiable. Openness also links accountability with credit. One important defect of closed review is that reviewers don’t receive academic credit. Finally, openness should eliminate some of the worst abuses of peer review, where reviewersunder the cloak of anonymitysteal ideas or procrastinate

BMJ, 1999

but it’s not clear to me, and indeed the results of those BMJ studies tend to bear me out, whether this is really a problem, or whether the perception is far worse than the reality. And I’ve had a paper that took nearly two years to get published.

I remain to be convinced that peer review is broken. The idea of some sort of clique quashing acceptance of manuscripts isn’t that far-fetched, pace Philip Campbell. Strong editors will get round it, but in some fields it’s quite possible for one or two individuals to make it really difficult to get anything published. This tends to be self-limiting though: in the extreme case, the field simply dies. In my experience of this, the community know who those people were, so open peer review probably wouldn’t improve matters. Double-blind will not work because it’s going to be reasonably trivial to figure out is the author. People will still sit on manuscripts, and we already know who does this when it happens.

Making the reviewers’ comments, signed or not, public might not ‘improve’ peer review: however I do believe it has value. There is virtually no training in reviewing papers, and if young post-docs and grad students could see a wide range of reviews of many different papers, surely that can only improve their reviewing skills? Maybe it would even serve to make the first submission better if nascent authors were to look at reviews in their field, and discover the common mistakes?

Having said that, I am keen to see greater accountability. On a personal level, I wouldn’t write anything I wouldn’t put my name next to, and I don’t actually see why peer review should be any different. At f1000, for example, we already put our Faculty Members’ names on their evaluations, and we call this ‘post-publication peer review’. (Our motives are slightly different of course: we’re saying that you should take notice because of who’s writing them.)
The EMBO Journal has been experimenting with publishing reviews of accepted manuscripts for a year now. I was amused to find that when I clicked on one of the reviews at random, it was a paper about my old friend talin. And it’s from Mark Ginsberg and Iain Campbell, FRS, with both of whom have I coauthored papers.

Small world.

Posted in Literature | Tagged: , , | 6 Comments »

Private investigations

Posted by rpg on 20 January, 2010

One of the really great things about science is its potential for self-correction. If you have an hypothesis, a result (strange or otherwise), a set of data, it can be tested by anyone. This is encouraged, in fact: when you publish you’re not just saying ‘look how clever I am’ but also ‘here’s something new! Can you do it too?’. This philosophy is diametrically opposed to that behind Creationism, say; or homeopathy. In those belief systems whatever the High Priest says is of necessity true, and experiment must bend around them until the results fit.

This means that, in science, a finding or publication that people get very excited about at the time can be shown to be wrong—either through deliberate fraud, experimental sloppiness (although the boundary between the two can be fuzzy) or simply because we’re as scientists wiser now than we were then. This happens, and it’s normal and part of the process. We should welcome it; indeed, my friend Henry Gee has claimed that everything Nature publishes is wrong, or at least provisional.

So what we have to do is be completely open about this, no matter how embarrassing it is for the journal that published the work in the first place.

You know where I’m going with this, don’t you?

It was Derek Lowe who first alerted me to a paper published in Science last year, with the ome-heavy title Reactome Array: Forging a Link Between Metabolome and Genome. This was flagged as a ‘Must Read‘ (free link) back in November, because according to our reviewer Ben Davis

If this worked it could be marvellous, superb.

However, as Ben said in his evaluation,

this work should be read with some caveats. Try as we might, my group, as well as many colleagues, and I have tried to determine the chemistry described […] In my opinion, this is a work that deserves a “Must Read” rating and I strongly encourage the reader to read the source material and reach their own conclusions.

And as Derek points out, Science published an ‘Editorial expression of concern‘, noting a request for  evaluation of the original data and records by officials at the authors’ institutions, as well as mentioning it on their blog. Heavy. Immediately I saw this, I let our Editorial team know we might have a problem and we published a note to warn our readers that the work described in the paper was suspect.

Today we published a dissent to the evaluation from Michael Gelb, who says

There are many reactions shown that seem unusual and controversial […] My colleagues and I have tried to decipher the chemistry shown in Figure 1 of the main text and in the supplemental material. Many of the indicated reactions seem highly unlikely to occur, and the NMR data showing that some of the structures that were made are confusing and controversial.

We’ve also published a follow-up from Ben:

I agree wholeheartedly with the sentiments expressed in the Dissenting Opinion. The chemistry presented in this paper and in the online SI has varied in its description and content worryingly over the last 2 months.

and, rather tellingly,

as yet no chemical samples or key reagents have yet been made generally available.

(One of the usual conditions of publishing in reputable journals is that you make reagents available to other scientists, so that they can repeat your work. Failing to honour this commitment is not playing to the rules.)

It’ll be interesting to see when, not if, the original paper is retracted; and by whom.

And this, people, is the self-correcting wonder of science. Remember this, next time someone starts rabbiting about scientific conspiracies, or sends you their new theory of general relativity, or anything else that sounds crazy. It probably is.

Read the rest of this entry »

Posted in Journals, Literature, Science | Tagged: , , , , , , , | 3 Comments »

Our brains are the original energy savers

Posted by stevepog on 16 October, 2009

“Hippocampal mossy fibers” may sound like abnormal skin growths on a rare African mammal (to me at least) but they are actually axons of granule cells in the hippocampus which deal with different types of bevaviour such as spatial learning.

The region is also where scientists have discovered the brain’s ability to be energy efficient, detailed in a paper given a high rating by two of our F1000 Biology Faculty Members.

Environmentalists may not be initally be excited by the brain’s power saving nature but the potential is there for the concepts gained to be applied to tech products, as soon as someone with the medical and tech smarts can make the link.

It got me thinking (and I often branch off onto tangents so try to stay with me), what function of the body would be a brilliant thing to replicate? The brain is an obvious one and I won’t even consider genitalia as an answer but what other organs/systems could be reproduced in technological form to make daily life easier?

Similar to the brain’s energy saving ability, the body’s heating and cooling system would be a perfect one.  Airconditioning units may be getting more environmentally-friendly and better at regulating room temperature but as good as our office A/C system is, after about half an hour it leaves me either too hot or too cold.

A quick Google search comes up with thousands of personal heating and cooling products but despite the marketing hype, there’s still nothing out there that really replicates the body’s ability to adapt to the weather conditions.

Over to you 🙂

Posted in f1000, Medicine | Tagged: , , , | Comments Off on Our brains are the original energy savers

How do we make sense of peer review?

Posted by stevepog on 15 September, 2009

As a follow-up to yesterday’s  tweet about the Sense About Science’s sensibly titled Peer Review Survey, a few thoughts. Out of the press reports on the Elsevier-funded study I’ve scanned though, a common phrase seemed to “no one has come up with a better method than peer reviewing, so we’re stuck with it”.

My colleague rpg has much more practical knowledge of the peer review process than I do, so as a relatively lay person I firstly wondered what the fuss was about. I know it’s of utmost importance to academics (and their funders) to have their papers appear in peer-reviewed journals.

And scientists who work outside the mainstream can miss out on exposure in their so-called top tier journals. But one of the great things that F1000 does is feature brilliant papers in lesser known publications, so they don’t get forgotten in the global soup of academic research.

Mark Henderson at The Times blogged on the strengths and weaknesses of peer reviewing, particularly the point that reviewers shouldn’t need to have their names known for peer reviewing to be an acceptable practice. Not an idea we subscribe to for business purposes at F1000 but he’s right in suggesting that young researchers may be scared off if they knew their name would appear alongside their reviews. Then again, as a former journalist I took that risk every time my byline appeared next to a contentious news story. If you want the knowledge and experience, you have to take some risks.

Some interesting stats in the survey that many have commented on related to plagiarism and fraud. While 81 per cent say that peer review should detect plagiarism and 79 per cent say that it should prevent fraud, only about 35 per cent say it is capable of doing both. I was discussing this topic with two recently graduated medical communications colleagues last week and they both said it would be an extremely difficult issue to police.

If a reviewer needed to have an in-depth knowledge of all major papers written on a particular topic, including remembering vital chunks of text that are likely to be plagiarised, they would probably also be in the running to replace Ben Pridmore or Brad Williams in the memory stakes.

Posted in Communication, f1000, Literature | Tagged: , | Comments Off on How do we make sense of peer review?