Faculty of 1000

Post-publication peer review

Archive for the ‘Indicators’ Category

On the run-29Jan10

Posted by rpg on 29 January, 2010

Vitek quotes a Polish proverb,

If you’re going to fall off a horse, make it a big one.

In that vein, take a look at this graph (don’t look too closely; it’s deliberately a tiny bit obscure):Graph of ffJ and Journal Impact factor

What I’ve been doing this week is mostly hacking away in Perl at some of the information in our database. As you may know, each evaluation in f1000 has a score associated with it, based on the rating given an article by Faculty Members. We’ve redone the scoring and I’ve worked out a way of combining those scores, or ‘article factors’ as we’ve taken to calling them, for each journal that is represented at f1000. This gives us a ‘journal factor’, ffJ. It’s our answer to the journal Impact Factor, in fact; and the graph above is the top 250 journals according to our ratings (in blue) with the appropriate Impact Factor in red.

You’ll notice right away that there isn’t a one-to-one correlation, and of course we’d expect that (the Impact Factor has serious problems, which I’ve talked about previously). I’m currently analysing our data a bit more deeply, and I’ll be writing a paper with that analysis, as well as talking about it at the March ACS conference in San Francisco.

Last Friday evening I went down the Driver with a couple of the dev team and a bunch of people from places as varied as BioMed Central, Nature Publishing Group, Mendeley and Mekentosj. We talked about what we’re variously calling cross- or federate-commenting. On the whole we’ve decided it’s a good idea, and we simply have to figure out how to do it. What this implies of course is that we’re actually going to allow user comments at f1000—and indeed that’s the plan. I’m looking forward to rolling out this functionality to you, not least because when people want to raise questions about articles evaluated on f1000, they’ll be able to.

While we’re on the mythical new site, we asked another web designer what they could come up with for us. And for the first time, all of us who saw the design liked it. So hopefully we’ll be able to get that implemented real soon now and I’ll be able to start showing you what you’re going to get, you lucky people. (Rumours that someone said “It looks like a proper website!” are completely unfounded.)

Interesting reviews

A couple of things you may have missed.

First, the (possible) mechanism behind photophobia in migraines. Turns out that people who are functionally blind, but sensitive to circadian input and pupillary light reflexes are susceptible to photophobia. Work in rats published in Nature Neuroscience implicates a (previously uncharacterized) multisynaptic or heavily networked retinal path.

In Biology, the problem of de novo assembly of DNA sequence reads into sensible contigs from massively parallel sequencing technologies has been addressed. This, if it works, would bring exciting concepts such as personal genomics that little bit closer. The paper is in Genome Research (subscription required) and you can read the evaluation for free.

And finally

Faculty of 1000  is big in Italy—or at least on Facebook. My post on the recycling of integrins drew an excited response from one Grazia Tamma, who was then mocked mercilessly by her brother!

Hang in there, Grazia; science is great and the cytoskeleton rocks.


Posted in Friday afternoon, Indicators, Metrics | Tagged: , , , , | 4 Comments »

How many more times?

Posted by rpg on 14 September, 2009

…what dreams may come
When we have shuffled off this mortal coil,
Must give us pause

Thomson, in a commentary in the Journal of the American Medical Association, reckon there ain’t nowt wrong with the Journal Impact Factor:

The impact factor has had success and utility as a journal metric due to its concentration on a simple calculation based on data that are fully visible in the Web of Science. These examples based on citable (counted) items indexed in 2000 and 2005 suggest that the current approach for identification of citable items in the impact factor denominator is accurate and consistent.

Well, they would say that.

And they might well be right, and you and I and Thomson Reuters might argue the point endlessly. But there are a number of problems with any citation-based metric, and a pretty fundamental one was highlighted (coincidentally?) in the same issue of JAMA.

Looking at thre general medical journals, Abhaya Kulkarni at the Hospital for Sick Children in Toronto (shout out to my friend Ricardipus) and co. found that three different ways of counting citations come up with three very different numbers.

Cutting to the chase, Web of Science counted about 20% fewer citations than Scopus or Google Scholar. The reasons for this are not totally clear, but are probably due to the latter two being of wider scope (no pun intended). Scopus, for example, looks at ~15,000 journals compared with Web of Science’s ten thousand. Why? The authors say that Web of Science ’emphasized the quality of its content coverage’: which in English means it doesn’t look at non-English publications, or those from outside the US and (possibly) Europe, or other citation sources such as books and conference proceedings. And that’s before we even start thinking about minimal citable units; or non-citable outputs; or whether blogs should count as one-fiftieth of a peer-reviewed paper.

Presumably some of the discrepancy is due to removal of self-cites, which strikes me as being just as unfair: my own output shouldn’t count for less simply because I’m building on it. It’s also difficult to know how to deal with the mobility of sciences: do you only look at the last author? or the first? I don’t know how you make that one work at all, to be honest.

That aside, I think curation of citation metrics is necessary: Kulkarni et al. report that fully two percent of citations in Google Scholar didn’t, actually, cite what they claimed to. That is a worrying statistic when you realize that people’s jobs are on the line. You have to get this right, guys.

But it’d be nice if we could all agree on the numbers to start with.

Read the rest of this entry »

Posted in Indicators, Journals, Literature, Metrics | Tagged: , , , , , | 3 Comments »

Big Bad John

Posted by rpg on 28 August, 2009

I’ve been remiss.

I should have talked a bit about the events of last Saturday: truth is I was struck by a stomach bug on Tuesday night and have been a little bit out of things. If you’re interested, there is a video of the ‘Fringe Frivolous‘ event of the Friday evening and lots and lots of photos on Flickr.

Martin Fenner has summarized all the blog links he could find, in a kind of citation-stealing Annual Review way. Yeah, we talked about indicators and metrics in the session with Victor and Ginny Barbour (PLoS), saying among other things that usage data and network metrics and our own F1000 factor aren’t necessarily replacements for the journal impact factor: rather they’re all complementary, and tell you diffirent things.

I’ll actually be talking about that a bit more at two upcoming meetings. The first is Internet Librarian International in London, 15-16 October; the second is the XXIX Annual Charleston Conference, 4-7 November in Charleston SC. That’s actually going to be a hell of a trip as it’s my youngest’s birthday on 5th. She’ll be ten: I’m going to miss it but there should be fireworks on the Friday or Saturday night that we can see.

Interestingly, a chap from Thomson collared me on Saturday after our session. As someone remarked to me later, this was quite a scoop: apparently Thomson don’t usually bother with small fish: I wonder if we spooked them?

Talking of which, there’s a fascinating paper about the ‘Matthew Effect’ in arXiv (‘the arXiv’?), The impact factor’s Matthew effect: a natural experiment in bibliometrics. Turns out (surprise, surprise) that papers published in high impact factor journals garner twice as many citations as the identical paper in a low IF journal. I don’t know if that’s because more people read high IF journals or because there truly is the impression that papers in them must be better, or what. Either way, I’d just like to say…

broken glass

broken glass

Posted in Indicators, Metrics | Tagged: , , , | Comments Off on Big Bad John

Where the streets have no name

Posted by rpg on 4 August, 2009

Alejandro brings my attention to ScienceWatch’s list of most-cited institutions in science.

This is the list of the ‘top’ twenty institutions out of just over four thousand. For some value of ‘top’, he says snarkily. Now, we know there are serious problems with citation metrics, but essentially it’s all we’ve got to go on, so it’s not a bad list.

The Most-Cited Institutions Overall, 1999-2009 (Thomson)

Rank Institution Citations Papers Citations
Per Paper
1 HARVARD UNIV 95,291 2,597,786 27.26
2 MAX PLANCK SOCIETY 69,373 1,366,087 19.69
3 JOHNS HOPKINS UNIV 54,022 1,222,166 22.62
4 UNIV WASHINGTON 54,198 1,147,283 21.17
5 STANFORD UNIV 48,846 1,138,795 23.31
6 UNIV CALIF LOS ANGELES 55,237 1,077,069 19.5
7 UNIV MICHIGAN 54,612 948,621 17.37
8 UNIV CALIF BERKELEY 46,984 945,817 20.13
9 UNIV CALIF SAN FRANCISCO 36,106 939,302 26.02
10 UNIV PENN 46,235 931,399 20.14
11 UNIV TOKYO 68,840 913,896 13.28
12 UNIV CALIF SAN DIEGO 40,789 899,832 22.06
13 UNIV TORONTO 55,163 861,243 15.61
14 UCL 46,882 860,117 18.35
15 COLUMBIA UNIV 43,302 858,073 19.82
16 YALE UNIV 36,857 833,467 22.61
17 MIT 35,247 832,439 23.62
18 UNIV CAMBRIDGE 43,017 811,673 18.87
19 UNIV OXFORD 40,494 766,577 18.93
20 UNIV WISCONSIN 50,016 760,091 15.2

Or is it?

Because as you know, we give the articles evaluated at F1000 a score. And it has not escaped our notice that once you start doing such a thing, you can start asking interesting questions. Admittedly we only look at biology and medicine (so far…), but according to this Excel spreadsheet I’ve just opened we have over five thousand unique institutions in our database. Hmm… I wonder if we might be doing anything with that?


And talking of authors I’d like to take this opportunity to shout out to my friend Åsa, whose recent work on inhibiting protein synthesis in secondary pneumonia was evaluated on F1000 Medicine (and who might one day get a nonymous blog cough).

Posted in Competition, Indicators, Journals | Tagged: , | 4 Comments »

More than a number (in my little red book)

Posted by rpg on 31 July, 2009

Shirley Wu kicked off an interesting conversation on Friendfeed yesterday, reporting on a conversation that questioned the ‘quality’ of our old friend PLoS One. Now there’s a debate that’s going to go round and round, possibly generating rather more heat and noise than useful work.

The conversation, thanks to Peter Binfield, then turned onto article level metrics as a way of assessing ‘quality’, and that’s when I got interested.

Euan Adie wants to know,

Do [Nascent readers] think that article downloads stats should be put on academic CVs?

and provides a very cogent argument as to why paying too much attention to single article metrics is not necessarily a good idea.

I’m not saying that download stats aren’t useful in aggregate or that authors don’t have a right to know how many hits their papers received but they’re potentially misleading (& open to misinterpretation) and surely that’s not the type of data we want to be bandying about as an impact factor replacement?

Now, I’m paying attention to this not just because Faculty of 1000 is a stakeholder in this sort of discussion, but also of  Science Online London. This is the successor to last year’s blogging conference, and this year I’m on the organizing committee.  Together with Victor Henning of Mendeley and Ginny Barbour of PLoS I’m running a session on… article level metrics.

Which means it’s going to get interesting, as there are strong feelings on both sides—and maybe I should trawl through the attendee list and make sure they are all briefed.

Bring it on.

I’m going to say that I agree with Euan that download stats are a bit suss, actually. Just like citations, they don’t tell you how the article is being used; they don’t tell you if the user (or the citer) thinks this paper is good or bad. As Euan puts it,

A download counter can’t tell if the person visiting your paper is a grad student looking for a journal club paper, a researcher interested in your field or… somebody who typed in an obscure porn related search that turned up unconnected words in the abstract.

(That reminds me of a story my boss in Cambridge told me. He was an editor on a particular well-respected cell biological journal, and one day they found that one article was getting massively hit, and it was all search engine traffic. On examining the referrer stats it turns out that the paper was all about our friend C. elegans.  The cause of all the traffic was not a sudden upsurge in global nematode research, but rather the juxtaposition of the words ‘sex’ and ‘anal’  in the abstract. Quite.)

The problem with the Impact Factor, and also with article level metrics, is that there is no indicator of quality. About the only automated system that has any hope of doing this is the kind of network analysis that Johan Bollen is doing. Briefly, these network maps show you how people are using papers by following a user’s click history.

funky network connections

funky network connections

In any event, I suspect that people are going to have to learn to be more sophisticated in their use of metrics/indicators, and the tyranny of the impact factor will be shaken. It starts here.

Posted in Indicators | Tagged: , , , , | 2 Comments »

Somewhere over the rainbow

Posted by rpg on 27 July, 2009

Somewhere in the depths of PLoS One an article lurks…

Liz Allen and her friends at the Wellcome performed an interesting study on papers that came out of labs that were at least partially funded by Wellcome grants. What they did was to figure where each of the nearly 700 papers ended up being published, and then looked at bibliometric indictors (cough impact factors cough).

Nothing new there, really. The clever bit was to persuade a ‘college’ of experts (I hate that word, sorry) to review the papers, and then compare this with the bibliometric indicators… and with the Faculty of 1000. Funnily enough, highly-rated papers did not necessarily get cited highly (so the Impact Factor doesn’t help here), but the F1000 ratings agreed pretty well with their college of experts, and was also able to predict citation rates to some extent.

We were pretty stoked about this paper, I can tell you: and we hope to have a chat with Liz next month and show her some interesting stuff we’ve been up to. It plugs directly into this:

Tools that link expert peer reviews of research paper quality and importance to more quantitative indicators, such as citation analysis would be valuable additions to the field of research assessment and evaluation.

I’ll write more about that nearer the time. Or even, given the bonkers-mad workflow I’ve got going on, after the time. Until then, you can check out the more in-depth analysis and a fascinating discussion over at my personal blog.

Posted in Indicators | Tagged: , , , , | 6 Comments »