The world of peer review is badly broken, at least according to the blogosphere. Anonymity of reviewers could lead to poorer reviews because reviewers remain unaccountable. Consequently, fully open reviews are used at some journals such as F1000Research. In addition to the reviews being open, F1000Research makes submitted papers fully available to readers, which can speed up communication of the science.
There are other potential benefits of open reviews. If readers could see all aspects of reviews and the responses, they might understand the science better by seeing the entire review process. And reviewers might write more careful reviews, although if anonymity influences how you write reviews you should consider what that means. I’m sure plenty of other good reasons for fully open reviews exist. I don’t claim expertise.
However, there are benefits of anonymity for reviewers. Vindictive scientists exist. Let’s consider the public attempt by a senior editor at a prestigious journal to punish Dr Isis, a pseudonymous blogger who had criticised him. Not all scientists are vindictive; similar criticisms can lead to different (and more humble) responses in different people. And even those seeking revenge are unlikely to be vindictive all the time. But it seems entirely possible that some authors might seek revenge against a reviewer who wrote an unfavourable review. And their revenge might not be public, or even known to the victim.
Another consideration – the identity of authors might also influence reviews. At least some aspects of peer review at some journals depend on track record. Essentially, hot shots can get an easier ride. And reviews might even be gender-biased (see also this and this), or unfairly biased in other ways. One response would make reviews double blind; the identities of the author and reviewers remain concealed to all but the editors.
Interestingly, these two very different responses (less anonymity versus more) are seen as valid responses to (admittedly different) concerns about the current system of peer review. I’m not sure of the best model of peer review; the evidence is not extensive.
I don’t have a firm stance, but I have some concerns about open peer review. Reviewers might modify open reviews based on the prestige and power of the authors of the paper being reviewed. Proponents of open peer review might argue that a good review will be publicly validated regardless of who authored the reviewed paper. However, reviewers can choose various ways of writing a negative review, ranging from the rude “This is the worst piece of pseudo-science ever submitted to a journal” through to the meek “I have some reservations about the authors’ analysis and interpretation”. And even if written in a perfectly collegial tone, reviewers choose which aspects to emphasize. But if open reviews tend to differ from closed reviews, I can’t see how the identity of the author would not influence reviewers, even subconsciously.
Further, what happens when reviewing a paper that criticizes another? Clearly, the reviewer will disappoint at least one party. If both the critique and the article being criticized have errors, then the reviewer might even achieve the honour of annoying both parties! The reaction to such reviews will depend on how the review is written, but also at least in part on the personality of those being criticized. Only some of the reaction is in the hands of the reviewer, which makes me uncertain about open peer review. Some authors will occasionally take a review personally even when it is written in good faith.
While I’m uncertain about the best model for peer review, I’m willing to try the various options. So I agreed to do an open peer review for the journal F1000Research, in which submitted articles and the reviews are public. Others can also comment, even on the reviews.
The article that I reviewed criticized a paper in the journal Science by Bradley Efron, one of the world’s leading statisticians. I consciously considered my role as an impartial arbiter of two competing pieces of work. I would have done that anyway even if the review was anonymous and concealed. However, if I were a more junior academic, I doubt I would have been as confident advertising an error by a world-leading scientist, even if in this particular case, the only information I have suggests that Bradley Efron is certainly not vindictive!
As it turned out, my review was longer than the article being reviewed! In fact, my review almost became a comment in itself. Who reviews such peer reviews? The authors certainly get a chance to respond publicly. And anyone can comment on my review. And having just checked, it turns out Aaron MacNeil did – thanks for the comment Aaron.
What do you think? Is this a good model for science? Should someone review my peer review?
But more importantly, what are your opinions of the best models for peer review? And particularly, do you have evidence to support your opinion?
Edit: I meant to link to this blog post – clearly, writers use pseudonyms for many reasons. Similarly, there are likely to be many reasons why reviewers might wish to remain anonymous.
Edit 3: Another particularly relevant tweet (this one by Luca Borger) noting the importance of editors in moderating peer review. They can (and should) help iron out some of the major problems of peer review, but they won’t always be successful.
Edit 4: F1000 Research have written a blog post about the #SixWordPeerReview hashtag. They compare the short, pithy and generally scathing mock reviews on Twitter to my rather long one. While I fair well in the comparison (except in some aspects – I don’t think my review competes for laughs), I don’t think the hashtag #LongWindedPeerReview will ever quite take off on Twitter. Luca Borger suggested some alternatives, including #MOOR (Massive Open Online Review). Which, of course, leads to MOOR, MOOR, MOOR. As I said in the tweet – I apologize in advance.
Edit 5: A new paper in F1000 Research that asks reviewers to sign all their reviews, generated some discussion on Twitter. The conclusion from Twitter seems to be that we’re not ready for 100% open peer review.