Anonymity and peer review

The world of peer review is badly broken, at least according to the blogosphere. Anonymity of reviewers could lead to poorer reviews because reviewers remain unaccountable. Consequently, fully open reviews are used at some journals such as F1000Research. In addition to the reviews being open, F1000Research makes submitted papers fully available to readers, which can speed up communication of the science.

There are other potential benefits of open reviews. If readers could see all aspects of reviews and the responses, they might understand the science better by seeing the entire review process. And reviewers might write more careful reviews, although if anonymity influences how you write reviews you should consider what that means. I’m sure plenty of other good reasons for fully open reviews exist. I don’t claim expertise.

However, there are benefits of anonymity for reviewers. Vindictive scientists exist. Let’s consider the public attempt by a senior editor at a prestigious journal to punish Dr Isis, a pseudonymous blogger who had criticised him. Not all scientists are vindictive; similar criticisms can lead to different (and more humble) responses in different people. And even those seeking revenge are unlikely to be vindictive all the time. But it seems entirely possible that some authors might seek revenge against a reviewer who wrote an unfavourable review. And their revenge might not be public, or even known to the victim.

Another consideration – the identity of authors might also influence reviews. At least some aspects of peer review at some journals depend on track record. Essentially, hot shots can get an easier ride. And reviews might even be gender-biased (see also this and this), or unfairly biased in other ways. One response would make reviews double blind; the identities of the author and reviewers remain concealed to all but the editors.

Interestingly, these two very different responses (less anonymity versus more) are seen as valid responses to (admittedly different) concerns about the current system of peer review. I’m not sure of the best model of peer review; the evidence is not extensive.

I don’t have a firm stance, but I have some concerns about open peer review. Reviewers might modify open reviews based on the prestige and power of the authors of the paper being reviewed. Proponents of open peer review might argue that a good review will be publicly validated regardless of who authored the reviewed paper. However, reviewers can choose various ways of writing a negative review, ranging from the rude “This is the worst piece of pseudo-science ever submitted to a journal” through to the meek “I have some reservations about the authors’ analysis and interpretation”. And even if written in a perfectly collegial tone, reviewers choose which aspects to emphasize. But if open reviews tend to differ from closed reviews, I can’t see how the identity of the author would not influence reviewers, even subconsciously.

Further, what happens when reviewing a paper that criticizes another? Clearly, the reviewer will disappoint at least one party. If both the critique and the article being criticized have errors, then the reviewer might even achieve the honour of annoying both parties! The reaction to such reviews will depend on how the review is written, but also at least in part on the personality of those being criticized. Only some of the reaction is in the hands of the reviewer, which makes me uncertain about open peer review. Some authors will occasionally take a review personally even when it is written in good faith.

While I’m uncertain about the best model for peer review, I’m willing to try the various options. So I agreed to do an open peer review for the journal F1000Research, in which submitted articles and the reviews are public. Others can also comment, even on the reviews.

The article that I reviewed criticized a paper in the journal Science by Bradley Efron, one of the world’s leading statisticians. I consciously considered my role as an impartial arbiter of two competing pieces of work. I would have done that anyway even if the review was anonymous and concealed. However, if I were a more junior academic, I doubt I would have been as confident advertising an error by a world-leading scientist, even if in this particular case, the only information I have suggests that Bradley Efron is certainly not vindictive!

As it turned out, my review was longer than the article being reviewed! In fact, my review almost became a comment in itself. Who reviews such peer reviews? The authors certainly get a chance to respond publicly. And anyone can comment on my review. And having just checked, it turns out Aaron MacNeil did – thanks for the comment Aaron.

What do you think? Is this a good model for science? Should someone review my peer review?

But more importantly, what are your opinions of the best models for peer review? And particularly, do you have evidence to support your opinion?

Edit: I meant to link to this blog post - clearly, writers use pseudonyms for many reasons. Similarly, there are likely to be many reasons why reviewers might wish to remain anonymous.

Edit 2: There’s some commentary about this on Twitter – too many tweets to link to individually, but some conversations can be found via Laura Williams and JacquelynGill. Thanks to all.

Edit 3: Another particularly relevant tweet (this one by Luca Borger) noting the importance of editors in moderating peer review. They can (and should) help iron out some of the major problems of peer review, but they won’t always be successful.

Edit 4: F1000 Research have written a blog post about the #SixWordPeerReview hashtag. They compare the short, pithy and generally scathing mock reviews on Twitter to my rather long one. While I fair well in the comparison (except in some aspects – I don’t think my review competes for laughs), I don’t think the hashtag #LongWindedPeerReview will ever quite take off on Twitter. Luca Borger suggested some alternatives, including #MOOR (Massive Open Online Review). Which, of course, leads to MOOR, MOOR, MOOR. As I said in the tweet – I apologize in advance.

About these ads

About Michael McCarthy

I conduct research on environmental decision making and quantitative ecology. My teaching is mainly at post-grad level at The University of Melbourne.
This entry was posted in Communication and tagged , , , , , , , , , . Bookmark the permalink.

7 Responses to Anonymity and peer review

  1. Hi, it’s great to get some first-hand experience about alternative publishing systems.

    I wanted to ask one think that wasn’t clear to me from the post: does F1000 require you to reveal your name?

    Because it seems to me that your biggest concerns are actually not with the review being open (by which I understand accessible to other readers), but with the review being not single or double blind. I think I agree with this, of all possible combinations of open/non-open * single-blind/double-blined/signed, I would probably prefer open single-blind, with the encouraged option to sign if they want, and the option to do comments.

    I know biogeosciences do it that way http://www.biogeosciences.net/review/review_process_and_interactive_public_discussion.html , and PeerJ as well https://peerj.com/about/how-it-works/

    If more and more people sign, the possibility of suggesting reviewers, as mentioned above, would maybe need revaluation, as people that become know as “mild-hearted” reviewers through their signatures might get swamped with review request.

    On the other hand, if identities are known, editors could potentially even factor in the review history (I know experienced editors do that anyway), i.e. take the praise of someone who always loves the manuscripts a bit less serious than that of someone who tends to be rather critical otherwise.

    • Hi Florian – thanks for the comments.

      Yes, F1000 Research reveals names of reviewers. F1000 would like reviews to become recognized contributions to science by individuals.

      I like the idea of open yet anonymous reviews. In fact, I’d quite like to post the review comments on some of my papers, although whether journals and reviewers would be happy with that is unclear. Also, interactions between reviewers and authors in an open review process might be different if the reviewers remain anonymous.

  2. Pingback: Recommended Reading | January 2014 | Cindy E Hauser

  3. joselahoz says:

    Great post Mick. I would vote for double-blind reviews (ie anonymous authors and reviewers, names only known by editor), I still haven’t seen an argument against it that fully convinces me. It might be true that for some iconic data sets or studies, the authors could be guessed, but I think that in general it is still fairer than knowing the names of authors. Regarding the risks of reviewers not being anonymous, a middle ground solution is used in the Frontiers suite of journals (www.frontiersin.org): reviewers are anonymous while the review is taking place, but their names are disclosed for accepted papers, so that they can take credit and be accountable for their work. No system will be perfect, but I think this is a neat idea.

  4. Thanks for that post Mick. Good points well made.
    Not raised in your piece, or much anywhere else that I’m aware of, is the tangentially related matter of being asked by journals/editors to nominate reviewers. In some cases (eg PeerJ) one is asked to nominate potential editors as well. I find this a very curious practice, and given the general anonymity of the process I’ve had difficulty deciding sometimes whether my nominated reviewers actually took the gig (or were even contacted for that matter). I find the practice curious because (i) it’s not a terribly independent process, and (ii) surely the editor is a reliable judge of who to seek for reviews. I guess given the load of reviews that need to occur a bit of new blood or nomination of potential candidates from left field could be a good thing. I don’t have any difficulty understanding the request about who NOT to ask for a review (going back to your point about vindictiveness and revenge and all that jazz)!

  5. Tim Vines says:

    One key point about anonymous review: you’re only anonymous to the authors and the other reviewers. The editor knows exactly who you are, and if you’re a jerk to the authors the editor will judge you accordingly. Given that they’re probably a senior figure in your field who will be reviewing your grant applications and manuscripts, that might not be a great idea.

  6. One more benefit of anonymity. Younger researchers are less likely to be taken seriously by senior, well-established researchers (gender is likely important as well). A leading, senior researcher, can probably convince an editor that this newbie doesnt know anything and dismiss major criticisms. But if the author dont know who wrote the review, they are forced to carefully consider it because it might be written by a leading researcher in the field.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s