Under certain circumstances, the editors of journal A use ‘arbitrating’ reviewers. These reviewers advise an editor where, for example, an editor has split reviewer reports or a rebuttal to a decision that was based on split reviewer reports. This reviewer has sight of the other reviewers’ reports as he/she both evaluates the manuscript and assists the editor, through their advice, to arrive at an informed editorial decision. Historically, the use of arbitrating reviewers has arisen out of a discussion between an author and the handling editor, with their use being at the discretion of the editor. More recently, the use of arbitrating reviewers has become more routine and has not always involved a discussion with the authors concerned.
The journal’s editors are currently reviewing this peer review option, partly in response to an author’s complaint that arbitrating reviewers bias the peer review process. We would welcome the committee’s feedback on the use of arbitrating reviewers and to have their advice regarding any additional safeguards the journal should put in place when using these reviewers to maintain unbiased peer review. We will use this advice to formulate a new set of editorial guidelines on arbitrating reviewers.
This interesting case provoked much discussion. Most agreed that they would not use the term “arbitrating reviewers”. The third reviewer is providing extra information for the editor, who will then decide whether or not to accept the paper. So the editor is in fact the arbitrator, not the reviewer. By using the term “arbitrating reviewers”, it could send a confusing message to the author that the reviewer is the one making the final decision or “arbitrating” on the acceptance of the paper. The editor can keep the author informed of the process, but does not need to give details on how many reviewers are consulted.
The Forum advised that the editor should obtain consent from the reviewers to share their reports with a third reviewer. The Forum noted that there is no evidence that the editor is introducing bias, even by showing a third reviewer the previous reviews. All agreed that the editor needs to make the decision on what he thinks is best for his journal. The journal could, perhaps, audit the process and see if there is any evidence of bias (i.e. whether papers that undergo review by a third reviewer are more or less likely to be accepted than those that are reviewed by only two).
COPE's feedback was discussed at the journal’s annual editors meeting and alternatives to the arbitrating reviewer arrangement are being considering, such as changing the name used for these reviewers, or perhaps seeking advice on papers that have conflicting reports in a different way.