FORUM DISCUSSION TOPIC: comments please

Who “owns” peer reviews?

Two trends have recently come together within scholarly publication; open review, and the desire to give credit to reviewers (see also [1]).  At the convergence are organizations like Publons and Academic Karma who wish to openly acknowledge the work of peer-reviewers by recording, not only the amount, but also, in some circumstances, the content of individuals’ peer-review activity. 

Academics may view services like this as a way to regain control over their reviews and so may be keen to sign-up and provide their data.  Journals, on the other hand, often conduct confidential review processes and wish to restrict the sharing of comments exchanged during peer-review.

Emergence of these services therefore prompts a number of concerns and questions as to how best ensure author, editor, reviewer and journal interests are protected.

Questions

  • Does it violate confidential/blind peer-review to reveal reviewer comments even after publication?
  • What can journals do to make sure reviewer comments remain confidential?
  • How can reviewers ensure they are able to share and get credit for their work?
  • Can service providers work with journal editors, publishers and reviewers to help facilitate openness and transparency in peer review?

Reference
1. http://scholarlykitchen.sspnet.org/2015/06/17/the-problems-with-credit-for-peer-review/

This will be discussed at the start of the next COPE Forum on Wednesday 9 September. Please do leave any comments below, whether or not you are planning on joining the meeting

Comments

  • Posted by Wolff-Michael Roth, 27/8/2015 12.50pm

A peer review is a set of comments on a text; without that text, the comments on their own do not make much sense (in most cases). If reviewers were to start writing more extensive descriptions of the study so that their own comments make sense, then certainly the confidentiality appears to be violated, especially in communities where scholars know each other or, where the particulars of writing can be traced back to the author. (e.g., I have had reviews in the double blind process, where the reviewer writes "quintessentially «my name»").

  • Posted by Joaquim Armando..., 27/8/2015 1.03pm

Q: Does it violate confidential/blind peer-review to reveal reviewer comments even after publication?

A: YES, Unless one of two things happen: the comments are revealed with the consent of the reviewer. Even more so, regarding the reviewer identity. My Journal (Computers and Graphics) practices double-blind review. Disclosing the reviewers' identities without their express written consent would be no less than a breach of trust and promises

A: What can journals do to make sure reviewer comments remain confidential?

Q: Editors should be especially careful to sift through reviewers comments and remove anything that would disclose unwittingly the reviewer's identity. However, I have witnessed on occasion, reviewers voluntarily disclosing their identity to stress a point (e.g. why they had such an authoritative command on subject matter being reviewed)

Q: How can reviewers ensure they are able to share and get credit for their work?
A: Mostly by allowing their quality reviews to be made public after a paper gets accepted

Q: Can service providers work with journal editors, publishers and reviewers to help facilitate openness and transparency in peer review?
A: Not sure what is meant by this. If you mean services like ORCID providing a list of venues for which a given person has done peer-review, yes, that is a good thing, since ORCID is somewhat of an independent entity which can provide such information impartially and thus be trustworthy. Regarding recognizing OUTSTANDING individual reviews, we may have to stick with Journal/Editorial Board recognition (my Journal awards such certificates of outstanding performance on a regular basis)

  • Posted by Vidhya Iyer, 28/10/2015 6.31am

Some of the papers published by even "reputable publishers" are evidence of how difficult it has become for them to handle the submissions. It would be pointless to make it OK for them to publish reviewers' comments after they sift them for information that would reveal the reviewers' identity. Best to ask the reviewers to digitally sign a consent form in the beginning to state their approval to publish their comments and reviews and not leave it up to the publishers and editors to do more work when the work they already have is proving too much for them.

  • Posted by Liz Hamp-Lyons, 27/8/2015 1.17pm

Will someone at the meeting specify who will benefit from open reviews? How does it benefit the reviewer? The author/s? The discipline? Who might suffer?

  • Posted by David Riley, 2/9/2015 1.43pm

As a 20 year medical editor I have infrequently received salacious reviews when the process was blinded and reviewers seemed to feel free to "slash and burn". This has not happened with open peer review. I do accept that neither system, blinded or open peer review is without problems. I prefer to open reviews. The reviews names are not advertised to the author and there is of course the issue of what you do with the reviews after the article is accepted.

David Riley

  • Posted by Charon Pierson, 1/9/2015 11.28pm

I don't think that confidentiality and blinding are necessarily the same. A review can remain confidential (i.e., between the reviewer, editor, and author) even if the names of the reviewers and authors are revealed within the review process. Once a paper is published and comments about a confidential review (blinded or not) are posted on a public site, that seems to me to be a violation of confidentiality. If all parties agree, which I think could be difficult to implement, that review comments can be posted on publication review sites, then there is no breach of confidentiality. The process has to be transparent which seems to be where things fall apart. I think the question has to be who controls this whole process? If it is left to 3rd parties running these post-pub review sites there is potential for conflict of interest and breaches of confidentiality. If it is left to the publishers, who actually pay for and run the pre-pub manuscript review systems we all use, there has to be a transparent policy about who "owns" the reviews. I don't know that anyone has a consent form where reviewers must agree to transfer ownership/copyright of their review over to the journal or the publisher. I'm not even sure if that would be legal. One other point to consider is that circumstances have likely changed between the time a pre-pub review is done and the time the article is published and available for scrutiny from the scientific community (i.e., other research on the topic could have been published in this interim). It seems to me that a lot of this post-pub review takes place in Letters to the editor, brief comments, etc., within the scholarly journals community.

As an editor I frequently run a report for a reviewer who needs documentation of service as a reviewer, but I do not include any specifics about the manuscripts reviewed. I just provide details about quality and timeliness of reviews, which seems to be all academics need for their portfolios. I'm not sure what other kinds of "credit for their work" is required. Does this mean that reviewers need others to see how clever and smart they are in their reviews? And, who are these others that need to see how much work a reviewer does?

  • Posted by Sarah Kraak, 2/9/2015 10.18am

I agree with the comments I read above. I would also like to say that I don't understand what sense it makes to publish reviews. A typical case is as follows: Author submits manuscript with flaws, reviewers detects flaws, editor allows revision dealing with flaws, author revises remedying the flaws, editor accepts for publication only if the flaws have been dealt with satisfactorily. I don't see any reason for making this public, as the final published paper has (hopefully) remedied the flaws. I see the peer-review system as one helping each other to prevent flawed papers from being published. If flaws remain, or disagreements, these can be published as Letter to the editor and the like, as Charon Pierson above already mentioned.

  • Posted by Joaquim Armando..., 2/9/2015 12.44pm

Occasionally (every so often) I stumble upon reviewers comments that reveal a solid command of the matters under review, excellent scholarship and very careful articulation. I would find these comments worth publishing with the consent of the reviewer, for their credit.

  • Posted by Jane Alfred, 2/9/2015 2.02pm

A key question raised at the Royal Society's Future of Scholarly Scientific Communication earlier this year was this: who owns a reviewer's report? It was clear from the academics present that they believe they own the reviewer reports they write, and can decide what to do with them, providing they do not breach confidentiality. Publishers were perhaps more of the view that they had ownership over reviewer reports as a piece of confidential advice they'd commissioned on a particular paper. As Charon Pierson's comments
reflect, this seems to be a key question for the community to resolve. Do publishers explicitly state what their position of ownership is over reviewer reports? Should they? Both publishers and reviewers have certain rights and interests over reviewer reports and a good starting place might be to define what these are and how the interests of each party can be addressed while maintaining the confidentiality of peer review. Opening up peer review in various ways might also help to address some of the issues being raised. In terms of why researchers want credit for their work as a reviewer, I see no evidence from the work I do with the academic community that this is about proving how smart they can be. It has much more to do with having the important, and often time consuming, work they do as reviewers recognised and credited in some way, so it can formally contribute to their career progression as a researcher . This is particularly important for early career investigators, who often do not receive credit from their group leaders for the reviewing work they undertake on their boss' behalf. Perhaps this is something that COPE members could consider too - whether to have policies that enable collaborative peer review, with all contributing parties recognised and named, while maintaining confidentiality.

  • Posted by F.P. Glasser, 2/9/2015 3.15pm

The Journal of which I am Chairman of the editorial board wishes to reward its referees (who, if members of the Editorial board) receive a free subscription. But above and beyond that we are considering publishing annually a list of referees who have acted in the course of a year. Many of our younger referees would like to have authentication of their efforts as it provides written evidence of their contribution to scholarship. We would not plan to associate reviews with specific submissions nor would we state if their reviews led to acceptance/ rejection of the paper(s) under review.
I see little evidence that publication of the reviews themselves will be of benefit to anyone.
With respect to continuing service, we will discontinue using any referees who provide inappropriate reviews. These are read and assessed.

  • Posted by Dr. Akhilesh K...., 2/9/2015 5.09pm

I would like to suggest that the reviewers comments should be confidential and there should not be a disclosure of comments without their permission as it leads to breech of confidentiality. I am publishing an Journal (International Research and Publications in Medical Sciences IRPMS) and I would like to know that how we can motivate reviewers and how we can give credit to their work, so they can be our reviewers in future also.

  • Posted by Anonymous, 2/9/2015 6.28pm

This is a very important topic. First, it seems giving public credit to reviewers is something that's being imposed on the scientific community by bureaucrats looking for another easy number to replace a researcher's reputation. Like the use of the IF to rate individual articles, this is something that should probably be resisted. Even in blinded review, editors know who the good reviewers are, and the good reviewers are in turn rewarded with trust, reputation, and editorial positions. Being a shoddy reviewer undermines your reputation as an academic.

The questions:

Q. Does it violate confidential/blind peer-review to reveal reviewer comments even after publication?

A. Absolutely. The authors submit to the journal with an expectation that their submission is confidential and, unless the paper is published, will remain so indefinitely. The only way that reviewer comments could (ethically) be made public at any point is with the authors' express permission (perhaps by checking a box at the submission phase). However, revealing that person X reviewed for journal Y in a particular year is probably not an ethical problem, as there's no information about which paper they reviewed.

Q. What can journals do to make sure reviewer comments remain confidential?

A. In the absence of express permission from the authors, they should remind reviewers that their comments are confidential and should not be made public. Publishers should not engage with organisations like Publons until they remove all public peer reviews from their websites.

Q. How can reviewers ensure they are able to share and get credit for their work?

A. As mentioned above, there is plenty of credit for good review work, it's just largely private and not easy to measure. Journals or publishers could also launch reviewer credit initiatives of their own, like the 'top 300 reviewers' that we do at Molecular Ecology (see http://www.molecularecologist.com/2015/04/mol-ecols-best-reviewers-2015/)

Q. Can service providers work with journal editors, publishers and reviewers to help facilitate openness and transparency in peer review?

A. They could, if a) openness and transparency in peer review was desirable (which I am convinced it isn't), and b) these organisations were respectful of peer review ethics.

Last, it's worth reflecting on how these reviewer credit organisations (especially Publons) have been approaching publishers. For journals that are not signed on with Publons, reviewers have been allowed to post a) the manuscript title, b) the authors' name, and c) the full reviewer comments on the website, in violation of peer review confidentiality. Publons then tells publishers that they should sign up, as their journals can then control the information reviewers can post onto their site. It seems wrong to use breaching review confidentiality as an incentive for publishers to get involved.

  • Posted by Andrew Preston,..., 4/9/2015 12.32pm

This is an important topic and something we have spent a lot of time considering in consultation with the reviewers on our platform, our publishing partners, and other stakeholders -- all with the goal of building what is best for research.

We've written up a full summary of our thoughts on our blog and welcome comments there or here:
http://blog.publons.com/post/128303366528/who-owns-the-review

Since the discussion specifically mentions Publons, we request a presence at the COPE forum on September 9.

One quick correction to the anonymous (heh) comment above:

> Publons then tells publishers that they should sign up, as their journals can then control the information reviewers can post onto their site.

This is not true. Any journal can set a privacy policy on Publons, partner or otherwise. Please see the post above.

  • Posted by Irene Hames, 4/9/2015 5.54pm

This is a very important and timely discussion. Increasing transparency in peer review is, in general, a positive thing. Being able to view reviewers’ reports tied to journal articles can be very valuable, especially so for early career researchers, giving them the opportunity to see what reviewers’ comments, editorial correspondence and reviewer responses look like. But it’s the prerogative of each journal to decide what kind of review process it wants to operate, and what, if any, level of openness it wants to have. A whole range of factors will influence the discussions and decisions. Most journals don’t, however, seem to have discussed this, or if they have, haven’t made their expectations clear in the information they post on their websites.

It’s crucial all journals do make their policies and expectations clear, so that authors and reviewers are aware of these when they’re deciding whether to submit a manuscript to or review for a journal. There may even be a range of possible scenarios for a single journal, for example as outlined by the Royal Society’s journal Royal Society Open Science in its guidance
http://rsos.royalsocietypublishing.org/content/open-peer-review-royal-so...

With the advent of new third-party services* it’s become more important than ever to have clarity, as researchers, especially those at an early career stage or from areas of the world without a long research tradition, may otherwise inadvertently contravene what are considered the ethical norms in scholarly publishing. Process transparency and appropriate agreement from the various parties when there are options available makes for fairer and more ethical review, and minimises the chance of problems and relationship breakdowns.

I’ve avoided using ‘ownership’ of reviews as that doesn’t really need to come into it. Anyone who submits to or reviews for a journal or other publication venue enters into an agreement; as with anything else, if they don’t like the T&Cs they can take their labour and loyalty elsewhere.
(*Declaration: I had quite an extensive discussion with Daniel Johnston from Publons in the Scholarly Kitchen post linked to in the reference given in the opening text of the introduction to this discussion topic.)

  • Posted by Lachlan Coin, 8/9/2015 7.38pm

As the founder of Academic Karma, I think this is a really important and timely discussion for the research community.

Just a quick note on our process first - reviewers who have used our platform for review are offered the opportunity to make their reviews public once the paper they reviewed is published. They can also reveal their identity or stay anonymous if they prefer. We then email the author of the paper to ask them if they are happy for the review to become public, or if they would like to keep the review private.

With regards to the questions posed in the post, I think the first thing to point out is that the question 'who owns the peer review' has been asked before in different forums, and its quite clear that the reviewer owns the copyright to the review unless they sign a copyright transfer form, which is very rarely done. So the reviewer has every right to publish their comments once the paper is published. At Academic Karma we are admittedly taking a very conservative approach of emailing the author for permission to publish the review.

I agree with Andrew Preston that the most important question is actually : 'can publishing reviews help improve science and the research process'. Given the rise in the number of retractions, and the rigour of the peer review process being called into question at several journals, we think the answer is clearly yes.

  • Posted by Bahar Mehmani, 8/9/2015 8.35pm

Making peer review a measurable research output is a great step in giving credit to reviewers. Researchers, specifically early career and young ones have to be recognized for the time and effort they put on peer review on a voluntary basis. But this shouldn’t come with the cost of breaching confidentiality. Recognizing reviewers doesn’t necessarily mean publishing peer review reports.
On the other hand, upon having the consent of reviewers, authors, and editors publishing peer review reports alongside the articles (and if possible author’s response to them) not only helps reviewers to get credit for their input, but also provides a useful resource for young and fresh reviewers.
As it is mentioned in previous comments, journal publishers can acknowledge their reviewers in different ways. For example, in Elsevier we are providing our journal reviewers a personal review profile page that is only accessible to them. Reviewers can see their 5 year review history records per journal (only the year and month of review submission is recorded) and collect their perks and rewards from there. It is then up to them to make their review profiles public and/or list their peer review records for non-Elsevier titles (again only the journal title and review year will be listed). More about this initiative can be found in our recent article: http://www.elsevier.com/connect/giving-reviewers-more-of-the-recognition...
More about publishing peer review reports next to articles can be found here:
http://www.elsevier.com/reviewers-update/story/peer-review/elsevier-pilo...

  • Posted by Donald Samulack, 9/9/2015 5.04am

It seems to me that the offer of a manuscript by a journal to a reviewer in blind peer review is a form of "privileged" if not "confidential" communication between the journal and the peer reviewer; it was likely the author's intent that the manuscript was to be considered a privileged communication if they chose to submit to a journal with policies of traditional blinded peer review.

Also, remember that the author owns copyright over their intellectual property until they turn over that copyright. The paper is in fact on loan to the journal until the journal requests transference of copyright (if their publishing model dictates as such). This gives further credence to the fact that any effort by the journal to seek peer review is understood by the journal and the original author to be under the auspices of at least a privileged communication by the journal to the peer reviewer. The unpublished original manuscript is not yet available in the public domain, and may never be made available in the public domain; in fact if not published, it should be solely the authors choice as to when and how any aspect of their intellectual property is to be brought into public domain.

It is my opinion that because of the "privileged" nature of the communication between the journal and the peer reviewer, there is in fact an inferred contract of privileged "work for hire" between the journal and the peer reviewer, as soon as the peer reviewer accepts to undertake the peer review (whether or not there is financial compensation for undertaking the peer review; this work-for-hire is further strengthened by the instruction and template nature of some journal's peer review; another level of intellectual property). It is my belief that even if the manuscript is not labelled as privileged or confidential, as an inferred privileged or confidential communication through processes of knowingly blinded peer review (known by all parties), any works derived from such original author's intellectual property should not be considered the peer reviewer's intellectual property, and hence they should not have any right to copyright over their "derived works"; much like a person who does paper crafts does not own commercial copyright (right of sale) over a Photoshopped print of a rubber stamp impression. The peer reviewer certainly should have no right to post or effect the posting of their reviews in public display by their own hand, or through the means of another without first the expressed written consent of the journal (and maybe even the author) who initiated the de facto work-for-hire contract between the journal and the peer reviewer..

I believe that this is a legal issue under both contract law (local) and copyright law (international). I encourage the legal community to weigh in on this, as well as the Copyright Clearance Center. It is time for clear and concise formal legal ruling on this issue. I personally believe that there is scope for a case to be made regarding "tangible harm" to the author, especially if the author can show that once the manuscript was rejected from first publication, the public posting (known or unknown to the author, or to the journal) of an unfavorable peer review derived through privileged communication, prejudiced its publication in another journal or medium.

I hope that these thoughts provoke much debate regarding this important issue. The rights of all parties are at issue, and they need to be clarified.