Case discussion: repeated complaints about a review

6 November 2020

Case summary

A reader, whose identity was unclear, alerted a journal about an author who had recently been found guilty of publication misconduct and had had a dozen papers retracted. Subsequently, the reader repeatedly emailed the journal to complain that one of its published review articles was by the same author, cited six of the retracted papers, and should thus be retracted. The editor found that nearly a fifth of the review’s text was affected by the retracted papers, but it was difficult to judge whether the review’s conclusions were affected.

Question for the COPE Forum

Should the journal publish a correction with a note alerting readers of the retracted references, or, because of scarce resources and possible further retractions of references, issue only a note?

Forum advice and follow up

The Forum agreed that readers should be informed of the retracted references, and advised publishing an expression of concern rather than a correction. Readers could then judge the review for themselves.

Some Forum attendees suggested that the editor ask the author whether the review should be retracted or ask the author’s institution for advice. Others suggested that the editor write an editorial discussing whether the retraction of the references affected the review’s conclusions. In response, the editor published an expression of concern and considered the case closed.

Case discussion

This archived COPE Forum case deals with a repeated complaint urging retraction of a review that referred to retracted research. It is categorised under three COPE Core Practices:

Post-publication Discussions and Corrections: “Journals must allow debate post publication either on their site, through letters to the editor, or on an external moderated site, such as PubPeer. They must have mechanisms for correcting, revising or retracting articles after publication.”
Allegations of misconduct: "Journals should have a clearly described process for handling allegations, however they are brought to the journal's or publisher's attention. Journals must take seriously allegations of misconduct pre-publication and post-publication. Policies should include how to handle allegations from whistleblowers."

Journals need to appropriately handle reader and stakeholder concerns, complaints, and allegations of misconduct, whether raised through informal messaging (emails, online posts) or through formal correspondence (submitted letters, commentaries). Journals should assess whether concerns are genuine and warrant follow-up and formal investigation.

Other relevant COPE Core Practices are:

Complaints and Appeals: “Journals should have a clearly described process for handling complaints against the journal, its staff, editorial board or publisher.”
Journal Management: “A well-described and implemented infrastructure is essential, including the business model, policies, processes and software for efficient running of an editorially independent journal, as well as the efficient management and training of editorial boards and editorial and publishing staff.”

These two core practices are related in the case, because the journal owner should ensure there are clear policies, procedures, guidelines, internal processes, and staff training programmes on how to efficiently and fairly manage complaints, correspondence, and repeated correspondence.

Correspondence

At the outset, there should be a contact person and triage system in place to identify and follow up valid concerns about content quality, as raised in both formal and informal correspondence. For processing formal correspondence that criticises a published article, journals can follow COPE’s “Handling of post-publication critiques” flowchart. For handling criticisms of published articles when raised by informal correspondence, journals can follow the COPE flowcharts titled “How to respond to whistleblowers when concerns are raised directly” and “How to respond to whistleblowers when concerns are raised via social media”.

Some journals consider various factors for formal reader correspondence, such as recency, originality, length, number of illustrations and references, the need for evidence to support points, whether new data are allowed, research ethics, and conflicts of interest. There may also be policies and guidelines on if or when peer review will be used and a limit on the number of rounds of responses. Some journals explicitly say they do not allow the formal correspondence section to be used to report possible errors. Reader correspondence relating to post-publication corrections should be investigated separately rather than immediately published. If appropriate and with consent, the reader may be acknowledged in the correction.

However, there is also a need early on to filter out frivolous, baseless, incomprehensible, or overly vague claims and old/resolved issues, as well as potential or likely cases of libel, trolling, cyberbullying, and personal attacks. Repeated and frequent correspondence from a person who keeps raising the same, or even new, concerns about a particular author or continually disagrees with editorial decisions could become harassment and be a drain on resources.

In this case, repeated correspondence was received by email and the identity of the whistleblower was unclear. Whistleblowers and readers raising concerns have the right to be anonymous or use pseudonyms, as explicitly noted in COPE Forum case 11-20. Complaints and allegations of misconduct, however received, are to be evaluated on their merit and evidence. After the conclusion of a journal investigation, the correspondent should be told the case is closed. One timely appeal could be reasonably entertained on the provision of evidence. Otherwise, if an appeal is repeated on the same grounds, as in case 11-20, the editor can reiterate that the case is closed.

COPE has published relevant advice on managing persistent, prolific, or unreasonable correspondence in “Addressing ethics complaints from complainants who submit multiple issues: COPE Discussion Document”. If necessary, the editor may need to seek advice from ethics, legal, and institutional representatives, especially if the reader persistently includes libelous, offensive, hostile, or illegal content in their correspondence.

Ideally, the journal would have a standard operating procedure to handle different types of concern and determine if the points raised relate to, for example, differences in interpretation or opinion, possible errors needing correction/clarification, allegations of misconduct, or a complaint about journal procedure. If needed, a post-publication round of peer review could be initiated. If a specific allegation of misconduct has been made, journal staff can follow relevant COPE flowcharts.

The journal needs to address all valid concerns from a reader, and it would be more convenient to deal with all concerns at once. The journal and authors would also need to discuss if so-called errors really are errors. Distinctions should be made between fixable mistakes in a paper; alternative viewpoints; and ongoing scholarly progress and debate where previous findings are reinterpreted or overturned by new research thanks to technological or methodological advancements. Furthermore, data or findings from new research should not normally be added post-publication as a correction to update a paper; a new research paper should be prepared. Nevertheless, some journals do allow authors to publish meta-analysis update articles and online continually updated (“living”) systematic reviews.

Transparency of reviews

In this case, the persistent complaint related to the inclusion of now retracted material in a review article. As noted by the Forum, the journal's duty is to update readers on the integrity of its content by adding an expression of concern about the six retracted references.

The possibility of inclusion of the six references as indicating excessive self-citation or review bias was not explored. It was also not stated if the review was a narrative review, systematic review, or meta-analysis. Narrative reviews are especially prone to selection and reporting bias. Transparency of review methodology is thus important for reproducibility. If the conclusion of the review was centred on the retracted material, retraction of the review would have been justified. Original research articles that relied on retracted material for a major part, such as the methods, data source, or conclusions would also meet the same fate.

Systematic reviews and meta-analyses should be robust (eg, using PRISMA-Protocol guidelines), registered publicly in advance (eg, on PROSPERO), or submitted to a peer-reviewed journal as a protocol paper or registered report before the actual research starts. Meta-analyses may need statistical adjustments to minimise or correct for biases. An additional sensitivity analysis that omits inferior primary studies may be performed to test whether their omission affects the overall conclusion.

If the review in this case were a meta-analysis, retraction would have been appropriate if an additional analysis that omitted the retracted studies yielded results that rendered the published conclusion invalid. Alternatively, an addendum showing a relevant reanalysis may have been appropriate if the conclusion were unaffected after the omission of the retracted studies. The contamination of reviews, and other article types, by retracted (or later retracted) studies is a major problem if wrong conclusions are acted on by practitioners, policy-makers, and other stakeholders. Appropriate ways of addressing this problem are needed. The COPE Seminar 2021 had one session on “Reducing the inadvertent spread of retracted science”, and industry efforts are being made to detect citations of retracted papers. The Cochrane website has guidelines for dealing with potentially problematic studies that were included in a meta-analysis.

Transparency of policies

As mentioned, for full transparency journals should clearly display their codes of practice, editorial and review policies, and complaints/appeals procedures. Such information can be based on COPE’s Core Practices and website resources, as well as other industry/society guidelines, publisher guidelines, and local regulations. Retraction is warranted only for certain reasons listed in the COPE Retraction Guidelines, such as if the conclusion is unreliable and/or there is an ethical problem with the data, research, reporting, or peer review. As part of an investigation, the editor might consider asking for the advice of the original peer reviewers of the paper or new reviewers to inform the editor’s decision.

Journal guidelines should remind readers that all correspondence must remain civil, legal, and ethical, and that the editor's decision is final. If the reader in this case were to persist in demanding a retraction, they should be explicitly told the journal's policies and that the outcome of the investigation is final. If the reader were to appeal based on a procedural point and the journal is a COPE member, COPE’s Facilitation and Integrity Subcommittee could be asked for advice on whether due process had been followed.

Readers might resort to direct correspondence with the author. Unfortunately, if the parties are academic rivals, there is a risk that the correspondence could become unreasonable, personal, or vexatious. The reader may even escalate their action by complaining about the authors to their institution, colleagues, and funder, but this situation would be beyond the journal's remit.

If the reader’s correspondence with the institution were professional and raised a valid concern that the institution in turn referred to the journal, the editor could follow relevant COPE guidance contained in “Cooperation between research institutions and journals on research integrity cases: guidance from the Committee on Publication Ethics (COPE)”.

Trevor Lane on behalf of the COPE Education Subcommittee