Meta-analysis: submission of unreliable findings

Case number: 
12-11

Case text (Anonymised)

A meta-analysis was conducted of about 1000 patients included in a number of small trials of a drug for emergency management administered by route X compared with route Y. The report concluded that administration by route X improves short term survival.

Chronology
 

  1. The paper was submitted to our journal in September 2011 and after peer review was returned to the authors for revision in November 2011.
  2. In the letter sent to the authors, the editor stated: “Before coming to a final decision on your paper we will need to see your responses to our referees' comments. We will also need you to discuss the preliminary results of the large randomised controlled trial (RCT) recently presented at a national meeting which conflict with and may negate the conclusions of your meta-analysis.”
  3. The revised version was sent back to us in January 2012. It contained only one mention of the large RCT without quoting any of its findings. The covering correspondence discussed the RCT findings that had been recently presented and speculated as to why they appeared different from the findings of the meta-analysis.
  4.  We accepted the meta-analysis in January 2012. We considered that the differences described by the authors were irrelevant, because the large RCT had not, at that time, been published in a peer-review journal and the only information available was from data presented at a meeting.
  5. We now know that the authors of the meta-analysis were fully aware of the findings of the large RCT at the time they submitted the revision because the RCT paper had already been accepted by a high profile journal and the lead author was co-author on the meta-analysis submitted to our journal. None of this was revealed to the journal prior to accepting the meta-analysis
  6. In March 2012, the high profile journal published the large RCT which randomized more than 2000 patients to drug treatment by the two different routes. The main conclusion was of no difference in survival for route X versus route Y. This finding rendered meaningless the finding of the meta-analysis accepted by our journal 6 weeks previously.
  7. The authors of the meta-analysis were then emailed asking if they would now update their meta-analysis with inclusion of the RCT data.
  8. The response was negative but an email from another co-author (who wrote the editorial accompanying the RCT in the high profile journal) agreed “it makes no sense to report a meta-analysis claiming death reduction considering available data”. He then copied us in an email he had sent to the lead author of the meta-analysis in January 2012, before it was sent back to our journal: “just to let you know that I am finishing an editorial on (the RCT) which will likely come out very soon with the main Ms....I would suggest that you try to include (the data from the RCT) into your meta-analysis ASAP”
  9. The authors chose not to include the data from the RCT in the revised version of the meta-analysis they submitted to our journal, even though they had available those data. Since then the authors of the meta-analysis have steadfastly refused to update their paper. Meanwhile the editorialist for the high profile journal has asked that his name be removed from the meta-analysis in our journal.

The authors of the meta-analysis, one of whom was the lead author of the high profile journal report, had full access to the RCT data at the time they were preparing their revised paper for our journal. They knew that the main finding of the RCT contradicted the conclusion of their meta-analysis and ignored the suggestion of a co-author (the editorialist) to include the RCT data in their revised paper to our journal.

COPE states that journal editors should consider retracting a publication if they have clear evidence that the findings are unreliable. The authors of the meta-analysis knew their findings were unreliable at the time they submitted their revised paper and we now wish to have the paper retracted

Advice: 

The Forum agreed there were grounds for retraction of the paper. Clinical decisions are often based on meta-analyses and the editor cannot rely on all readers being aware of the newly published meta-analysis in the other journal. However, the ideal situation would be for the author to correct the published paper. Although the author has refused to do this, the Forum suggested that the editor should contact the author again, asking him to correct the paper. The editor should tell the author that if he refuses to correct the paper, then the editor will be left with no option but to retract the paper.

The Forum suggested that the fact that  the editor did not ask the authors to wait until the results of the RCT were available before submitting their final paper has contributed to the confusion surrounding the case. Going forward, the editor should consider revising journal policy to request authors to send any related papers under submission to them when they submit an article.

Follow up: 

Following the Forum’s advice, the editor emailed the corresponding author of the paper, copying in the co-authors, stating that he hoped the authors would agree to update the meta-analysis whereupon the matter would be concluded. He told the authors that if they did not agree to provide an update, he would retract the paper. The editor received no reply and therefore retracted the paper.
The retraction notice stated that the findings of the paper were unreliable because they failed to address data from the large RCT, to which the authors had access prior to submission and which contraindicated the paper's conclusion. The notice said that authors were asked to update the paper to include the RCT findings but, with the exception of one of the authors, they declined. Owing to this difference of opinion, this author asked to be removed from the list of authors, a request to which the journal acceded. The notice stated that under these circumstances, the matter was considered by COPE who recommended retraction and this paper has now been withdrawn.

Resolution: 
Case Closed
Year: 

Comments

  • Posted by Surprised reader, 1/8/2012 10.33pm

Interesting case, and also slightly bizarre. Why won't the authors of the meta-analysis add the additional data? There appears to be no good reason not to include them and many reasons to include them. Very odd.

  • Posted by martin, 15/2/2013 7.25am

I have been searching for the case number 12-11 as I had a few things to get cleared. Are you sure that the chronology of the case that has been enumerated is correct. I say so because I read it otherwise in another page. Please get back.

  • Posted by Neal Haddaway, 2/3/2014 3.23pm

Surely meta-analyses should be transparent, repeatable and objective, in the same way that all systematic reviews are. If meta-analyses are only accepted in when written in a transparent way anyone can update the meta-analysis for them and publish that. The risk of publishing retractions is that once the paper is out there with a stamp of approval, it's very difficult to stop people using it, especially if the retraction has taken time. Post-publication peer-review like PubPeer would help, too.

  • Posted by M Barnish, 29/2/2016 6.03pm

A meta-analysis is supposed to be up to date at the search date, which may be up to 6 months prior to submission according to received wisdom. It may or may not be up to date at publication, but will be more up-to-date than any other previous meta-analysis. I consider this a totally incorrect decision.