Possible omission of information essential for conclusions in a research paper

Case number: 
14-11

Case text (Anonymised)

In 2013, our journal published a paper describing an observational study comparing two drugs (A and B) for the management of a chronic disease over a period of 10 years. The conclusion in the paper was that mortality was higher in group A (97 deaths) compared with the other group B (52 deaths) (hazard ratio 1.76, 1.22 to 2.53; P=0.003). This analysis was done after adjustment for a large number of confounders, and was approved by our statistical advisor. The authors of the papers did acknowledge that this was an observational study, and did state that residual confounding might be present.

In 2014 we received a letter of concern by a researcher, employed by the company selling drug A, who felt that the authors of the 2013 paper omitted essential information that might impact on the conclusions. It appears that the routine management of this disease has changed substantially over the 10 year period, and this should have been treated as a confounder for which statistical adjustments should have been made. This change in routine management of the disease is documented in a paper published in 2014, but the researcher felt that these authors were probably aware of this much earlier and should have disclosed this information during the review process of their 2013 paper.

In our initial response in July 2014 to the letter of concern, we asked the researcher who sent us the letter of concern to send us a detailed rapid response to the 2013 paper, which we could publish. We have also asked advice of our statistical advisor who reviewed the 2013 paper, and he acknowledged that this information might impact on the statistical calculations and thus the conclusions of the paper. But with the data available to him, he is not able to make a definitive assessment of how much impact it would have. He has suggested to put these questions to the authors of the 2013 paper.

Question(s) for the COPE Forum

  • The researcher who raised the concern has not made his concerns public by sending us a rapid response that we could put to the authors and publish, with their response, on our website. We will certainly put the questions to the authors of the 2013 paper, but we wonder if we should publish these concerns?
  • Another problem is that, due to the complexity of the statistical calculations, we are entirely dependent on the authors to judge whether the routine management data would have seriously impacted the conclusions of the 2013 paper.
Advice: 

The Forum agreed with the course of action of the editor to date—namely, inviting the researcher to write a formal note, stating his concerns, that can be made public and the authors can then be invited to make a response. This ensures the process is transparent. But if the researcher who raised the concern does not want to make his concerns public, there may be little that the editor can do.

One suggestion was that the editor could publish the concerns anonymously and invite the authors to respond. Another suggestion was to treat this as you would a whistleblower by investigating the issue, and asking the authors to respond specifically to the questions raised by the researcher if necessary. It is clear that the editor has concerns about the paper and these should be addressed in some way.

So, the best option may be for the journal to publish the concerns, not necessarily revealing the researcher’s identity, and invite the authors to respond.

Follow up: 

The researchers who sent a letter of concern sent a letter to the editor which was published, and which was answered by the authors of the original paper.

The journal’s statistical advisor has found the response satisfactory. The editor considers the case now closed.
 

Resolution: 
Case Closed
Year: 

Comments

  • Posted by Peter Auster, 8/10/2015 1.49pm

The implication that the original authors selectively ignored the analytical issue of treatment modifications over the study period falls on the unfounded assumption that "the [responding] researcher felt that these authors were probably aware of this much earlier and should have disclosed this information during the review process of their 2013 paper." I suggest that it is equally plausible that without the subsequent 2014 paper, such an assumption, and the need to test for such patterns, would have required another study (at least based on what appears in this commentary). That said, the letter pointing out this issue and subsequent response by the original author is the way the scientific enterprise should work. No implication of misdeed was necessary for this advancement in thought to play out.