You are here

Home Resources Forum discussion topics

Artificial intelligence (AI) and fake papers

March 2023

Watch the introduction to "AI and fake papers" with Dr Sarah Elaine Eaton and Marie Soulière.

The rise in the role of artificial intelligence (AI) in the writing, creation and complete production of research papers has taken the publishing world by storm in the last year. This has significant implications for research integrity, and the need for improved means and tools to detect fraudulent research. The advent of fake papers and the systematic manipulation of peer review by individuals and organisation has led editors and publishers to create measures to identify and address several of these fraudulent behaviours. However, the detection of fake papers remains difficult as tactics and tools continue to evolve on both sides.

With the proliferation of paper mills (profit oriented, unofficial and potentially illegal organisations that produce and sell fraudulent manuscripts that seem to resemble genuine research) and the recent release of advanced writing and image creation tools, we are raising for discussion in this forum various ethical questions surrounding the use of AI for both fake paper creation and the production of papers based on valid research. We aim to discuss the aspects of authorship, bias, originality, and using AI tools to counteract AI fraud.

During this COPE Forum (23 March 2023) we started a discussion about these ethical issues, which COPE will use to develop a discussion document on this topic. 

Back to top

Questions for discussion

1)  Is it an acceptable, and ethical, practice for an author to use AI to write a scholarly article?

2)  Can we detect whether an article has been written by AI? Is it important to do so?  If so, why?

3)  Can an article written by an AI trained on existing articles be considered original? 

4)  What are the implications of AI-generated texts for plagiarism of words, images, and ideas (with algorithms trained on existing articles)?

5)  Could peer review be performed exclusively by dedicated AI tools (designed to detect fraud and validate data and figures)? 

6)  What kind of bias could an AI introduce if it were to write or peer review articles?

7) What are some considerations for equity and inclusion that need to be taken into consideration (e.g., for scholars with disabilities who might use AI tools as an assistive or adaptive technology)?

Discussion hosts

Dr Sarah Elaine Eaton, Associate Professor, University of Calgary. Elected Council Member, COPE

Dr Eaton’s research focuses on academic ethics in higher education. She is the Editor-in-Chief of the International Journal for Educational Integrity (Springer Nature), a Q1 (Education) journal published by BMC Springer. Dr. Eaton is the co-founder and co-editor of Canadian Perspectives on Academic Integrity.

Dr Marie Souliére, Senior Publishing Manager, Frontiers. Elected Council Member, COPE

Dr Soulière leads strategic publishing projects in open-access publishing, with a specific focus on research integrity and quality peer review, balanced with operational efficiency and automation. She was heavily involved in developing Frontiers’ artificial intelligence review assistant (AIRA).

Back to top

Comments from the COPE Forum, March 2023 - NOTE, comment do not imply formal COPE advice, or consensus

  • Artificial Intelligence (AI) is a broad term encompassing computer systems, algorithms and technologies that exhibit behaviour or perform tasks that can be considered ‘smart’. These could include making decisions, recognising and translating speech, and different types of visual perception. 
  • Machine learning (ML) is a subset of AI and consists of algorithms that detect patterns based on a pool of training data, and use them to predict, detect and make decisions. Natural language processing (NLP) is another subset of AI which transforms and uses language.
  • ChatGPT is an example of both ML and NLP and is generative: it creates new content. Such generative AIs can create text, code, images, videos and full research articles.
  • AI tools are already being used in scholarly publishing, for example in pre-peer review checks (e.g., language quality, confirming that a submission is in scope for the journal) or as part of peer review (e.g., identifying reviewers, checking for data or image manipulation). See COPE’s discussion document on AI in decision making and COPE’s seminar on Trustworthy AI for the future of publishing.
  • The current focus for attention is the use of AI to create whole articles or images, and particular, on the creation of false data, analysis and attributions. In this context AI tools can be used maliciously, for example by paper mills.
  • COPE has recently issued a position statement on the use of AI as an author. This stated that tools like ChatGPT cannot fulfil the ICMJE criteria for authorship and that the use of such tools should be transparently declared.
  • Detection tools are currently in development, and it is likely that existing tools, for example, for the detection of matched text, will start to incorporate AI detection. Several companies, including CrossRef together with Turnitin/iThenticate have been working on the detection of plagiarism of ideas for some time in the context of paper mills. They, and others, are also working on tools for the detection of image manipulation and image duplication (and copyright infringement), but the latter is hindered by the fact that no tool can have access to all possible images in scholarly publishing.
  • The question of whether AI-generated text can be considered original is an important one, especially since the tool is trained on existing work. There are arguments being made on both sides, but it is important to keep the ultimate responsibility of humans in mind. This includes responsibility for the finished product, for fact-checking, and for the verification of citations.
  • Many of the most pressing questions about AI and plagiarism surround non-text content: images, ideas and so on. Again, attribution is key: publishers should expect all content to be generated by human authors unless there is a declaration to the contrary.
  • AI tools are being used to create original review articles, but it is important that the author can provide and declare accurate references. Peer reviewers and editors should be alerted to the fact that some tools will fabricate references, and place enhanced emphasis on this aspect of the review process.
  • It is currently not always clear exactly what material is used for the training of LLMs, including whether it encompasses data from sources which for which access is closed (including, potentially, articles previously written by an author). Generally speaking, most of the training data are open access, but since the material used for detection tools is proprietary it is not always easy to probe more deeply. Usually, the larger the dataset used for training, the better the tool works. 
  • It is important to declare not only the tools used by authors, but also the version, so that future readers will know what was being used at the time that an article was produced. Tools are likely to change very quickly in scope and accuracy.
  • There are related copyright issues, for example if an AI tool returns work very similar to something that is already published (text, images, art, etc). It was noted that these issues already exist where human authors are collating information from different sources or utilising tools for certain processes. However, relinquishing control over the production of content either in part or in whole does not absolve the author from their responsibilities. It is the human who remains responsible and accountable for any copyright infringement.
  • The use of AI tools is not inherently unethical and may be useful, for example, for authors who do not write in English as their first language, who have learning disabilities (e.g., dysgraphia) or other difficulties with writing. In that respect, artificial intelligence can be a useful assistive or adaptive technology. The key factor is responsibility and clarity over use. The need for detection comes where their use is not declared. 
  • With regard to peer review, AI tools may be useful, but responsibility again remains with humans, for example in dealing with subsequent queries from authors. A report generated by an AI detection tool can only assist humans in establishing originality, and assessing other potential issues.
  • There is debate ongoing about the type of bias an AI might introduce if it were writing or peer reviewing articles. Some tools are trained on information which contains bias. Sometimes this will be obvious to an editor; for example if it returns Americanised spelling when the journal requires British English spelling. However, as the AI tools develop it will likely become harder to spot these signs. An alternative view is that bias may be reduced as the training data are improved. Editors should also remember that humans can be biased too, and that there may be tasks which AI tools can do with less bias than humans. It is an area where further attention will be necessary. We can often only detect bias when we see it in use.
  • Higher Education institutions are considering how to handle AI tools with students and staff. Some are working on supporting its use in ethical ways, mindful that students will need to operate in a world where AI is ubiquitous. However, responses are likely to differ at different institutions. Concerns about its use for writing assignments and answering exam questions are pervasive.
  • COPE’s consideration of these issues is informed by the variety of backgrounds of its Council members. It has also recently launched membership from the University and Research Institute sector where AI is forming part of the discussion over guidance and standards. 
  • Some proposals were made for actions where an editor suspects use of AI tools but the author denies it. It was noted that this happens with standard text plagiarism already, and in those cases it becomes an editorial decision on whether to believe the author or not, following an investigation. The editor might consider whether there are other signs of ethical misconduct; they might contact the author/s to ask for additional information, raw data/images, etc. They could also consult the peer reviewer or get advice from the publisher and editorial board. Editors in Chief should not feel alone or unsupported in dealing with these matters. The resolution may come down to balance of probability, with the final decision lying with the EiC.
  • Editors and authors should consider the degree of assistance given by an AI tool. Translation and grammar correction tools (e.g. https://www.deepl.com/en/translator; https://www.writefull.com/) are commonly used already and are unlikely to need a declaration of use (although human checking is still advised). Where a tool has contributed to content creation, analysis, paraphrasing, however, there should be greater transparency, especially as these tools come to be further integrated into word processing and database programmes. It is never a wrong course of action to offer transparency and declare the use of a tool and the version used. Using AI tools for translations may bring separate issues of copyright, but again, it is up to the human to take responsibility for ensuring compliance with regulations.
  • Whatever tools an author uses to assist them the fundamental principle is that they must be able to assert accountability for the output and its originality. If outsourcing some tasks allows them still to make these declarations truthfully then this may not be an issue in terms of publication ethics. Whenever they can no longer truthfully assert their full responsibility then publication ethics have been breached, whether this is via use of an AI tool or a service like a paper mill or a ghost writer.  

Back to top

Further resources

Back to top

We welcome comments on this discussion from members and non-members. 

Your comments

You must be logged in to leave a comment on the COPE website. If you are not a member of COPE you can register to create a guest account.

Comments will not appear immediately. We review comments on our website before publishing them, to ensure they are respectful and relevant.

Comments

  • Posted by Susan Garfinkel, 10/3/2023 3.11pm

As more and more issues become identified with AI and fake papers, I would like to hear what journals are doing when they detect such issues, and also what the expectation is when issues may be referred to an institution. I ask because institutions are struggling with what is the best way to handle these issues. 

  • Posted by Bridget Sheppard, 23/3/2023 2.46pm

I was interested to see how ChatGPT would answer question 1, it said 'I can be used by authors to assist in writing a scholarly article. However, it's important to note that I'm a tool and not a substitute for the critical thinking and domain expertise that is required to produce a high-quality scholarly article. I can help with tasks such as summarising research, generating ideas, suggesting references, or even first drafts, but it's ultimately up to the author to verify and validate the information presented. Authors should ensure that the content is accurate, relevant and appropriately cited. It’s important to be transparent about the use of language models in the writing process.’

  • Posted by Gary Balian, 24/3/2023 7.58pm

The discussion yesterday on AI and publication ethics was very useful and informative.

1) It would be helpful to read any guidelines COPE can share with editors as to the way we might be able to deal with authors of papers that have been generated using AI software. 2) If additional information becomes availabe regarding programs that can detect the use of AI in the preparation of manuscripts, sharing it with publishers and editors would be very helpful. 3) Developing software that identifies contextualized similarity of data, presented in tables, images and graphs might be more challenging but could become crucial to the decisions made during peer review. 4) Meanwhile, cross ref similarity checks using iThenticate prior to beginning the peer review process has become an essntial step.      

  • Posted by Zachary Kenny, 5/7/2023 7.05pm

In the comments from the COPE forum, you state: 

  • The current focus for attention is the use of AI to create whole articles or images, and particular, on the creation of false data, analysis and attributions. In this context AI tools can be used maliciously, for example by paper mills.

I'm curious as to why you mention paper mills specifically? Is there a concern that a proliferation in AI generated content is going to result in more articles being printed on paper??

  • Posted by Alysa Levene, 7/7/2023 8.30am

Hi Zachary

Thank you for your comment. 

Paper mills are organisations which create articles for sale. Generative AI tools could potentially enable these bodies to do this at a much greater scale, and with few indicators that the work is not authored by the person named as the author.

COPE has produced guidance on handling paper mills.

Posted on behalf of COPE.

  • Posted by Mujtaba Ankush, 24/2/2024 4.28pm

With the increasing use of artificial intelligence tools in papers, is there a policy that must be followed for journals to protect the manuscripts that are published? Also, is improving the language (if english is a second language) in writing research through their use of artificial intelligence tools considered a convincing reason to reject the article? What are the acceptable percentages for article (if there are percentages such as Turnitin similarity)?

  • Posted by COPE response, 27/2/2024 1.08pm

Thank you for your comment, Mujtaba. It is currently difficult to accurately detect undisclosed use of AI, therefore we advise that journals focus on creating policies which offer clear guidance to authors on how and when it can be used. It is currently expected that authors should declare any use of AI tools, including the name, version, and what prompts were given. Tools to help with language and editing are not usually regarded in the same way since they are helpful in allowing authors to access publishing in second languages. 

We do not advise that journals adopt minimum percentage criteria, but instead combine reports with human checks. Two articles may have similar percentages reported by a piece of software, but while one could represent writing features which are common and non-concerning, another may have a small number of full passage which look suspicious.