A recent survey in which we asked COPE members to vote on the diversity, equity and inclusion topics they would like to discuss, bias in peer review was voted the topic of most interest, so we devoted the October 2021 COPE Forum topic to it, where Forum participants discussed the issues. While this important topic aligns most closely with COPE Core Practice: Peer Review Processes, its impact is wider reaching. The explicit and implicit biases, or conscious and unconscious biases, of editors and peer reviewers could prejudice which manuscripts are reviewed or ultimately accepted for publication. As COPE member Cambridge University Press explains in their information for peer reviewers: “Reviewers must give unbiased consideration to each manuscript submitted. They should judge each on its merits, without regard to race, religion, nationality, sex, seniority, or institutional affiliation of the author(s). … Rooting your review in evidence from the paper or proposal is crucial in avoiding bias.”
Diversity and inclusion in peer review survey
Diversity and inclusion initiatives seem to be key to helping reduce bias in peer review. As part of Peer Review Week 2018, COPE conducted a survey to understand more about how the community views this topic. Recently, we wondered if and how attitudes, policies, and/or practices have evolved over the past few years. To that end, COPE largely reproduced the survey during Peer Review Week 2021, and below are highlights of the responses.
COPE received 267 responses to the new version of the anonymous survey. The overall profile of respondents was similar in both versions, with 60% of the respondents being journal editors and 18% working for publishers in 2021. Perhaps it is, therefore, not surprising that the responses to several of the questions asked were nearly identical to the previous survey; these included how important diversity and inclusion in peer review, to whom diversity and inclusion apply, and definitions of peer review diversity.
COPE received notable responses, however, to the question about the extent to which one’s publication/organisation has achieved diversity. From 2018 to 2021, respondents reported a 9% increase in journals/publishers providing in-house training to promote diversity and inclusion in peer review, as well as a 7% decrease in respondents who thought their journal/publisher had achieved the ideal level of diversity in its peer reviewer pool. Two new reply options to this question also garnered significant responses, with 69% of journals/publishers actively working toward increasing the diversity in its peer reviewer pool and 29% of respondents reporting their journal/publisher had achieved an ideal level of diversity in its peer reviewer pool.
Detailed below are other significant shifts in response to the kinds of diversity seen as being important in peer review. Two of the fifteen new reply options to this question in the 2021 survey also garnered significant responses, with 67% of respondents indicating that Career Level was important and 63% indicating that Geographic Diversity was important.
- The importance of Expertise Areas/Specialities decreased by 27% (from 88% in 2018 to 64% in 2021).
- The importance of Sex/Gender Identity increased by 14% (from 77% in 2018 to 88% in 2021).
It is also worth highlighting some of the free-text responses received. The replies to a question about what changed with respect to diversity, equity, and inclusion in their editorial policies/practices or organisation over the past 3 years ranged from “nothing” to “unconscious bias training.” Finally, several respondents provided general comments, including:
- “Diversity must be promoted and achieved without compromising the integrity of the process.”
- “Diversity an [sic] inclusion in the process is essential in a global and diverse world.”
- “The key issue in peer review is fair objective evaluation of submitted work.”
- “Only knowledge and qualifications matter”
- “We may never have a fully representative review panel but at least we can understand their biases.”
These survey results indicate an overall positive change over the past 3 years in attitudes, policies, and practices about diversity and inclusion in peer review. Not surprisingly, however, there is still more for publications and organisations to do. Meaningful and lasting change takes time, of course, and we want to thank those who not only participated in this survey but also contribute to ongoing efforts to make the peer review process more diverse, equitable, and inclusive for everyone.
Questions for discussion at the Forum
With the ultimate goal of eliminating bias in peer review, we hope you will share during the COPE Forum some of the practical steps you are taking to reduce bias.
- How does your publication/organisation attempt to manage the effects of bias in peer review?
- How is your publication/organisation working to invite a more diverse range of editors and peer reviewers?
- Does your publication/organisation ask peer reviewers to ensure not only that the correct references are cited but also that the references reflect a diverse range of authors?
The Forum discussion summary is not considered formal COPE policy but may be used to develop other types of COPE guidance.
Bias in peer review presentation and quick polls
Bias in peer review presentation
We asked the Forum members two quick poll questions at the beginning of the Forum:
Comments from the Forum, October 2021
NOTE, Comments do not imply formal COPE advice, or consensus.
- Often journals say that their editorial boards have proportional representation approaching or matching that of their discipline, or of the society. But rather than using it as a benchmark, perhaps journals can lead in this area, especially if the society or discipline is not as diverse as it could be.
- One of the difficulties in evaluating diversity in journals in terms of reviewers or papers being published is that journals do not always know the characteristics of their reviewers or authors. It might be easy to evaluate geographic diversity, but the gender or ethnicity of reviewers is not easily identifiable unless that information is appropriately captured.
- Data will tell us if we are making improvements, if diversity is increasing, and if our processes are improving and becoming more inclusive. But we cannot obtain this information unless people are willing to share their characteristics. However, there are many issues around how journals might collect and store that information.
- It is the outcome that matters; do we have evidence that bias in peer review is decreasing?
- At the COPE Virtual Seminar, Holly J Falk-Krzesinski, one of the speakers in the session on diversity, equity and inclusion, noted that when collecting data, participants provided their gender on the reviewer intake forms, for example, but if asked about other issues, less and less information was provided. Holly also noted the challenges around defining categories in a way that works across different regions and cultures.
- For most journals there is room to improve diversity in their reviewer pools and on their editorial boards. How much responsibility should editors and publishers take on in being allies for underrepresented groups? What could be helpful is for editors and publishers to take leadership roles in this area until these underrepresented groups have their own voice.
- Obtaining data on diversity is a challenge. In terms of increasing diversity in reviewer pools, some have work underway to capture data on reviewers from ScholarOne. It is important to ask the right questions to obtain the correct type of data, and great care needs to be taken around the terminology used. Also, a journal cannot mandate that reviewers provide this information. Confidentiality and GDPR issues make it difficult to obtain these data, and without them it is difficult to measure any improvements in diversity.
- One way of improving the diversity of the reviewer pool is to have a diverse editorial board. If editorial boards are as diverse as possible, this will have a beneficial effect on obtaining a broader and more diverse pool of reviewers.
- One of the challenges is that we do not have the data, but do we not therefore take action? It is clear that there are systemic issues around research and academic publishing that create barriers for people who come from historically marginalised groups. We need the data, and we need to know where we are now so that we know how we are improving, although obtaining the data is challenging. In the absence of data, do we do nothing? What is encouraging is that many journals and publishers are trying to change and challenge their own unconscious biases, to encourage wider participation via their commissioning and through their editorial boards. COPE has provided tips on diversifying editorial boards.
- There is a real opportunity to change the assumptions built into our infrastructures. So, for example, in ScholarOne, what are the fields that are collected for users? In research co-produced with indigenous communities, for example, reviewers may be community members who do not have an academic background. However, ScholarOne cannot record this type of affiliation, if it is not an institution. So, how do we collect these data? With the increased emphasis on the community benefit of science, data on this source of diversity also needs to be collected and journals need to provide instructions on how to do this.
- How have people been collecting data on what is representative for their disciplines? In some countries there are data available looking at higher education institutions and disciplines of study, and some publishers and journals have also tried to look at these data.
- Is unconscious bias guidance for editors something that COPE can help with? COPE will shortly be publishing a discussion document on DEI, and further materials will be developed in due course.
- What do we mean by bias and diversity? Should we think more about representation? For individual papers that are being reviewed, are we getting the right mix of reviewers for that particular paper? Are we using people who are experts in this area? For example, if there is a patient population that is largely African American, are we using African American reviewers?
- Even if you do have a broad range of reviewers, they can still have unconscious biases. A broad mix of reviewers, in terms of ethnicity and gender, could still have an academic bias against an author, for example.
Related resources
- Diversity and inclusivity discussion document, COPE guidance
- Driving diversity, equity, and inclusion to shape the future of publication ethics, COPE Seminar 2021
- Diversifying editorial boards, article, 2021
Your comments
Anyone (members and non-members) can comment on this discussion.
Comments are reviewed and, on approval, added below.
- Login to your account or register
to post comments
About this resource
Full page history
-
18 February 2022
Added comments from the Forum
Comments
to post comments
I see that making the reviewers' name public on the article after being accepted can play a role in reducing intentional bias in reviewing. This can only help the situations when the reviewers are non-rationally biased to support the article. In some cases, it would be hard for the journal to discover such intention. However, the community or institutions of the reviewers and authors can easily find if there was a bias in the review process when the name of the reviewers are known. Some journals have this policy already. As the reviewers know that their name will be published with the article, they will be more conservative and may refuse to review from the beginning if a known relationship with the authors is defined in their community.
to post comments