Peer review models

The worsening situation involving, as described, "A growing shortage of skilled reviewers, organised fraud in the form of peer review rings, fake papers and manipulated results,..." can no longer be held in check by an unpaid police force, with prosecuting counsel acting pro bono. Instead, peer review needs to become professionalised, in the sense that noticeable sums of money change hands, in return for high quality (hence time-consuming) reviews. This might also mitigate the "growing shortage of skilled reviewers". In no other profession would practitioners donate (or be expected to donate) their hard-won expertise and personal time in order to sustain a central pillar of the entire enterprise.

All research is essentially either aimed at consolidating/developing existing knowledge or disrupting/overturning it. It is commonly accepted that publications of all research are endorsed through peer review. However, the academic community often neglects a significant discrepancy between consolidating research and disruptive research. For consolidating research, there are often many people working on the existing knowledge, who may be peers of the author. However, for disruptive research, there are also many people working on the existing knowledge, who may be competitors of the author of the disruptive research. Furthermore, since disruptive research creates new knowledge, there may not be any peer to review it. The current peer review process is using “peers”, who are advocates of existing knowledge and essentially competitors of the author of disruptive research, to review disruptive research and determine its publication, which is unscientific.

Most researchers are likely familiar with the recent report by Park et al. (2023) (https://www.nature.com/articles/s41586-022-05543-x) and the blog post by Mastroianni (2023) (https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review), which discuss the role of peer-review in academic publishing. While the former suggests that peer-review does not necessarily increase the number of disruptive research, the latter argues that it can stifle innovative ideas and is primarily aimed at reinforcing established knowledge. Both writings raise concerns regarding the potential misuse of peer-review in evaluating disruptive research. In reality, journals typically require authors to identify opposing reviewers who have conflict of interest in the research being published. However, this approach is not always feasible for authors of disruptive research who may struggle to find reviewers without conflicts of interest. In addition, advocates of established knowledge may use their positions in the peer review process to block publication of disruptive research, further exacerbating the problem. Unfortunately, the current peer-review system does not adequately address these issues.

A recent Chinese blog post discussed rules and their exceptions (https://blog.sciencenet.cn/blog-612874-1390958.html). The author recounts a case that has become a hot topic on the internet. A crowded restaurant requires diners to queue in an orderly fashion, but a pregnant woman is given priority by the staff. Some argue that this violates the rule and is unfair to those who are waiting in line. However, a public investigation found that most people believe this exception for pregnant women is reasonable or tolerable. The author suggests that rules should generally be followed, but if an extraordinary circumstance arises, breaking the rule may benefit the public and prevent misfortune. For example, while cars must follow traffic rules such as stopping at red lights, if an emergency situation arises, the car behind may need to pass the car in front, and running a red light may be permissible in some cases. These are the amended rules, and the author has provided more examples of corrections. When taking a bus and where no designated seats are available, passengers abide by the “first come, first serve” rule to obtain seats. However, in consideration of older, weaker, sick, and disabled passengers, priority should be given to them. To address this need, special seats have been designated for such people. A queue is formed at train stations to purchase tickets, but priority is given to servicemen. Universities use exam scores as a criterion to accept or reject applicants. However, in exceptional cases such as remarkably talented athletes, the university may make corrections to this rule. In reality, these corrections already form part of the rules. At end, the author cited an ancient tale: a man and a women meet at a narrow bridge, the women then obeys social tradition (i.e., mores) not passing the bridge so as not to contact the man’s body. Shortly, there comes floods, the man hugging the bridge’s pillar was drowned.

All research (consolidating and disruptive) need to be published, this is the foundation of science. If the current peer review process persists in rejecting necessary corrections, it could have dire consequences. Therefore, I would like to propose a reform of the peer review process as follows:

1) All research submitted to journals is initially divided into two types: consolidating and disruptive. When submitting their research, authors are required by the journal to define the type of their research (they are aware of which category their research belongs to). If the definition is incorrect, the submission will be returned to the authors. Different review processes are applied to each type of research.

2) For any consolidating research, the current peer-review process remains.

3) For any disruptive research, a new review process is developed. Disruptive research must differentiate itself clearly from consolidating research through four sections: a) an introduction of existing knowledge; b) evidence proving the existing knowledge or its primary part to be incorrect; c) newly proposed knowledge; and d) validation of the newly proposed knowledge. When such research is submitted, the editor considers three reviewers: one who is an expert on the existing knowledge, whose role is to evaluate the author's introduction of the existing knowledge for completeness and check whether the author's evidence (the existing knowledge or its primary part being incorrect) is technically valid. The other two reviewers are experts in a wider knowledge area, whose role is to evaluate the author's evidence (the existing knowledge or its primary part being incorrect), check whether the author's newly proposed knowledge is scientifically valid and technically tested, evaluate the author's validation, and assess the research's significance to the scientific community. The editor uses the reviewers' comments to make a decision. The assessment of the author's validation should be moderate because a disruptive research in its early stages usually performs weakly.

Hello, I was told that this would be the best place to make a suggestion for an update to the COPE guidelines. I believe that, when publishers overinvite (i.e. invite more than 2 reviewers to referee a paper at once), they start to exhaust the already limited pool of reviewers. This is especially true if more than 2 reviewers agree to the task, since this creates redundant work.

So I am suggesting that if all publishers agree to limit the number of concurrent invitations to reviewers to 2 (and to 1 in cases where a review has already been received), then there will be more available reviewers to find.

Of course, there are cases when more than 2 reviewers are required, but its just a guideline and those cases would be the exceptions.

I wrote a related blog post here: https://medium.com/p/31fac0bbe998 In short: it's less work to invite fewer reviewers, and if everyone does it, it could lead to better and faster peer-review.

I hope that's a helpful suggestion!