During last March ICML (International Conference on Machine Learning), the academic conference dedicated to machine learning (machine learning) oldest in the world, rejected 497 scientific articles at once after detecting that 506 reviewers had resorted to the artificial intelligence (AI) to write your evaluations. They had violated a rule which they themselves had agreed to respect.
This conference is organized by the International Machine Learning Society (IMLS), a non-profit organization, and has been held annually since 1980. Every year, researchers working in the field of AI submit their scientific papers in late January or early February to ICML. Those papers They are reviewed by a committee made up of other researchers in this field with the purpose of evaluating them and publishing them if they finally pass a thorough review that normally lasts several months.
Decisions to accept or reject articles are usually communicated to authors during the month of May, and the ICML conference is usually held in July. Publish in ICML, NeurIPS (Conference and Workshop on Neural Information Processing Systems) or ICLR (International Conference on Learning Representations) is equivalent to what in other disciplines it would be to publish in the scientific journals Nature or Science. But ICML has a serious problem: its authority is being questioned in r/MachineLearninga Reddit community specialized in machine learning which has more than 2.5 million subscribers.
A perversion where reviewers don’t have time to review
Before moving forward, it is worth stopping at a very important milestone: the number of scientific articles received by ICML is growing overwhelmingly year after year. In 2023 it received 6,538 papersand in 2024 no less than 9,653 articles, which represents a growth of 48%. The root of the problem lies in the fact that the number of qualified reviewers is not increasing with the same rhythm as the number of scientific articles that need to be evaluated.
As I mentioned a few lines above, ICML rules establish that reviewers cannot lightly resort to AI to carry out their evaluations because this procedure can introduce bias. In fact, a study carried out on ICLR 2024 has revealed that scientific articles evaluated with AI models They tend to receive higher scores than those reviewed with the conventional procedure. This is the problem. For the 2026 edition, ICML offered evaluators to choose between two policies: one that prohibited the use of AI and another that allowed it, but with conditions. Only those who chose the first option and failed to comply were sanctioned.
Of the 506 offenders, only 398 were reciprocal evaluators who had submitted a ‘paper’
However, there is one relevant fact that is worth not overlooking: the 497 scientific articles that were rejected in March of this year were reviewed by offending reciprocal evaluators. This simply means that they are researchers who simultaneously act as authors and reviewers, so their scientific article was penalized due to their violation of the ICML rules of conduct. Of the 506 offenders, only 398 were reciprocal evaluators who had submitted a paper.
Interestingly, the detection system that ICML has used consists of hiding specific instructions within the PDFs of articles pending review. Those instructions are invisible to a human reader, but any AI model processing the document interprets them and includes specific, trackable phrases in the evaluation. ICML has not used generic AI detectors. Of course, each case detected was manually verified to verify that a violation had actually been committed when preparing the evaluation.
What is happening reflects an unappealable reality: the review system has failed and needs to be rebuilt. The reviewers can’t cope. Neither those of ICML, nor those of NeurIPS, nor those of ICLR. The number of qualified reviewers should grow at the same rate that the number of scientific articles that need to be evaluated, and it is not happening. Furthermore, this scenario has introduced another problem: acceptance or rejection decisions have acquired a random aspect that threatens the consistency and reliability of the evaluations.
It is still not entirely clear what path should be followed to resolve this problem beyond the need to increase the number of qualified evaluators. One option is to improve the transparency of the review process publishing all evaluations. Even those of rejected articles. The evaluation process could also be transformed into a two-way procedure in which authors also evaluate the quality of the reviews they receive. In this way, the evaluators will have a history that will prove their good work. We will see what strategy the conferences finally implement. In 2027 we will clear up doubts.
Image | Charlesdeluvio (Unsplash)
More information | ICML
GIPHY App Key not set. Please check settings