Executive Summary

Key numbers

Four cells. The first two are how far a repeated review moves. The second two are the price of trying to settle that movement.

Source: Zongrui Yang et al., "Can We Believe AI Peer Review?" (arXiv:2610.19339, 14 October 2026), Tables 1, 3, 4 and 7.

The papers come from ICLR 2025 and ICLR 2026, a hundred from each, 200 in all. They were not picked at random. The sample was stratified on the official subject area the conference assigns and on the band the human reviewers' average score fell into. The scale is the one ICLR uses, 1 to 10, where 1 is a strong reject and 10 a strong accept.

Three settings sat in the reviewer's seat. GPT-5.6-Sol ran at medium reasoning effort, GPT-6-Luna at low, and GPT-6-Sol also at low. Each setting reviewed each paper ten times, which comes to 6,000 reviews in this base experiment alone. Neither the prompt nor the paper changed. The only thing that changed was which run it was.

▲ Table 1 of the paper. The AUC column reads how well the ten-run average separates the papers that were accepted from the ones that were rejected.

Two things sit side by side in those columns. The share of papers that held one score for all ten runs tops out at 56.5% in the best setting, and the other two land at about three in ten. Yet the column right beside it, the spread, is packed inside a single point. No model called a paper an 8 one time and a 3 the next. It kept changing its mind between a 4 and a 5.

The rank correlation between the ten-run average and the human reviewers' average score stayed between 0.346 and 0.409. The AUC for predicting acceptance runs from 0.648 to 0.714. That is not a machine reading by a wholly different standard, and it is not one that overlaps enough to stand in for a person.

A question closer to an actual decision sits behind the single score. When two papers are set side by side and one of them is called the better, does that order survive a second review? The researchers split the review results into two halves and counted how the same pair of papers came out on each side. The figures for GPT-5.6-Sol run like this.

▲ Rank-consistency figures from the paper, for GPT-5.6-Sol. In the other two settings, moving to five reviews lifts order agreement to somewhere between 74% and 86%.

One common reading can be cleared away here. Calling a single review a coin toss does not match this table. The order actually reversed in 1.15% of pairs. The trouble with a single review is not that it returns the wrong order but that it returns no order at all. In 36.71% of cases, one pair in three, the two papers tie. Ranking 200 papers on ten whole-number rungs from 1 to 10 leaves plenty of them standing on the same rung.

Five reviews give the average a decimal place, and ties loosen to 13.38%. Not every freed pair settles back into its original order. Reversals climb alongside, from 1.15% to 2.15%. More reviews put an order on more pairs, and some of those orders point the other way from last time.

Lining the three settings up adds one more thing. Ties after a single review come to 36.71% for GPT-5.6-Sol, 35.60% for GPT-6-Luna and 42.14% for GPT-6-Sol. The setting that looked steadiest in Section 1, with its 56.5% repeat agreement, is the one that stalls most often when two papers have to be told apart. Five reviews leave it at 25.08%, while the other two drop to 11.85% and 13.38%. Its reversal rate runs the other way, lowest of the three at 0.34% after one review and 0.79% after five. Returning the same score on one paper and giving different papers different scores move separately in this table.

With ties this common and scores this loose, one fix comes to mind first, which is to cut the scale into smaller pieces. The researchers tried it for real. They built a protocol that required a decimal place on every score, and had each model review the same 200 papers five more times.

The intended effect arrived. The standard deviation for GPT-6-Luna fell from 0.390 to 0.329, down 15.6%, and GPT-6-Sol went from 0.286 to 0.249, down 13.2%. A second number landed in the same table, one nobody had asked for. The mean score rose by 0.693 points for Luna and by 0.266 for Sol. Both 95% confidence intervals clear zero by a wide margin.

What a score means has changed. A 4.8 under decimal scoring is not the same place as a 4.8 under integer scoring. An organization that fixed its cut-off at 4.5 and then refined nothing but its scoring scale will find a different group crossing a line it never touched.

The second experiment copied the way AI review actually gets used. Review comments come back, the manuscript is revised, the revision is scored again, and the number is checked for a rise. Three roles were split apart. One writes the edits, one picks which of them to apply, and one scores the original and the revision. GPT-6-Luna and Claude Haiku 5.5 took turns in each seat. Of 9,327 candidate edits, 8,459 were approved by both selectors, and 199 papers had their original and revised versions scored three times each. One of the 200 dropped out because it carried 224 figures, past the image limit on the Claude side.

▲ Table 7 of the paper in full. Eight role combinations are possible, and the one where Haiku writes the edits, Luna selects them and Haiku scores never entered the experiment.

The first row and the third row make the situation plain. The manuscript is the same and the edits are the same. The only difference is the model doing the scoring, and one of them hands over 0.474 points while the other takes 0.348 away. That is a gap of 0.822 points. Who selected the edits barely registered, since the second and fourth rows land within a hair of the first and the third.

Down the columns, the difference sits with the scorer rather than with the edits. All four rows scored by Luna went up, whoever wrote the edits. Luna gave its own revisions +0.474 and +0.472 points, and added +0.302 and +0.281 points to revisions written by Haiku. In the three rows scored by Haiku, Luna's revisions were cut both times, and only Haiku's own cleared zero, at +0.050 points. The lower end of that interval is 0.000.

Counting directions instead of amounts leaves the mismatch where it is. Sorting the 199 papers into up, unchanged and down, Luna's scoring raised 120 and lowered 25. The same edits under Haiku raised 32 and lowered 104. The two scorers pointed the same way on 28.1%, 30.2% and 41.2% of papers, depending on the setting, and those figures count the papers both of them left unchanged.

The interesting part is how well the two models agree on the original papers. The correlation on original scores is 0.804, and correcting for the noise in repeated review lifts it to 0.843. On the score change that revision brought, those same two models fall to 0.415 and 0.407. Corrected, the pair reads 0.521 and 0.518. For revisions Haiku wrote and chose for itself the figure drops to 0.085, with nothing left to call a relationship, and that row is the only one left without a corrected value.

The researchers did not stop at naming the problem. They priced it. How many more reviews does it take to state a paper's mean score within a chosen margin at 95% confidence? The numbers below are additional reviews, with the first ten set aside.

▲ Table 3 of the paper. The observed-variance columns take the variance measured across the ten pilot reviews as it stands, and the conservative columns use the 95% upper bound on that estimate, allowing for the uncertainty in the estimate itself.

Even the steadiest setting needs six more runs to guarantee a margin of ±0.5 points. Guarding against an underestimated variance takes twenty-four. Halving the margin multiplies the count by four. All of this covers a single paper, and every review carries a token bill.

The practice these figures aim at is clear enough. Most AI scoring today uses whatever one run produced. Table 3 converts that one run into a number that says how much information it holds. What the authors press is less a recommendation to run more reviews than the point that no threshold can be drawn while the error on a single score is unknown.

The stage here is a conference, and the structure is far more common outside one. Screening applications, reviewing supplier documents, assigning a quality grade to a dataset: the same machinery is already in place. A résumé stands where the manuscript was, an internal quality grade where the ICLR score was. In that seat, one number draws the cut-off line and sets the ranking.

The paper does not show that the model is wrong. It shows that the same input does not return the same output. In the vocabulary data quality has used for a long time, that is a reproducibility problem. Human review comes with a step that measures agreement between reviewers, and a low number there sends the standard back for a rewrite. When AI takes over the scoring, that step can quietly go missing. An expectation that a machine will be more consistent than a person has taken the place of checking.

This is the third article here to take up the reliability of AI review. The two before it were a detection study estimating that 21% of the 76,139 reviews at ICLR 2026 were AI-generated and an experiment on 4,080 manuscripts whose prose was rewritten while the content stayed fixed, where AI review scores moved anyway. Those two asked who wrote it and how it was dressed. This paper reaches a question that sits in front of both. If nothing is changed and the question is put a second time, does the same answer come back?

The authors record their own limits. The budget calculation holds for a fixed model version, prompt and service environment, so a change of setting moves the target with it. The human scores used for comparison come from public records, and ICLR 2025 and ICLR 2026 did not share the same evaluation criteria. The revision experiment ran on two models only, and whether the edits preserved the facts never went through a blind check by a person.

The homework this experiment leaves behind stands all the same. For an organization scoring with AI, the cheapest check available is feeding the same input twice. Ten runs are not needed. The difference between two scores from two runs is enough to show how thick the line currently in use really is.

Thank you for reading this far. The full paper is on arXiv. Does a score your team assigns with AI come back the same when it is measured twice? If you have ever measured it, we would be glad to hear what came out.