Executive Summary

Key Numbers

Source: Nature Computational Science (24 August 2026), main text and supplementary material

Peer review at AI conferences is buckling under volume. Submissions to ICML rose from 1,676 in 2017 to 12,107 in 2025, and NeurIPS went from 3,240 to 21,575 over the same stretch. The pool of experienced reviewers has not grown at that rate, so conferences lean on graduate and undergraduate reviewers and on language models. How much the outcome wobbles has already been measured. NeurIPS 2021 sent 10 percent of its submissions through two independent committees, and the two disagreed on 23 percent of the accept-or-reject calls. Rerunning review from scratch, that experiment estimated, would change about half of the accepted list.

The question this team asked was not whether review is sloppy. It was whether someone else holds a better signal about the same paper. Authors know the theoretical foundations, the limitations and the long-term promise of their own work better than a reviewer does, and much of that never fits inside a paper or a short rebuttal. The difficulty is getting it out of them honestly.

Ask authors to score their own papers and everyone awards themselves top marks. So the researchers asked for an order instead of a score. An author with several submissions to the same conference cannot call every one of them the best, and which of your own papers is stronger is an easier question than how many points each one deserves. The mathematics for combining such an ordering with review scores was proposed in 2021 under the name isotonic mechanism, and this experiment is the first large-scale test of that theory inside a live conference.

The survey went out on 26 January 2023, immediately after the submission deadline, and closed on 10 February, before reviews were released. It stated that responses would not affect peer review and would not be shared with anyone. It was also not framed as predicting acceptance, personal contribution or future citations. What authors answered was which of their own papers they considered stronger; citations were bolted on afterwards as an external yardstick for whether that ordering carried information. Of 6,538 submissions and 18,535 authors, 5,634 people responded (30.4 percent), and the 1,342 among them with multiple submissions supplied rankings. At least one author ranked each of 2,592 submissions, 39.6 percent of the conference.

Citations were collected through the Semantic Scholar API from July 2023 to November 2024, roughly 16 months. Because titles change between versions and authors get added, papers were matched by edit distance on both title and author list. The final dataset held 797 authors and 1,527 unique submissions, averaging 14.73 citations with a median of 4.

The paper an author placed at the top of their own portfolio averaged 19.99 citations, exactly twice the mean of the one they placed at the bottom (two-sided paired Wilcoxon signed-rank test, P < 0.001). Since that gap could be nothing more than the familiar fact that accepted papers get cited more, the researchers counted again within each decision outcome. Among accepted submissions the top-ranked mean was 27.64, or 2.00 times the bottom-ranked mean; among rejected or withdrawn submissions it was 12.42, or 1.95 times. Strip away the conference's stamp of approval and the author's ordering still worked.

Narrowing to authors with at least three ranked submissions turns the two-way split into a staircase. Among accepted papers, those ranked first, second, third and last averaged 41.57, 19.39, 14.32 and 12.83 citations. Among rejected or withdrawn ones the same positions averaged 9.75, 8.40, 6.34 and 6.18. Every step down the ranking took citations down with it.

Means are easily dragged around by a few outsized papers, so the researchers looked at the upper tail separately. Of the 1,527 submissions, 22 passed 150 citations, and 17 of those had been ranked first by at least one of their authors. Three had been ranked second and five ranked last; one paper was ranked both first and last by different co-authors and one was ranked neither, which is why the parts add up to more than 22. Screening the same 22 papers by pre-rebuttal review score instead of by author ranking catches 14. Those 22 papers are about 1.44 percent of the dataset, close to the share typically selected for oral presentation at major AI conferences.

In the most-cited paper in the dataset, the two signals pointed in opposite directions. Its authors ranked it first among their own submissions, yet its post-rebuttal review score was lower than their other papers and it was accepted only as a poster. It ended with 979 citations, the dataset maximum and nearly three times the maximum for any paper an author had ranked last.

Head to head against review scores, the ranking was the more consistent signal. Post-rebuttal scores were significant among accepted papers (P = 6.66 × 10⁻³), but neither pre- nor post-rebuttal scores were significant among rejected or withdrawn ones. Author self-rankings were significant in both groups (accepted P = 6.00 × 10⁻³; rejected P < 0.001). In a negative-binomial regression controlling for review scores and reviewer confidence, final decisions, portfolio size, review counts and ranking completeness, moving from an author's lowest- to highest-ranked paper was associated with 1.81 times the expected citation count. The 2.00 figure above is a comparison of raw means; this 1.81 is a coefficient after controls.

The ranking survived even when the scores were pinned down. The researchers counted a pair only when both of its papers had review scores inside the same interval, across four bands from the 3s to the 6s. Within those bands the paper an author ranked higher generally still accumulated more citations, and the gap widened in the higher score bands rather than closing. Two papers with the same review score looked identical to reviewers; if their citations diverged anyway, the ranking held information the scores did not.

The alternative explanations fell away one at a time. Excluding self-citations left the gap intact. If the difference came from authors promoting the paper they had privately backed, it should open up after the conference, but in the two months right after ICML opened, accepted top-ranked papers averaged 1.65 citations against 0.69 for accepted bottom-ranked ones. Given that citations usually take months to appear, a gap that early is hard to pin on publicity. Mean arXiv posting dates were 4 March and 8 March, near enough to rule out a head start. The same pattern held when the outcome measure was swapped for Google Scholar citations or GitHub stars.

An author's extra knowledge sits outside the manuscript: how far the experiments were pushed, which assumption is thin, what the next paper can bolt onto this one. A reviewer has to read the manuscript against a deadline, and under that constraint an easily quantified gain such as a benchmark number registers ahead of long-term importance. The paper's own framing is that author judgments and reviewer judgments "may therefore contain different, complementary information."

So can a machine that reads manuscripts well stand in for the author? The researchers pulled 263 within-author pairs of accepted papers and gave GPT-4o mini the text alone, asking it to pick the stronger one. The papers the model preferred averaged 21.69 citations against 19.61 for the ones it passed over. On the same pairs, the authors' own ordering split 27.00 against 13.82.

The researchers limit the reach of this comparison themselves. It runs on 263 pairs, all of them accepted papers, and they write that it "does not establish how stronger models or broader paper pools would perform." Turning it into a verdict that machines cannot replace people runs ahead of the evidence. Still, manuscript text alone did not yield as much signal as the authors held. The paper closes the passage by noting that the result "indicates that both the comparative format and the source of the judgment matter."

Move the experiment outside scholarly publishing and the picture turns familiar. One object carried two labels. Several expert reviewers produced one of them over several weeks; the author produced the other in a few minutes. The resources behind them are not comparable, and the cheap one tracked future citations better.

That is not to say the labeling technique did no work. Asking for an order rather than a score is what pulled honest answers out and what made the question answerable. In a supplementary simulation, asking reviewers for rankings improved score calibration too, though that simulation is a mechanism and sensitivity check whose bias and noise parameters were not estimated from the ICML data. Hence the paper's line that both the format and the source matter. But one comparison held the format fixed and changed only the source. GPT-4o mini compared two papers in exactly the author's format, reading exactly the author's text, and could not reproduce the gap. When the format is identical, what remains is what the labeler knows about the object.

Across 500 holdout splits predicting the top citation quartile, self-ranking indicators reached precision 0.318 and recall 0.307 against 0.307 and 0.297 for post-rebuttal score indicators. Taken as a whole, a thin margin. But the margin shrinks among accepted papers, where self-ranking precision and recall were both 0.261, and opens up among rejected or withdrawn ones. Of the 22 papers past 150 citations, 7 had been rejected by ICML, and 4 of those had been ranked first by at least one author. The authors describe all of this as "incremental predictive information rather than sufficient standalone performance." The value showed up less in sorting what had already passed than in fishing good work out of the discard pile.

Data pipelines have discard piles too: samples a rule filtered out, labels sent back for rework on low confidence, ambiguous cases parked for later. When we revisit that pile we usually reach for the more expensive route, assigning expert annotators or rewriting the labeling guide. What this paper shows is that there is a question to ask before that one. Who understands this data best, and where do they sit in the pipeline right now? And are we collecting their judgment at all?

Of course, none of this means authors should review their own papers. What made honest answers possible was that the ranking was fully walled off from the review, so a response could not touch the responder's own outcome. Keep the maker separated from the grader, and take one extra signal from the maker on a separate line. That is the structure of this experiment.

This has not stayed at the proposal stage. Citing the 2023 findings, ICML 2026 adopted a policy that uses author self-rankings to check review quality. Once pre-rebuttal scores are in, an isotonic mechanism computes projected scores consistent with the author's ranking, and submissions where the projection and the actual scores diverge sharply are placed in disagreement categories visible to area chairs and senior area chairs. The direction of the discrepancy is not revealed. Flagged papers can go to emergency review or to a re-examination of the existing reviews. Roughly 1,000 submissions were flagged as high or very high disagreement, about one per area chair.

In this design the ranking never makes the decision. It only points at which papers to look at again, and the looking is done by people. The cheap signal is used not as a substitute for the expensive one but as a device for deciding where to spend it. The paper puts it as directing "scarce reviewing attention to cases where author information and reviewer scores disagree most."

There is a condition the paper leaves open here. The 2023 experiment got honest answers on the promise that the ranking decided nothing. Once rankings do start affecting decisions, authors' incentives change with them. The comparative format blocks the move of calling every paper your best, but whether that alone is enough remains unanswered, which is exactly why a mechanism that binds rankings and review scores together is needed.

This is an observational study. It cannot say the rankings produced the citations, it covers one year of one conference, and the 30.4 percent who responded selected themselves. The yardstick is imperfect too. A citation records only that a later researcher noticed the paper and found it worth connecting to, and as the authors note, citations "can be positive, neutral or critical" and do not establish that a paper is correct. The same experiment has since run at ICML 2024 and 2025 and NeurIPS 2025, but those papers are still waiting for citations to accumulate.

The authors do not claim self-assessment replaces peer review; they write that it can complement it, and that rankings "should provide targeted side information rather than replace reviewer judgment." The journal's own News & Views piece stays inside the same boundary, summarizing the work as an experiment showing "that when researchers privately rank their own submissions, those rankings forecast future citations more reliably than peer-review scores."

What is left is a sense of cost. The extra resource this experiment spent was one survey, sent the day after a deadline. When we reach for a more expensive labeling route, the option of asking the person who already knows the answer is usually not on the list at all.

Academic

• 1.Su, B. et al. (2026). "Self-rankings as a predictor of scientific impact beyond peer review." Nature Computational Science, 24 August 2026. (Open access, CC BY 4.0)
• 2.AlShebli, B. (2026). "When authors are the best judges of their work." Nature Computational Science News & Views, 2026.
• 3.Beygelzimer, A., Dauphin, Y. N., Liang, P. & Vaughan, J. W. (2023). "Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment." arXiv:2306.03262.