Executive Summary

Handing a paper to an AI reviewer is no longer an experiment. AAAI 2026 attached a machine-written review to every one of the 22,977 papers in its main track. So what does that review actually catch? A team at the University of Cambridge, the Flatiron Institute and elsewhere has proposed one way to measure it. They planted errors whose answers they knew into thirty real papers, then asked twelve AI peer-review systems to go find them. What this article carries over is not the ranking on that scorecard but what pulled the scores apart.

Handed the identical 27-word prompt, the three frontier models came out roughly 50 points apart in total detection. Among those three, the lowest detection rate and the highest stood fivefold apart. Then the prompt was replaced with a long itemized checklist, and the spread narrowed by roughly half. The models that climbed most, though, were the ones that had been weak to begin with, and the fix carried a price. In one configuration that kept the model and the tools in place and swapped nothing but the prompt, the cost per paper jumped fivefold. And the error type caught most often turned out to be the type flagged wrongly most often.

Everything so far is a number from the paper's tables and appendices. "Rewriting the prompt is a way of lifting weak models" and "judges agree only where the question has an answer key" are our readings of those numbers, and the body marks each one where it appears. The argument here is not that AI review cannot be trusted. The authors put results that cut against their own tool in the same tables, ran a self-preference test, and published their costs. The argument is this: even measurement this honest leaves this much unmeasured. The figures below come from arXiv 2611.17052 v1, posted 12 November 2026 and read on 16 November 2026.

48.6 points

Detection spread across three models on the free-form prompt

Astrophysics corpus: 60.5% against 11.9%. On the ML corpus the spread is 52.8 points

24.0 points

The same spread once the structured prompt replaced it

Same corpus, same three models. What narrowed it was the prompt, not a change of model

5× cost · 6× time

What the prompt change alone added per paper

Measured in the Claude Code configuration with model, sandbox and tools held fixed. Detection rose 32.2 points

0.90 → 0.32

Judge agreement with an answer key, and without one

Two panels sharing two of their three seats sit at 0.90–0.93 on matching and 0.32–0.58 on false positives

1

Plant errors whose answers you already know

The thing to watch in this benchmark is the question it asks. The researchers took thirty papers that had actually entered the publication pipeline and planted 357 errors in them, each one with a known answer. Then they had twelve AI peer-review systems review those papers and counted how many of the planted errors showed up in the reviews. With the key in hand, "how many did it find" stops being an estimate and becomes a count.

Why that differs becomes clear by comparison. The common way to evaluate an AI reviewer is to measure how closely it resembles a human-written review. Use that ruler, though, and the errors humans missed stay invisible forever. The paper's introduction names the gap directly: evaluation has mostly been framed as "alignment with human-written reviews" or run as error injection on ML papers alone, leaving out questions such as "false-positive rates or error detection as a function of the error severity". False-positive rates and detection by severity are the two cells this benchmark sets out to fill.

1.1Thirty papers drawn from a single day of submissions

There are two corpora. The astrophysics corpus is the 15 papers no longer than 20 pages among the 24 submitted to arXiv astro-ph on 22 April 2026, with a median length of 13 pages. The ML corpus is 15 papers drawn by stratified random sampling across eight subfields from the 74 eligible cs.LG submissions of 15 July 2026, median 9 pages. Both carry the same condition: one day's submissions, 20 pages or fewer. Longer manuscripts and other days are not represented by these results.

Splitting the corpus across two fields is the heart of the design. AI review benchmarks so far have covered ML papers almost exclusively, which means they measured only on territory the evaluated models already know well. The authors describe their astrophysics run as "the first controlled comparison of AI peer-reviewer systems on a real, non-ML scientific paper corpus". As the rest of this article shows, the difference between the two corpora explains more here than anything else.

1.2Four error types, 357 injections, three severity levels

The planted errors come in four types — citations, numerics, mathematics and general issues — for 357 across both corpora. The uneven split is deliberate. Numerics and mathematics carry 119 each, the most of any type, and general issues the fewest at 29.

Error type Astrophysics ML Total What it touches
Citations454590Makes a cited paper say something it never said
Numerics5960119Alters numbers in the tables and the body text
Mathematics5960119Drops a term from an equation or swaps an operator
General issues141529Puts the abstract at odds with the body, or the method with the results
Total177180357

Injection counts by error type (paper §2.2). Each paper got three injected copies, and each copy carries about four errors.

A severity ladder sits on top of that. Every error comes at one of three levels: subtle (S1), clear (S2), blatant (S3). For a citation error, subtle shifts a quoted value by one or two error bars, clear deletes a qualifying clause, and blatant reverses the direction of the conclusion outright. For a numerics error, the three rungs are a second-decimal drift, a swapped digit, and percentages that add to more than 100. The ladder is what lets you ask back what "catches 80%" actually means. The strongest systems caught 57–73% even at the subtlest level, while the weakest configurations failed to clear 5% even at the most blatant.

One caveat rides along. Severity labels were calibrated within each error type, not across types. A blatant mathematics error and a blatant citation error are not claimed to be equally hard. No sentence comparing severity across types can be written from this data.

1.3Fifteen clean copies ride along

The most valuable part of this design is not the planted errors but the papers left alone. The researchers ran 15 clean copies, with nothing injected, through the same systems. Every flag raised there is a candidate for a wrong call. The share of flags on clean copies judged to be clearly wrong is what this article calls the false-positive rate.

A ruler that measures detection alone and one that measures detection alongside false positives see different things. With the first you cannot tell a checker that catches a lot from a checker that flags a lot wrongly. Surveying what prior benchmarks do and do not cover, the authors write: "No prior work grades severity levels or uses a cross-family adjudication panel; only one covers papers beyond ML/NLP; only one provides false-positive control; and only one integrates mechanical verifiers. This work is the first to combine all five." Read the other way round, that also says only one prior study had false-positive control at all.

1.4Why the measurement is needed now

If AI review were still confined to the lab, this measurement could wait. It is already out. The AAAI 2026 pilot covered the whole main track, 22,977 papers, and finished in under a day. The team reporting that pilot writes that in surveys of authors and program committee members, participants not only found the AI reviews useful but preferred them to human reviews on axes such as "technical accuracy and research suggestions". That is a survey result, not a measurement of accuracy. But the very axis on which people felt the reviews were technically accurate is the one this benchmark puts a number on, and the number says 50 points of spread.

The background pressure points the same way. arXiv took in 40,363 submissions in September 2026 alone, a monthly record, and from 1 October capped each author at two submissions per month. The reviewing side is no different: as this blog reported earlier, 21% of the 76,139 reviews submitted to ICLR 2026 were classified as entirely AI-written. That story is about AI occupying a seat meant for a human; this one is about what the AI in that seat actually catches. Two problems with the subject and object swapped, standing side by side in the same year.

1.5The method has an ancestry

Fact. The paper names its own lineage. "Our 'dose-response' design extends perturbation-based behavioural evaluation with graded severity levels", it says, citing the CheckList family of behavioral tests for NLP models as an ancestor — perturb the input a little at a time and watch how the model's behavior shifts.

Our reading. There is an older method with exactly this shape, from software engineering: mutation testing, proposed in 1978. Plant small faults in a working program on purpose, count how many of them the existing test suite catches, and score the suite by that number. The share of faults killed is the grade the tests receive. For nearly half a century it has been used to test not the program but the apparatus that tests the program, and it has now come back in the same shape to test AI reviewers. The data world uses the same idea: this blog covered validation rules exercised against deliberately corrupted batches. The name mutation testing, though, is our analogy rather than the paper's word.

2

One prompt, one in ten against six in ten

Same single sentence, same papers — and one system flagged one error in ten while another flagged six. The prompt handed to the three frontier models was a 27-word sentence, written the way you would ask a colleague: read this paper and note any errors you find. Under that condition, total detection on the astrophysics corpus was 60.5% for GPT-5.5, 36.7% for Claude Opus 4.7 and 11.9% for Gemini 3.1 Pro.

The ML corpus reproduces almost the same ordering and the same gaps: 63.9%, 43.9%, 11.1%. The paper puts the gap in a sentence of its own — "three frontier LLMs differ by up to 48.6 pp in total detection on astrophysics … and by 52.8 pp on ML" — and the abstract compresses that to "about 50 percentage points". What looked like a choice of review tool was in fact a choice between catching one error in ten and catching six.

Model (free-form prompt) Astrophysics total detection ML total detection
GPT-5.560.5%63.9%
Claude Opus 4.736.7%43.9%
Gemini 3.1 Pro11.9%11.1%
Highest − lowest48.6 points52.8 points

Only the three single-LLM rows with the free-form prompt, taken from Table 2 of the paper. The denominator is every error injected into that corpus (177 for astrophysics, 180 for ML). The two figures on the bottom row are stated in the paper's own text.

Where the gap opened is also a single line in the paper: "Mathematics and numerics drive this discrepancy." What pulled the models apart was mostly mathematics and numerics errors. Those two types return in section 4. They are the types with the highest detection rates, and also the types where wrong flags pile up.

This table says how many each system caught; it says nothing about which ones. Appendix L measures that separately. Line up the per-injection hit-or-miss records of six reviewer systems on the astrophysics corpus and the pairwise correlations scatter between 0.2 and 0.54. The authors' summary: reviewers with similar overall detection rates catch visibly different sets of errors.

The strangest pair is the one where both members have topped a column. The authors' multi-agent system and the CLI agent with the structured prompt sit at 0.27 overall, and at about 0.04 on citations and numerics. On precisely the two types where each is the best of all systems, the errors they catch barely overlap. Two systems built nothing alike, meanwhile, rise to 0.48 once they share a prompt — that is a single call and a tool-using agent loop running the same instructions.

A caveat is attached: 0.04 does not mean the two run in opposite directions. Two systems detecting 80% and 91% of the numerics errors already coincide on about 75% of them for that reason alone. A correlation near zero means they overlap no more often than those rates imply, not that they disagree. Zoom the grain out to whole papers and the picture changes again. Across the ten systems whose detection rates are not near the floor, the mean correlation on which papers are difficult is 0.39. Reviewers broadly agree on which manuscripts are hard, and go their own way on what to flag inside them.

2.1Strip the condition and the sentence turns false

The 50 points carries a condition, and that condition drives the rest of this article: when all three models were given the same one-sentence prompt. As section 3 shows, change the prompt and the gap between those same three models halves. Lift out "models differ by 50 points" on its own and you have built a false sentence from a true number.

The systems themselves are also built very differently. Twelve reviewer systems ran across the two corpora: six single-LLM configurations differing in prompt, two CLI agent configurations, the authors' own multi-agent system, and three external reviewers taken off the shelf. One of those ran on astrophysics only, which is why the ML corpus has eleven rows. "All twelve systems were run on both corpora" is a false sentence.

2.2What happened when humans sat the same exam

Planting errors and counting how many get found predates AI. Two classic experiments on human reviewers have exactly this shape. A 1998 study in JAMA planted eight errors in one manuscript and sent it to 420 reviewers; among the 221 who responded, the median number found was two out of eight. Not one reviewer found more than five, 16% found none at all, and 33% recommended acceptance with the errors still in place. A randomized controlled trial of 607 reviewers published in the Journal of the Royal Society of Medicine in 2008 found an average of 2.58 of nine major errors, and training did not raise it.

As rates, those come to 25% and 28.7%. The weakest configurations in this benchmark (11.1–11.9%) fall below that and the strongest (70.6–81.1%) above it. It is not a direct comparison, though. The error types differ, the fields differ, the denominators differ, and above all this benchmark did not measure a human baseline of its own. What these two numbers support is only that the same design, run on a different population, produced those figures.

The human numbers carry one more layer. In a randomized controlled trial of 238 reviewers at two orthopedic journals, reviewers who recommended acceptance found 17.3% of the planted errors while those who recommended rejection found 39.1%. Reviewers look less hard when they like the conclusion. Whether a machine reviewer carries a conclusion preference is a question this benchmark did not ask, but on the human side it is already measured that how you read matters as much as who reads.

2.3What we measured before, and what gets measured here

This blog has covered AI peer review four times, and all four measured an indirect signal. The closest of them is worth separating from this one. Asking whether an AI gives the same paper the same score twice measured reproducibility: does it return the same answer? Even the most consistent model stopped at 56.5%. What gets measured here is accuracy: is the answer right? The two axes are independent. A system can be perfectly consistent and consistently wrong, or unsteady and still land on the right spot.

A third axis has been measured too — AI reviewer scores moved across 4,080 papers when only the prose was rewritten. Reproducibility, rhetorical sensitivity, accuracy. Of the three, accuracy is the one practitioners ask about first, and it was also the one with the fewest published numbers.

3

A new prompt cut the spread between models in half

Hold the models fixed, change only what you tell them, and the 48.6 points from section 2 drops to 24.0. On the ML corpus 52.8 points becomes 30.0. Both numbers come straight from the paper's body. Roughly half. Choosing a model and writing the instructions sit on opposite pans of the same scale, and the second pan is heavier than it looks.

The replacement here is the structured prompt: a 5,946-word document setting out, item by item, what to check in what order and what rises to the level of a flag. Against the 27 words of the free-form prompt, that is roughly 220 times longer.

3.1The models that climb most are the weak ones

A narrower gap does not mean both ends moved toward the middle. The bottom came up far more. Gemini 3.1 Pro gained 25.4 points on astrophysics and 40.0 on ML. GPT-5.5, given the identical prompt, gained 0.8 points on astrophysics — effectively nothing. The authors give the reason in a phrase: the structured prompt narrows the cross-provider spread "by boosting the performance of the weakest models".

Putting the two corpora side by side is what makes that precise. On ML, Gemini 3.1 Pro gained 40.0 points, Claude Opus 4.7 gained 28.3 and GPT-5.5 gained 17.2. On astrophysics, in the same order: 25.4, 16.6 and 0.8. Both corpora agree that the gain runs in order of prior weakness, but only one cell deserves to be called flat, and it is GPT-5.5 on astrophysics. On ML even the strongest model picked up 17.2 points. The sentence that holds across both corpora is "the weaker the model, the more it gains" — not "a good model gains nothing from a better prompt".

Our reading. Rewriting a prompt is not a uniform push applied to every model. It is closer to making a model that already knows how do something it was not doing. Adding a good prompt to a good model does not double anything, because the model was already there. On a weak model the same prompt triples the number.

3.2Model and tools fixed, prompt swapped

The cleanest isolation of the prompt effect is the CLI agent configuration. The model (Claude Opus 4.7), the sandbox, the list of available tools and the arXiv access were all held fixed, and only the prompt document was swapped. Detection rose 32.2 points on astrophysics and 38.9 on ML. The authors note that the same swap "triggers extensive tool use": once the instructions changed, the same model started using its tools in an entirely different way.

What changed is in the execution logs. With the structured prompt the agent ran a median of 9 turns and as many as 26, and wrote and executed 21 multi-line Python scripts over the astrophysics corpus. With the free-form prompt it read the PDF and answered, writing no scripts at all — though the authors add that it knew it could. One tool neither configuration ever reached for was the computer algebra system, even though the structured prompt encourages its use. Same model, same permissions, and changing only the instructions changes tool use qualitatively; but instructions that recommend a tool do not guarantee it gets used.

Four conditions are worth restating. Inside the CLI agent, with model, sandbox, tools and arXiv access held fixed, changing only the prompt. Carry "changing the prompt adds 32 points" out of that box as a general statement about all systems and it becomes false.

3.3There are three prompts, not two

Read only this far and the lesson looks like "write longer prompts". An appendix experiment breaks that reading. To separate the two jobs a long prompt does — telling the model what to check, and telling it how to report — the researchers wrote a 471-word concise structured prompt that does only the first. That makes three prompts, not two: 27 words, 471 words, 5,946 words.

The middle prompt looks like a bargain. In the CLI agent configuration it captures 24.3 of the 32.2 points the long prompt delivers. But false positives climb with it. On astrophysics the three single LLMs land at 17–27% false positives with the concise prompt, against 1–9% with the free-form prompt and 3–6% with the structured one. The CLI agent sits at 16.7%, spiking alone between the free-form run (7.4%) and the structured run (3.5%). Most of the extra wrong flags are mathematics, the authors write.

What the middle prompt does also depends on the corpus. On ML the concise prompt alone pulled the spread between models from 52.8 points to 27.2 — narrower than the 30.0 the long prompt produced. On astrophysics 48.6 points became 48.0, which is to say nothing happened, and the only prompt that narrowed the spread there was the structured one. The same prompt erases most of the model difference on one corpus and leaves it untouched on the other. "Fix the prompt and the models converge" is half true at best without naming which corpus and which prompt.

Follow a single model through all three and the shape is sharp. Gemini 3.1 Pro on the astrophysics corpus: detection climbs steadily across the three prompts, while the false-positive rate spikes fourfold in the middle and settles back down.

One model, three prompts Astrophysics corpus · Gemini 3.1 Pro (paper, Appendix H, Table 11) 0% 10% 20% 30% 35% 11.9 20.9 37.3 7.0 27.3 5.6 Free-form 27 words Concise 471 words Structured 5,946 words Detection — denominator: 177 injections FP rate — denominator: clean-copy flags

The legend spells out that the two lines have different denominators. Detection is how many of the 177 injected errors were found; the false-positive rate is how many of the flags raised against the 15 clean copies were judged clearly wrong. The two lines share an axis without counting the same thing.

3.4The fix carries a price

Money and time move with it. The cost table in the appendix holds the most operational numbers in this article. In the same CLI agent configuration, with model, sandbox and tools held fixed and only the prompt changed, the authors report cost per paper at "∼ 5×" and wall time at "∼ 6×". In absolute terms: $0.33 to $1.73, and 1.4 minutes to 8.7.

Configuration Cost per paper Time per paper
CLI agent · free-form prompt$0.331.4 min
CLI agent · structured prompt$1.738.7 min
Single LLM · free-form prompt$0.45~1 min
Single LLM · structured prompt$0.55~2 min
Multi-agent system (Skepthical)$2.41~24 min

From Table 12 in Appendix M of the paper, for the astrophysics corpus: corpus-wide spend divided by the 60 paper versions. The 24 minutes is wall time for one 12-page astrophysics paper on a single Google Cloud Run worker (1 vCPU, 2 GiB), not an average across systems. The two self-hosted open-weights reviewers carry no API cost and are left out of this table.

What the whole benchmark cost to run tells a second story. Across both corpora the total was $932.82: $623.73 to run the reviews and $264.52 to adjudicate whether each error had been caught. 28% of the budget went to counting whether the answers were right (our calculation from Table 13). Grading is not dramatically more expensive than reviewing, then, but it is nowhere near free either. As section 6 shows, that grading is also the weakest link in this design.

Put two numbers side by side and an easily confused cell comes into focus. The per-paper figures above are the price of running a review once, not of verifying that the review was right. Divide the astrophysics corpus's total benchmark spend by its 15 papers and you get $34.78 per paper (our calculation from Table 13) — roughly fourteen times the $2.41 it costs to run the most expensive reviewer once. The cost of running a checker and the cost of measuring whether the checker worked differ by an order of magnitude. Which is why most organizations hold only the first number.

Among the systems calling external APIs, the authors report that per-paper cost varies "roughly sevenfold" between the cheapest and the most expensive configuration. In absolute terms it looks small: at $2.41 a paper, a thousand papers comes to $2,410. The 24 minutes per review is another matter. For anyone who has to push thousands of submissions through a deadline week, that figure stops being a budget line and becomes a capacity problem.

Three lines for practice. Swapping models is expensive and rewriting the prompt looks cheap. But if you are already on a strong model, rewriting buys almost nothing back (GPT-5.5 on astrophysics gained 0.8 points); a half-rewritten prompt buys wrong flags instead (17–27% false positives on astrophysics with the concise prompt); and even a properly rewritten prompt carries a price — five times the cost per paper and six times the time.

4

The error type caught most is flagged wrongly most

Wrong flags cluster exactly where the scores are best: mathematics. Strong configurations catch mathematics errors at rates in the 70s to 90s, and it is also the type the authors call "the least reliable". Pool the mathematics flags raised by the frontier systems and 30% on astrophysics and 67% on ML were judged false. The same calculation on general-issue flags gives 3%. On both corpora, half of all wrong flags came from mathematics.

Why they cluster there is guessable. Re-deriving an equation is the task a model throws itself at hardest, which makes it correspondingly easy to slip mid-derivation and write the slip up as the author's error. The thing to hold onto is that 30% and 67% are not false-positive rates. The denominator is different. The table below separates the three rates this article uses.

Name Denominator Range in this article
Detection rateEvery error injected into that corpus (177 or 180)1.1% to 81.1%
False-positive rateEvery flag raised against the 15 clean copies1.0% to 59.1%
Share of flags that were falseEvery flag raised of that error typeMathematics 30% and 67%, general issues 3%

All three are percentages and all three count something being wrong. Different denominators make them non-interchangeable. "The false-positive rate on mathematics is 67%" is a false sentence. What is true is that 67% of the mathematics flags raised by the frontier systems on the ML corpus were judged false — and even that is a pooled figure across several systems rather than one system's.

4.1The false-positive rates themselves are mostly low

Fairness first. Measured against the clean copies, false-positive rates were low across the board. Setting aside the two self-hosted open-weights reviewers, every system on astrophysics came in below 9%, and on ML six of nine came in below 10%. The exceptions are the three single LLMs with the structured prompt, which land between 11.5% and 22.7%. The authors' own multi-agent system is low on both: 2.8% on astrophysics (106 flags) and 4.0% on ML (99 flags).

"Most systems came in below 10%" is a true summary; change it to "every system" and it turns false. Six of nine on ML is two thirds, and the other three are in double digits. The two open-weights reviewers were at 18.5% and 59.1%.

Which way false positives move when the prompt changes also depends on the corpus. The authors call the effect "mixed". On astrophysics the spread between providers narrowed: GPT-5.5 came down from 8.7% to 3.4% and Claude Opus 4.7 rose from 1.0% to 5.2%, meeting in the middle. On ML all three providers went up. The only configuration that improved on both corpora was the CLI agent. The long prompt helped detection on either corpus; on wrong flags it did not.

4.2What the lowest false-positive rate turns out to be

Look into the configuration that posted the lowest false-positive rate, though, and there is a twist. The system that came first on astrophysics at 1.0% is Claude Opus 4.7 with the free-form prompt. Here is how the authors describe that group: the lowest rates belong to the single LLMs on the free-form prompt, which "raise almost no mathematics or numerics flags".

Recall that the same configuration detected 36.7%, and the picture resolves. The easiest way never to flag wrongly is never to flag. Pick a checker on its false-positive rate alone and the quiet one wins. This blog found the same trade-off running the other way in an experiment comparing seven data-cleaning agents, where the configuration that found the most errors also touched the most healthy values.

That 1.0% deserves one more look. Adjudicating a flag raised on a clean copy, the panel has three verdicts available: a confirmed issue, a clear false positive, and ambiguous. Only the middle one counts in the numerator of the false-positive rate. The configuration that scored 1.0% on astrophysics had a single clearly wrong flag — and 18 further flags the panel could not call either way and classed as ambiguous. Those 18 stay out of the numerator while remaining in the denominator, pushing the rate down and nothing else. Behind the cleanest-looking cell stood eighteen flags with the verdict withheld. What makes a panel withhold a verdict comes back in section 6.

4.3Shrink the denominator and an improvement reads as a decline

The cell most worth staring at in this article belongs to one model. When GPT-5.5 on the ML corpus moved from the free-form prompt to the structured one, its false-positive rate rose from 9.2% to 13.0%. Read the number alone and wrong flags appear to have increased. The actual count of wrong flags fell from 12 to 9. Only the ratio went up.

The denominator had halved. Total flags from the same model dropped from 131 to 69. The authors name the mechanism directly: the structured prompt "roughly halves the number of general-issue flags, which are rarely wrong", so the false-positive rate can rise even as the count of wrong flags falls. Take flags that are almost never wrong out of the denominator and the ratio looks worse while the average quality of what remains has not moved.

Fewer wrong flags, a higher rate ML corpus · GPT-5.5 · flags raised on the 15 clean copies (paper §3.2) Free-form 131 flags 12 FP rate 12 ÷ 131 = 9.2% Structured 69 flags 9 FP rate 9 ÷ 69 = 13.0% Only the prompt changed

Bar length is the total number of flags that configuration raised on the clean copies; the orange segment is the count judged false. The orange segment got smaller, the bar got smaller faster, and so the share the segment occupies grew.

This one cell transfers intact to every place quality metrics are handled. A report carrying only ratios lies here. Without the detection count and the total flag count recorded alongside, an improvement gets read as a decline and someone rolls it back. We have seen a judge's scale manufacture a difference that was not there before, in the case where one floor on a rating scale shifted LLM verdicts. This time it is not the scale but the denominator.

Since this article is itself about that confusion, we hold ourselves to a rule. Where two or more rates appear in the same paragraph, each one gets its denominator named. Detection rate is over injected errors, false-positive rate is over flags raised on clean copies, and share of false flags is over all flags of that type.

5

The ground where tools win is narrow

Bolt verification tools onto a model and the ground on which it wins turns out to be narrower than expected. Skepthical, the authors' multi-agent system, runs four branches at once: a general-issues module that reads the paper end to end with an ensemble of LLMs from different families, a mathematics module that re-derives equations through a computer algebra system, a numerics module that runs code to recompute the figures, and a citations module that fetches the PDFs of cited papers and checks them against what was claimed. On the astrophysics corpus it came first at 70.6% total detection.

A scaffolding ladder gives a sense of where that sits. Holding the structured prompt fixed and climbing from a single call to the CLI agent to the multi-agent system, astrophysics total detection runs 53.3%, 61.0%, 70.6%. On ML it runs 72.2%, 73.3%, 76.7%. Each rung is visible on astrophysics; on ML both rungs together are worth 4.5 points. What tools and scaffolding buy you differs that much between corpora.

What the first rung bought also splits by error type. Holding the model and the prompt fixed and adding only tools lifted astrophysics numerics by 12.0 points and mathematics by 16.4 — both types a script can check. On ML, with fewer figures and equations to check, the change was not significant. Tools pay off only on manuscripts that give them something to verify.

The win is clear enough where it happens. On citation errors Skepthical detected 46.7%, ahead of the runner-up CLI agent configuration (26.7%) by 20.0 points. On general issues it led by 26.7 points, and the authors report both differences as statistically significant. With a module that actually opens the cited papers, that looks like the obvious result.

Where those 20.0 points came from is the most operational point in this section. The runner-up CLI agent had the same arXiv access and the same structured prompt. Open its execution logs, though, and it never once fetched a cited paper. In the multi-agent system that step cannot be skipped. The authors read the contrast as a clear case of a verification step being enforced rather than merely offered. Handing a system a tool and making it use the tool are different acts.

5.1Change the corpus and the lead disappears

Move to the ML corpus and the citation lead is gone. Skepthical detects 53.3% of citation errors there, against 57.8% for the single LLMs with the structured prompt and 60.0% for the CLI agent. The abstract states the result outright: "This citation lead, however, is not observed on ML, where the models appear to know the literature better and Skepthical's citation detection rate is statistically indistinguishable from that of the best single LLM." The authors put that sentence, which cuts against their own tool, in the abstract.

An appendix digs into why. The researchers sorted the 45 injected citation errors into two groups: those a specialist could call wrong without opening the cited paper, and those that require checking the source. The first group holds 14 errors on the ML corpus against 4 on astrophysics — more than three times as many. ML literature is already memorized by the models, so opening the source mattered less.

Composition does not explain all of it, though. The authors calculate that even if the ML errors had been distributed across the groups in the same proportions as astrophysics, the single LLMs would still have caught 39.6%, and conclude that "about a quarter of the gain therefore comes from the composition and the rest from higher detection within every group". One line in the body sums up the whole section: "Retrieval makes a difference when the model does not already know the literature." The abstract puts the same point more broadly — tools and scaffolding help when they supply what the model cannot supply itself.

Isolate citation verification as a task on its own and a different picture appears, as this blog found when a cheap LLM judge matched frontier models at citation verification. In this benchmark citations are one of four types, and the one every system caught least often.

5.2Even the wins may not belong to the tool

A stranger number sits in another appendix. The researchers traced each error Skepthical found back to whichever of the four modules picked it up. Numerics behaved as expected: 27 of the 36 numerics errors surfacing in the reviews, 75%, came from the dedicated module. Citations: 11 of 21. Mathematics: 7 of 31. The other 24 came not from the module running computer algebra but from the general-issues module reading the paper straight through.

The authors also read this in their own favor, as evidence against the objection that their system scored well merely because the benchmark's error taxonomy mirrors its module names. That is fair. It equally means the return on bolting on a dedicated verification module may be smaller than its name suggests. What caught the mathematics was not the mathematics specialist but the reader taking in everything. One more note: the authors disclose that this attribution was itself done by a model in a single pass.

The authors devote a paragraph to the other side as well: the detection table makes mechanical verification look smaller than it is. A model reaching 80% numerics detection on its own and a system reaching the same 80% by running a tool and computing deterministically are not saying the same thing. The second leaves a record of which calculation was run and how, so it can be retraced; the first is a single inference, and there is no opening it up to see why it was right. The CLI agent sits between the two: same tools in hand, actually run on only part of the work, so only that part is retraceable. A single detection figure hides the difference. For whoever receives the review, whether the judgment can be re-run matters as much as how many errors it caught.

First place also depends on who the comparison is against. The lead reported in the body is measured against GPT-5.5 with the long structured prompt, and comes to 9.3 points on astrophysics. On ML that same GPT-5.5 is 4.4 points ahead instead, though not significantly. Switch the comparator to the concise-prompt GPT-5.5 from section 3 and the margins shrink further: 1.7 points on astrophysics and 0.6 on ML, neither significant. The authors record that dependence in the appendix themselves — the lead stated in the body rests on which prompt the comparator received.

5.3How the already-public reviewers did

The comparison set includes three reviewers taken from outside. Scores first: the two self-hosted open-weights models caught essentially none of the planted errors — 1.1% and 2.3% on astrophysics, 1.7% and 2.8% on ML. That holds even on ML, which is the literature they were trained on. Their false-positive rates ran the other way and were the highest of any system, between 18.5% and 59.1%.

Carrying those numbers without their conditions would be unfair. Both are small open-weights models trained to write reviews. One is an 8-billion-parameter model fine-tuned on roughly 79,000 ICLR and NeurIPS reviews; the other has 14 billion parameters. Neither was built to find errors. One of them also had its input truncated at the first 15 pages. So the accurate sentence is not "public AI reviewers are useless" but writing text that looks like a review and finding errors are different capabilities. That finding and fixing are different again is something this blog covered in a benchmark where AI agents found their own mistakes but could not repair them.

The third external reviewer is a free agentic review service released by the Stanford ML Group. It ran on the astrophysics corpus only and landed mid-pack at 40.7% total detection. The interesting part is not its score but the numbers it publishes about itself. Its technical overview reports that the correlation between two human reviewers is 0.41, while the correlation between its AI scores and human scores is 0.42. Its AUC for predicting acceptance is 0.75, below the 0.84 of human scores.

What stays with you from this section is that the most widely available AI reviewer publishes how closely it matches human scores, not how much wrongness it catches. Our reading on top of that: a human–human correlation of 0.41 against an AI–human correlation of 0.42 reads as "the AI does as well as a person", and it reads just as easily as "this was measured on an axis where people do not agree with each other either". The second reading is ours. The first fact is something the paper's own introduction raises.

One sentence on the conflict of interest. Skepthical, which came first, is the reference implementation built by this paper's authors, and the limitations section says plainly that the comparison is not resource-matched: it makes more LLM calls and draws on three providers. That the citation lead vanishes on ML, and that mathematics detection did not come from the dedicated module, are also the authors' own words. The paper declares no competing interests. The benchmark repository publishes the 357 injections, every system's adjudication records and the evaluation code, under CC-BY-4.0 for data and MIT for code. The fair thing to write is that they released the material needed to judge them.

6

A model wrote this exam as well

The 357 errors were not planted by hand but by an agent — Claude Sonnet 4.6 on the astrophysics corpus, Gemini 3.1 Pro on the ML corpus — and only injections that passed mechanical verification were kept. Models did the grading too. Deciding whether a review had caught a given injection fell to a three-judge panel drawn from different model families: Claude Sonnet 4.6, GPT-5.4 and Gemini 3 Flash, one vote each.

The point of this section is not to pick holes in that design. It is that this is where the authors write their limitations at greatest length, and that record is the most valuable passage in the paper. A model-built answer key is a problem this blog has met before, in a benchmark whose buggy key made AI agents look worse than they were and in a count of how often AI really does both write and grade the exam.

6.1Judges agree only where the answer key exists

The panels return two kinds of verdict. One is whether a review caught a planted error — the matching panel. The other is whether a flag raised on a clean copy was false — the false-positive panel. The first has an answer key: the record says which error was planted where, so the job is a lookup. The second has none. It has to decide whether the flagged passage is a minor but real problem or a spurious call.

Nearly the same judges, looking at the same papers, agree to wildly different degrees across the two tasks. On matching, the pairwise correlation between judges is 0.90–0.91 on astrophysics and 0.90–0.93 on ML. On false positives it falls to 0.32–0.58 and 0.37–0.53. The authors write of the first that "high agreement is what one expects when an objective key exists", and of the second that the call is "ambiguous when no key is available". The only thing that changed is whether the question came with a key.

The two benches are not quite the same three. Two of the three seats hold the same model, and only the third changes: Gemini 3 Flash on the matching panel, Gemini 3.1 Pro on the false-positive panel. One more narrowing is needed. The 0.32–0.58 range belongs to the panel adjudicating general-issue flags among the flags raised on clean copies. The other three types take different routes: numerics flags are recomputed in Python, mathematics flags are checked with a computer algebra system, and citation flags go to a second panel with the cited PDFs attached.

That condition does not blur the contrast. It sharpens it. The types a machine could verify were verified by machine, and only the type that amounts to human judgment was put to three models. Agreement collapsed precisely on that last one. It also explains why the third verdict seen in section 4.2, ambiguous, had to exist at all.

Questions with a key, and questions without Pairwise correlation range among three judges · astrophysics corpus (paper, Appendix D) Matching panel key exists 0.90–0.91 FP panel no key 0.32–0.58 0.0 0.25 0.50 0.75 1.00 Axis: Pearson r between a pair of judges. Closer to 1 means they returned the same verdicts.

Two ranges produced by two benches that share two of their three seats, working over the same reviews of the same corpus. The lower band is wide not because the judges differ but because the question has no key. Recomputed as Cohen's κ, the authors report, neither range moves by more than 0.03.

Our reading. This contrast is the deepest data-quality lesson in the paper. When you seat an LLM as a judge, what decides reliability is not the judge's capability but the nature of the question. A verdict with a record to check against comes out nearly the same whichever model you swap in; a verdict with nothing to check against is matched only half the time even among the same models. That is why detection rates in this benchmark are the more trustworthy number. Put the other way round: the false-positive rates throughout this article are not as solid as the detection rates alongside them.

The adjudication procedure itself is careful. Of the 4,084 adjudications across both corpora, 93.5% were unanimous 3–0 on the first pass and 6.5% split 2–1. Split cases plus certain other qualifying cases, 34.9% of the total, were re-adjudicated with the PDF attached, and 92.3% of those confirmed the first-round majority. In the end 97.3% were settled by the first pass or the escalation, and 2.7% were overturned.

6.2Five disagreements in the one paper a human checked

A human checked exactly one paper. One of the authors annotated every detection decision for a randomly drawn astrophysics paper (arXiv 2604.20975), blind to the panel's verdicts and to which system had written each review. Of the 132 comparable decisions, 127 matched — 96.2%, with Cohen's κ at 0.914. That is high agreement.

Where 132 comes from is worth stating. That paper's three injected copies carry four errors each, twelve in all, and each copy was reviewed by all twelve systems, giving 36 reviews and 144 adjudications. Twelve of those fall on three reviews for which no panel verdict was recorded, and drop out. The entire human-versus-machine comparison is one paper and 132 decisions.

The valuable part is the five that did not match. All five ran in the same direction: the panel credited a detection the annotator rejected. Detection on that paper therefore comes to 34.8% by the panel and 31.1% by the human. The machine judges were a little more generous.

Reading that as "the detection rates are inflated" overreaches. It is one paper and one person. The precise statement is that on the single paper a human checked, the panel was the more generous of the two, and all five disagreements leaned the same way. The authors list three limitations themselves: the check covers one astrophysics paper; with a single annotator, inter-annotator agreement cannot be measured; and the check covers detection verdicts only. The false-positive panel, which Appendix D shows to be the less reliable of the two, was never put next to a human.

6.3They also tested whether judges favor their own

There is a standing suspicion that an LLM asked to grade will go easy on text from its own family. A 2024 study reported the effect, and the authors cite it and run the same test on their own panel. The result points the other way. If same-family judges were more generous the mean would be positive; it comes to −1.37 points on astrophysics (95% CI −2.41 to −0.39) and −0.17 points on ML (−0.97 to +0.66). In the 2–1 splits, the share where the same-family judge was the lone dissenter is 25.2% on astrophysics and 23.2% on ML, below the one third it would have to exceed for self-preference. Overall leniency per judge was similar across the three providers, within 3 points.

Writing "models graded models, so this cannot be trusted" without mentioning that test would be unfair. The authors tested the most common objection directly and found no evidence for it. That said, self-preference is not the only risk in seating models as judges. The low agreement on false-positive verdicts remains a problem regardless. The broader case of a benchmark measuring something other than what it set out to measure is one this blog took up in an examination of whether benchmarks measure what their names claim.

6.4What this benchmark does not measure

The scope the authors claim in their limitations is narrow. The benchmark does not address "novelty, significance, adequacy of experiments, missing controls, interpretive judgement, or alignment with human-written reviews". Is it new, does it matter, is the experimental design adequate, are controls missing, is the interpretation sound — the questions closest to the heart of peer review are all absent. What gets measured here is one capability: spotting what is already wrong.

The authors also name three deployment risks: false confidence arising from uneven detection across error types, adversarial gaming of known weaknesses, and concentration of review infrastructure on a few LLM providers. The first is the closest to practice. If the overall average is 70% but citation errors sit in the twenties, passing that system guarantees almost nothing about citations.

One last line. The model that injected the errors into the ML corpus and the model that finished last under the free-form prompt are both Gemini 3.1 Pro. The paper never takes up the overlap. The self-preference test in the appendix covers the judging panel, not the injecting model. The conditions differ too: the injector ran as an agent over multiple passes, while the reviewer was a single call on a one-sentence prompt. We note only that the same name appears on both sides.

7

Why this matters to Pebblous

This paper is not a story about scholarly publishing. It is a story about testing the tester, and its design is the same one Pebblous runs first in any data-quality diagnosis: how many of the defects we planted does this pipeline catch? One thing differs. They published the number, along with the cost and the time it took. Most review pipelines have never measured it at all.

There is evidence that measuring changes things. A 2016 study that automatically checked the statistical reporting in 30,717 psychology papers found at least one p-value inconsistency in 49.6% of them, and inconsistencies large enough to change a conclusion in 12.9%. At journals that then built the same checker into their review process, the prevalence of those inconsistencies fell markedly. Attaching a checker did not stop at counting; it moved the prevalence itself.

7.1Five quality lessons this article leaves behind

Translated into data-quality terms, the findings above come to five lines.

  • Detection alone is half a measurement. Without false positives measured alongside, a checker that catches a lot and a checker that flags a lot wrongly look the same. And the easiest way never to flag wrongly is never to flag. That is what the 1.0% false-positive rate turned out to be.
  • Change the denominator and the same improvement reads as a decline. Wrong flags fell from 12 to 9 while the false-positive rate rose from 9.2% to 13.0%, because total flags fell from 131 to 69. If a dashboard keeps only the ratio, this cell gets read backwards.
  • A difficulty ladder exposes the performance curve. Without the subtle, clear and blatant rungs there is no telling what "catches 80%" means. A system at 57–73% on the subtlest injections and a system that cannot clear 5% on the most blatant are indistinguishable under a single average.
  • Verdicts with a key and verdicts without carry different reliability. Nearly the same three judges agree at 0.90 when a key exists and scatter to 0.32–0.58 when it does not. Which of our own quality metrics rest on keyless verdicts becomes the question that follows.
  • Who built the answer key caps the result. In this benchmark a model built the key and a model did the marking, and human verification covered one paper. In that one paper, all five disagreements leaned toward the machine being the more generous.

7.2Five questions to ask before handing review to an AI

Five questions come out of this paper that work as-is in a conversation with a customer or partner about automated review. None of them are specific to peer review. They carry over to code review, contract checking and data validation unchanged.

  • How many deliberately planted errors does our checker catch? If nothing has ever been planted, the answer is that we do not know.
  • How often does it touch something healthy, and over what denominator? Without a clean, defect-free sample running alongside, this number does not exist at all.
  • Does it catch evenly across types? One type at the floor turns the overall average into false confidence — the first of the deployment risks the authors list.
  • Did we change the model or change the instructions, and what did it cost? Rewriting a prompt looks cheap; here it multiplied the per-paper bill by five and the wall time by six.
  • Which of our quality verdicts have no answer key? That is where things give way first.

7.3Where this article stands

This blog has examined the exam and the examiner several times: a buggy answer key that scored agents too low, the real share of designs where AI both writes and grades, a bias check that turned out to rank reasoning, and an exam where the same row carried two different answers. This piece is the first in that series to take the opposite side — a worked example of how to measure properly.

Testing a checker against a planted defect list has clear limits of its own. You learn whether the types you planted get caught, and the types you never thought to plant stay invisible. There is a design that runs the other way: factory inspection AI that learns what good looks like instead of listing defects. Neither approach is complete, and running both beats trusting either one alone.

Sections 1 through 6 carry what the authors measured and what this report verified against the primary source; section 7 is work the paper did not do. Please read them separately. Every figure and every English quotation was checked directly against the full text and appendices of the arXiv release. Where this report calculated a value itself — the share derived by dividing the stage-level spend in Table 13, for instance — it says so on the spot. The per-severity figures in the paper's Figure 1 appear only as a chart, so nothing beyond the 57–73% stated in the body was carried over. As of writing, the arXiv record holds one version, v1.

R

References

The figures in this article come from three places. Values from the body, tables and appendices of reference 1 were taken directly from the full arXiv release, and reference 2 is the data and code that paper published. From reference 3 onward, entries are either sources that paper cites or studies this report brought in as an outside baseline. The human-reviewer baselines (9–11) are grouped separately because they are not values this benchmark measured but results from older experiments running the same design on a different population.

The backbone of this report (checked against the primary text)

  • 1.Zubeldia, Í., Bolliet, B., Villaescusa-Navarro, F., Villanueva-Domingo, P., & Cranmer, M. "Auditing AI peer reviewers: a benchmark on real scientific papers." arXiv:2611.17052v1, submitted 12 November 2026. Accepted at NeurIPS 2026. arXiv: 2611.17052 — corpus selection and injection procedure from §2, detection table from Table 2, false-positive figures from §3.2 and Figure 2, inter-judge agreement from Appendix D, the self-preference test from Appendix E, the human check from Appendix F, the three-prompt comparison from Appendix H Table 11, module attribution from Appendix I, the citation-error taxonomy from Appendix J, and cost and compute from Appendix M Tables 12 and 13.
  • 2.Benchmark release, github.com/inigozubeldia/skepthical-benchmark (data CC-BY-4.0 / code MIT) — carries the record of all 357 injections, the adjudication logs for every reviewer system, and the evaluation pipeline. The reviewer system itself is published at skepthical-ai.org under the Denario Labs name, and accepts one paper at a time as a PDF only.

AI review in deployment

  • 3.Biswas, J. et al. "AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot." arXiv:2604.13940, submitted 15 April 2026. arXiv: 2604.13940 — the 22,977 papers and the under-a-day turnaround in section 1, plus the author and program-committee survey results, come from here. The survey was reported by the team that ran the pilot, and a preference is not a measurement of accuracy.
  • 4.Jiang, Y., & Ng, A. Stanford Agentic Reviewer, Stanford ML Group. paperreview.ai/tech-overview — the 0.42 correlation with human scores (0.41 between humans) and the 0.75 acceptance-prediction AUC in section 5.3 are the values published on this technical overview. Detection and false-positive rates are not published.

The lineage of the method

  • 5.Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList." ACL 2020. — the study reference 1 names directly as an ancestor of its design.
  • 6.DeMillo, R. A., Lipton, R. J., & Sayward, F. G. "Hints on Test Data Selection: Help for the Practicing Programmer." IEEE Computer 11(4), 1978. doi:10.1109/C-M.1978.218136 — mutation testing, from section 1.5. Not part of the lineage reference 1 cites; the analogy is this report's.
  • 7.Panickssery, A., Bowman, S. R., & Feng, S. "LLM evaluators recognize and favor their own generations." NeurIPS 2024. / Verga, P. et al. "Replacing judges with juries: Evaluating LLM generations with a panel of diverse models." arXiv:2404.18796 (2024). / Ye, J. et al. "Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge." arXiv:2410.02736 (2024). — the self-preference test in section 6.3 and the cross-family three-judge panel design rest on these three.
  • 8.Dycke, N., & Gurevych, I. "Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework." arXiv:2508.21422 (2025). / Xi, S., Rao, V., Payan, J., & Shah, N. B. "FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers." arXiv:2511.21843 (2025). — the prior landscape referred to in section 1.3. The first is named as the only prior work with false-positive control.

Human reviewer baselines (not measured by this benchmark)

  • 9.Godlee, F., Gale, C. R., & Martyn, C. N. "Effect on the Quality of Peer Review of Blinding Reviewers and Asking Them to Sign Their Reports: A Randomized Controlled Trial." JAMA 280(3), 237–240 (1998). — the two of eight (median), the 16% and the 33% in section 2.2.
  • 10.Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. "What errors do peer reviewers detect, and does training improve their ability to detect them?" Journal of the Royal Society of Medicine 101(10), 507–514 (2008). — the average of 2.58 out of nine in section 2.2.
  • 11."Testing for the Presence of Positive-Outcome Bias in Peer Review: A Randomized Controlled Trial." JAMA Internal Medicine (formerly Archives of Internal Medicine) — the 17.3% against 39.1% in section 2.2, from 238 reviewers at two orthopedic journals.

Error prevalence and reviewing load

  • 12.Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. "The prevalence of statistical reporting errors in psychology (1985–2013)." Behavior Research Methods 48(4), 1205–1226 (2016). / Nuijten, M. B., & Wicherts, J. M. Advances in Methods and Practices in Psychological Science (2024). — the 49.6% and 12.9% in section 7, and the drop in prevalence at journals that built the checker into review.
  • 13.The official arXiv blog, blog.arxiv.org — the 40,363 submissions in September 2026 and the two-per-month per-author cap in force from 1 October 2026, both cited in section 1.4.

Related reading on the Pebblous blog