Executive Summary

A second AI that does nothing but check the first one's work raised the number of errors a scientist caught 14-fold. The result comes from DiscoveryAgent, a paper that researchers at Stanford, with co-authors at UC Berkeley, UCSF, and SLAC, posted to arXiv on November 16. The paper calls the checker an audit agent.

It is easy to read that figure as humans against AI, but both sides of the comparison were human. Working from the same research record, the scientists who had the audit agent recovered that many more of the deliberately planted errors than the scientists handed only the code and the report. Each group had four people, and the review lasted one hour. The sample is small, which is why this article reads the conditions alongside the size of the number.

Sections 1 through 4 report what the paper says. Section 5 is the reading this article draws from it.

Key Numbers

Four numbers take the measure of this paper: the gap the audit opened up, the accuracy of the audit agent itself, the model scores that came out of a steered hypothesis, and the size of the experiment those numbers came from.

Source: DiscoveryAgent: A Framework for Scientist-centric Agentic AI (arXiv:2611.21950, 2026-11-16).

14x

Extra errors a scientist recovered with the audit agent

3.50 per person against 0.25

0.965

F1 of the audit agent at picking out corrupted claims

50 claims were extracted and 25 of them deliberately altered

105

Baselines the protein variant-effect predictor beat

The steered version lost to only one of 106 ProteinGym baselines

4

People in each cohort behind the 14x

Four per group, with one hour of review

1

The Scientist Swaps the Hypothesis Between Rounds

DiscoveryAgent differs from other AI scientist systems less in how much autonomy it has than in where a scientist can step in. The research runs in rounds, and between one round and the next the scientist picks the hypothesis to test next, refines it, or replaces it outright. The design is meant to work across fields, and scientist-centric is the term in the paper's own title.

Structurally there are two loops. The outer loop scopes the work into hypothesis branches; inside a branch, a director agent tests a single hypothesis on its own, calling on sub-agents for literature search, data analysis, and review. The scientist works on the outer loop. Whatever runs inside stays untouched, and the decision left to the scientist is which branch comes next.

Checkpoints hold this design up. Every step is recorded: which data the agent handled, how it handled them, and what it used as grounds for moving on. Because the record survives, a single claim can be picked up later and traced back to where it came from, and the same analysis can be run again. Rather than trading autonomy away for supervision, the design keeps the autonomy and makes the run possible to retrace.

DiscoveryAgent's Structure — Outer and Inner Loop Outer loop — the scientist picks the next branch Inner loop — director agent verifies one hypothesis Director agent Literature search Data analysis Review Hypothesis A Hypothesis B — selected Hypothesis C Checkpoints — record the grounds for each step so it can be retraced later
▲ Pebblous original diagram, reinterpreting the paper's structure description

Steering shows up in the scores as well. On protein variant-effect prediction, the version a scientist steered beat 105 of the 106 ProteinGym baselines. Exactly one baseline did better. On variable-star classification the run produced the best classifier among the methods compared.

Three case studies show what steering actually looked like. DiscoveryAgent was paired with a researcher in astrophysics, economics, and genomics, and one study was run in each field. In the astrophysics case an astronomer with machine learning experience set the direction across 16 rounds, and over that span the agent trained 190 model configurations. The model that came out of it used photometry and images together, and it reduced photometric-redshift scatter for low-redshift galaxies by 18.6% against the image-only CNN and by 39.3% against the photometry-only MLP.

2

Two Experiments Put a Number on the Audit Agent

The audit agent does not stop at commenting on a finished report. It goes into the checkpoints, the code, and the logs, follows each claim back to its origin, and reruns an analysis when it has to see whether a number reproduces. The paper describes this as supervision that can be scaled up. Putting more human experts on the problem hits a limit quickly, while more agents can always be pointed at the record.

Performance was measured along two lines. The first put the audit agent on its own. The researchers extracted 50 claims from the research traces of three agent systems, corrupted 25 of them, sampled five such sets, and had the audit agent pick out the corrupted claims. On DiscoveryAgent's own traces the score was an F1 of 0.965.

The second experiment brought people in. Scientists were split into two groups and given one hour to review research output with errors planted in it. One group worked through the code and the report alone (the Code group), and the other had the audit agent alongside (the Audit group). The result was 3.50 unique errors recovered per person against 0.25, and that is where the figure of 14 times comes from. Each group had four people, a point the paper does not hide. The direction is clear; the exact size of the gap is something a larger sample will have to confirm.

Errors one reviewer recovered in one hour Code group Paper and code only 0.25 Audit group With the audit agent 3.50 Four reviewers per group, one hour each. Bar length is proportional to the mean recovered per person
▲ Pebblous original diagram, built from the per-person averages the paper gives
3

Some Research Traces Can Be Checked and Some Cannot

The same audit agent scored differently on records left by other systems. Audit accuracy fell on traces left by Kosmos, an autonomous research agent, and by a general coding agent, with the steepest drops on the Terra and Luna models.

The drop moves where the responsibility sits. How well an audit goes is not settled by how clever the checking side is. It starts with what the checked side left behind, and in what form. A system with checkpoints built into the design has claims that can be taken apart, and a system that hands over results alone gives even a good audit tool nothing to verify.

That 0.965 belongs to the audit agent paired with DiscoveryAgent, not to the audit agent alone. Auditability looks less like a feature bolted on afterwards and more like a property that has to enter the design at the start.

4

What the Audit Caught and What People Still Had to Do

Both remaining case studies produced results of their own. In the economics case, DiscoveryAgent and an economist showed that homeowners insurance premiums in Florida have grown more regressive. The median premium rose from $1,200 in 2013 to $2,200 in 2023, an 83% increase, and in 2023 the lowest income quartile spent 6.6% of income on insurance against 1.2% for the highest quartile. In the genomics case, work with a computational biologist identified relay genes in CD4+ T cells, the genes that both take in and send out many signals, and autoimmune GWAS genes turned out to be enriched among them at an odds ratio of 2.44, with primary immunodeficiency genes enriched at 11.66. The candidates came down to four: TUFM, CALHM6, ATXN2L, and FOSL2.

One of the errors post hoc auditing turned up came out of the genomics study. In Round 3, DiscoveryAgent had defined paralogs from the prefix of the gene symbol, grouping every gene starting with CD into one family and producing pseudo-paralog families that make no biological sense. The audit caught it, which also means nobody saw it while the analysis was running.

The places where a person had to keep hold of the work are just as clear. In the economics case DiscoveryAgent made stronger assumptions than a descriptive analysis would normally carry and veered into causal inference, which took heavy steering from the economist. The researcher on the genomics case pointed out a tendency to run large numbers of statistical tests and then emphasize the results that passed stringent filters, instead of presenting those analyses as exploratory. The hypotheses the agent first proposed also needed considerable work before they became biologically meaningful questions.

Taken together, the paper does not claim that verification has been lifted out of human hands. It claims that one scientist can cover more ground in an hour than before. The last call on what is wrong still belongs to someone who knows the field, and reading that a pseudo-paralog family makes no biological sense takes someone who knows the field too.

5

Why Pebblous Is Watching This Paper

To a team that has spent its working life confirming data and model quality by hand, the figure of 14 times says nothing about how capable the reviewers were. The Code group recovered 0.25 errors per person not out of laziness but because code and a report were all they held. The Audit group, which recovered three and a half in the same hour, held a tool that could dig through the record. The bottleneck was never the capacity to check. It was how much of the record had been left behind to check against.

Data quality work has the same shape. With no record of why a label was applied, which rule filtered a row out, or which version of the standard was in force, review turns into guesswork over the output. Automating the checking side in that state buys speed without widening what anyone can actually see. That is the point at which the worry about the check itself becoming one more black box to trust stops being hypothetical.

The paper's answer is about order. Settling what the audited side has to leave behind comes before making the audit tool better. Four things are worth checking before any automated review.

  • Does the data pipeline running today leave a record at each step that can be retraced, or does it leave only the final output?
  • Can a single quality verdict be traced back to the rule and the data it came from?
  • Has anyone confirmed that the same input still yields the same result? How many days does that confirmation take?
  • Is there a procedure for people to reopen a sample of what the automated review passed and what it rejected?

None of the four asks which checking tool to buy. All four ask whether the work is ready to be checked. Calling data ready for AI carries more than tidy formatting. It carries the ability to pull up later why things turned out the way they did. The 14 times in DiscoveryAgent did not come from hiring a smarter checker. It came from a system that left something to check.

R

References

Primary Source

Comparative and Background Work

  • 2."Kosmos: An AI Scientist for Autonomous Discovery." arXiv:2511.02824. The autonomous research agent DiscoveryAgent compared its audit accuracy against.
  • 3."OSVerify." arXiv:2610.18089. Earlier work on verification where expert review is limited.
  • 4."ScientistOne." arXiv:2605.26340. Earlier work that proposed the idea of auditing research integrity.