Executive Summary

Key Numbers

Source: You et al., FrontierPhysics: Evaluating Agents for End-to-End Physics Research, arXiv:2611.06910 (2026)

The four numbers below carry different denominators. 27.4% is the score of Claude Opus 5.5, while 48% comes from a hand reading of the failure records of a different model, GPT-6 Astra. 56 is the number of tasks, and 96.2% is the score the authors got when they ran this benchmark's grading standard over research they had already published.

The usual way to build a benchmark is to gather problems that already exist. Scrape textbook exercises, carve a sub-task with a settled answer out of a published paper, translate an old exam. This paper took none of those routes. It asked working physicists to turn one piece of research they had actually done over the past few years into a task.

Submissions came to 182, and 56 of them were accepted. They arrived from 42 researchers at 26 institutions. Each submission had to come with material showing that the research belonged to the person sending it: a paper, a patent, some evidence that their name is attached to the project. For 18 of the 56 tasks, the research underneath has already been published, and four of those papers ran in Nature.

A task that cleared the provenance check then went through review by another researcher in the same field. Reviewers were drawn from contributors whose own task had passed the first gate, and most were doctoral students (47%) or PhD holders (37%). The review form named three things to look at. Whether the task description spoon-feeds the route to the answer. Whether the rubric carries the physical insight only someone who has done that research would have. Whether the at-least-three trial runs submitted alongside the task contain a false pass or a false failure. The second item comes with a comparison attached: tell it apart from the low-signal items common in AI-generated rubrics.

There are nine domains: high-energy physics, condensed matter physics, astrophysics, quantum information, atomic/molecular/optical physics, fluid dynamics, statistical physics, optics and imaging, and chemical physics. Theory, computation, and experiment are all in there. The task counts are uneven. Condensed matter physics has the most at 12 and chemical physics the fewest at 4, with the other seven domains falling between 5 and 7.

The paper also records the time stacked up behind the tasks. By the contributors' own reports, each original project took roughly 270 days on average, and together they add up to more than 8,000 hours of research. The contributors average 420 citations. One sheet of exam paper has a whole research career sitting under it.

The paper opens by setting out what existing physics benchmarks look at. They have agents answer fixed questions, or solve one sub-task cut out of a research process, or reproduce a paper whose solution is already supplied. In all three the answer sits somewhere already. And all three check only whether the result came out right; the quality of the research barely enters.

A FrontierPhysics task asks for four stages at once. The agent has to find and read the relevant literature, draw up a research plan, carry out the research and the experiments, and write up what came out. Two things have to be handed in at the end. One is machine-checkable output: a computed physical quantity, a simulation result, a processed dataset. The other is a manuscript named paper.pdf. The time limit is ten hours per run.

Building it this way changes what gets measured. On an exam that only checks the answer, the route taken in the middle does not matter. On an exam that collects the manuscript as well, which prior work was leaned on, why that method was picked, and how far the limits of the result were spelled out all come into view. The ability to carry a piece of research to the end is layered on top of the ability to solve the problem.

Grading splits in two. One side is a deterministic verifier. A test script attached to each task compares the numbers an agent produced against reference values. There is no room for a person to step in, and the same input gets the same verdict. The other side is a rubric, and its core is written by the physicist who set the task.

The rubric has two layers. One layer applies to every task. It looks at reward hacking and hallucination, and at research practice: whether the literature was synthesized, whether the results were checked against themselves, whether claims were backed by evidence. The other layer is per task, and the contributor writes it. Whether the physics of that research was understood, whether the key references were found and used properly, whether the practical constraints of the experiment were taken into account, the kind of item you can only write if you have run that task yourself. Assigning each item a weight between 1 and 10 falls to the contributor as well.

Cutting across both layers, items split again into two kinds. There are blocker items, where breaking even one shuts off a pass, and the remaining items, which do not touch the pass decision and only produce a quality score. A blocker item is a condition the research needs in order to hold up scientifically, so it is judged 0 or 1 and nothing between. The remaining items are scored for quality in three steps, 0, 1, or 2. Blocker items carry weights too, but those values stay as metadata and never enter the quality score.

The definition of a pass follows from this. The deterministic verifier has to come back correct, and at the same time not one blocker item can be missing. If either one catches, the final score is 0. A low score on the remaining items does not block a pass. The quality score is there to separate the levels of work that already cleared both gates.

The party that actually holds the rubric and assigns the points is not a person. Every run was graded by a single reviewer-agent configuration, GPT-5.6 Sol on Codex CLI. The paper gives its reason for using an agent rather than a single model call. Items like reward hacking, hallucination, and literature search cannot be decided from the submitted manuscript alone, since the work log, the code, and the output files have to be followed too. The authors also swapped in Claude Opus 5.5 as grader and regraded the first round. Pass and fail verdicts came out nearly the same, but that grader was more generous toward work from its own model family.

A rubric like this invites one objection: that the bar is so tight no human would clear it either. The authors closed that off first. Grading their own published material against their own rubric returned an average of 96.2%. Finished human research clears almost all of this bar, which makes it hard to put an agent's low score down to the rubric. The scope of that self-check is worth setting out. It covered the 33 tasks that came with reference material, and only the items that could be judged from the manuscript were picked out and handed to the same reviewer agent. Of the 36 gradings that came back, 3 failed to clear a blocker item even though the material was the authors' own.

The scale of the exercise goes like this. Twenty-two models were each set against the 56 tasks three times over. That makes 168 runs per model, and every run used the highest reasoning setting that model offers. Four harnesses did the work of driving the agents, matched to the model family: Codex CLI for GPT models, Claude Code CLI for Claude, Grok build for Grok, Antigravity CLI for Gemini. Model and harness combine into 26 configurations, and the whole thing came to 4,364 runs.

Claude Opus 5.5 sits at the top of the table. A pass rate of 27.4%, 46 runs out of 168. And the sentence the authors put in their abstract is that this 46 is a ceiling. No configuration passed more than 46 of its own 168 runs. The current frontier is gathered at a spot that does not quite reach three tries in ten.

The same table carries another column, read per task. Opus 5.5 got through 25 of the 56 tasks at least once in three attempts (44.6%), and 8 tasks came through all three times. The floor is in the table too. Two configurations went 168 runs without clearing both gates once.

A score on its own says nothing about why the runs did not clear. So the researchers read the records themselves. The subject was the 168 runs of GPT-6 Astra (xhigh), and the task contributors and the organizers followed the work logs the agents left behind and sorted out where things went off. One run can carry several tags, so the shares below overlap. The denominator is not 168 either, but the 143 of those runs logged as failures.

The largest item was rigor of reasoning, at 48%. Self-verification that circles in place, or boundary conditions and extreme cases left out of account: both land here. Next comes physical intuition at 34%. At the point where the cause of a problem has to be named, the agent does not look where an expert would suspect first. Failing to tie the literature together accounts for 27%, and not following instructions as given for 26%.

The items missing from those four stand out more. Getting a physics concept wrong, botching a formula, code that would not run: none of them sit near the top. The last sentence of the abstract records exactly that. The main weaknesses of frontier agents are insufficiently rigorous reasoning and limited expert intuition.

The reading turned up one more thing besides the weaknesses of the agents. Of the 168 runs, 9 had the agent do the task properly and still get logged as a failure. The task description was ambiguous, or something went wrong on the verifier or reviewer-agent side. Correcting those nine moves the same configuration's pass rate from 14.9% to 20.2%. The authors pull in prior work at this point. When experts regrade major benchmarks, a sizable share of the failures turns out to come from grader errors, wrong reference answers, and ambiguous problems. Finding false failures takes domain expertise and time of its own.

This analysis should be read within its scope. The records that were read are not those of all 22 models but the 168 runs of one. The top pass rate belongs to Claude Opus 5.5 while the failure analysis looked at GPT-6 Astra, so nothing in the paper supports widening these four shares into a claim that every model gets stuck in these proportions. The correction just mentioned is the same. The authors write that GPT-6 Astra is the only configuration whose pass rate they can fix by separating out false failures.

The part of this paper that lasts is not 27.4%. That number will move within months. What lasts is the material the number came out of. Each task came from the physicist behind that project, the rubric came from that same person, and the authors turned that rubric on their own papers first, to be sure it was not simply too harsh. The process for making evaluation data has a longer life than the result it produces.

Turn it around and a question appears. Who made the exam our own organization uses to measure AI? In many cases it is not the person doing the job but an evaluation team that watched the job from outside and built from there. An exam built that way holds only what could be observed. The moment where an experienced hand looks at a result and senses that something is off is not observable, so it never becomes an item, and the fact that the slot is empty does not show up on the scoreboard either.

A few earlier posts on this blog have covered nearby questions. The defect floor in physics benchmarks looked at what a score measures when the answer key itself is wrong. The case where an original author gave an AI manuscript 2 out of 6 is a record of the same thing happening once, in one person's hands, and this benchmark repeats it at scale and with a procedure behind it. Set it beside a benchmark where AI grades problems with no answer key and the contrast sharpens. In both, a model occupies the seat that carries out the grading. The difference is what that model looks at to reach a verdict, and here the list of what to look at was written by the researchers whose own work the task came from.

An organization with an evaluation system of its own has three things it can check today.

• Did the people who actually do the work write our evaluation items, or did the items come out of a document describing that work?
• Are the items that decide a pass kept apart from the items that only take points off? Mixed together, a minor deduction and a fatal defect dissolve into the same average.
• What score do our own criteria give to work from an experienced hand? If the bar is one no human clears, a model's low score is not a story about the model.

Thanks for reading this far. The original is at arXiv:2611.06910. If your team has moved to having the people doing the work write the internal evaluation items, we would like to hear how the scores and the cost of reaching agreement changed.