Executive Summary

Key numbers

Source: Li et al., OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences, arXiv:2610.11118 (8 October 2026), main text and Table 3

A benchmark normally starts out holding the answer. Whether the task is an olympiad question or a theorem already in print, grading means laying the model's output next to a solution kept on file. OpenProblemBench went the other way and picked the problems where no such file exists. The paper notes that an unresolved problem offers no final answer to retrieve, so a system has to decide for itself which direction is productive, judge what evidence would close an argument, and notice when a derivation that looks convincing still leaves quantifiers, parameter ranges, or exceptional cases open.

Gathering the problems took two stages. A Large Knowledge Model first pulled candidate questions that looked open out of the literature, and researchers then went back to the source papers and everything published since to confirm that each one really is unresolved. What survived was 46 mathematics problems and 36 in theoretical physics, scattered across subfields such as combinatorics, geometry, statistical mechanics and quantum information. The oldest item traces back to Goldbach in 1742; the newest comes from work first posted in 2025 and published in 2026.

What the model receives is not a one-line question. Each problem statement carries the research context, the assumptions, and the progress made so far, along with the conditions a resolution has to satisfy. It reads closer to a briefing on where the problem currently stands. That list of conditions is what fills the space a stored answer would normally occupy, and grading, in the end, begins by opening the same list again.

The selection rule narrows the findings before any model starts work. The paper says it favors questions whose proposed solutions admit comparatively clear checks of their decisive mathematical or computational claims: problems where you can inspect the step the argument hinges on, confirm a construction against its own assumptions, or re-run the calculation the conclusion depends on. So these 82 are not a sample of open problems in general but open problems already filtered once for checkability. The authors note in their limitations that the 82 questions are uneven across subfields.

Even the premise that a problem is still open comes with no guarantee. The authors, based mainly at the University of Science and Technology of China and the Institute of Theoretical Physics at the Chinese Academy of Sciences, write that literature searches may miss solutions and that relevant statements may occur in training data. A future release, they say, should specify a literature cutoff and add human-researcher baselines. Which is to say that as it stands the benchmark has no column recording how many of the same 82 problems a person would solve.

With nothing to check the work against, the standard has to be built from scratch. OpenProblemBench builds it out of four models. GPT-5.6-Sol, Kimi-K3, GLM-5.3 and Qwen3.8-Max each read every submission independently, with no sight of what the others concluded.

The first thing a grader does is write down what the problem demands. The paper calls these the obligations. Once the obligations are on the page, the submission gets matched against them one by one, through its assumptions, its quantifiers, its decisive proof steps and its supporting computations. Targeted recomputation and source checks are permitted where the grader's access policy allows. The grader has to assess the argument as written, without filling in a missing lemma of its own, and then records a verdict together with any point where the submission claimed more completion or generality than its argument supports. Each verdict has to cite which requirements it was weighed against and which mathematical or computational evidence it rests on.

A verdict lands in one of three places. Solved means every task requirement is established. Partial means at least one relevant result beyond the supplied premises is established while some requirements remain open. Unsolved means no such result is established. Partial splits again into three levels: trivial for a direct standard application or an easy special case, nontrivial for a meaningful family, an improved bound or a useful reduction, and breakthrough for a result that clears a central obstacle and substantially changes the work that remains. There is a separate Incomplete box for reviews where the evidence was too thin to decide.

How the four scores get combined is part of the design too. Each grader receives equal weight, and the percentage in each outcome category is averaged across the four. Nobody takes a majority vote, and nobody appoints one grader as the referee and treats the rest as advisory. That choice governs how the numbers that follow should be read.

The scale comes out of a multiplication. Seven model configurations worked the 82 problems, producing 574 submissions, and since all four graders read each one the review count is 2,296. Of those, 2,288 carry a completed outcome assessment. The 2,095 reviews marked partial break down into 768 trivial, 1,252 nontrivial and 75 breakthrough.

The labels alone do not tell you where the line between solved and partial actually falls. So the paper lines up ten problems that six models all received and compares the submissions side by side, and finds four differences recurring. The first is whether the problem gets reformulated: one side found a representation that aimed straight at the requested conclusion, the other only tightened control over intermediate quantities. The second is whether finite evidence turns into a general argument, since confirming a handful of cases by computation still leaves the argument covering every admissible case to be built. The third is whether the gaps in a proposed proof get closed, because agreement between two implementations is not the same thing as establishing the mathematical correspondence the problem asked for. The fourth is that the Flash models do advance by putting up explicit constructions, but snag on deriving the auxiliary assumptions, covering the exceptional cases, or carrying the proof to the end.

None of the four turns on what the conclusion says. They all turn on whether the argument closes. You cannot ask whether the work is right, so you ask instead whether anything is still missing. The numbers in the next section are four separate answers to that question.

The abstract puts up three numbers. GPT-6-Astra leads at a mean of 14.0%, the full-size open models evaluated alongside it come in at 5.5–6.7%, and the lightweight Flash models at 2.4–3.7%. Read those three lines on their own and the field looks neatly tiered.

The table those averages were built from sits right there in the paper. Seven solver models run down the rows and four grading models across the columns, 28 cells in all, each one recording how many of that model's 82 submissions that grader counted as solved.

All four graders put GPT-6-Astra first. That much does not move. The count they attached to that first place, though, runs from 9 to 15. The most generous grader credited it with better than 1.6 times what the strictest one did. The paper itself treats rankings below the leader as less stable.

Second place genuinely changes hands. Under GPT-5.6-Sol the runner-up is GPT-5.6-Sol on 7; under Kimi-K3 it is Kimi-K3 and Qwen3.8-Max tied on 7, with GPT-5.6-Sol dropping a rung to 6; under GLM-5.3 and under Qwen3.8-Max the runner-up is Qwen3.8-Max. The models did not change and neither did their submissions. Only the reader did.

Add the columns up and you can see that the graders have different hands as well. Totalling the solved verdicts in each column gives GPT-5.6-Sol the fewest at 30 and Kimi-K3 the most at 45. Yet the stingiest of them, GPT-5.6-Sol, handed Astra the highest count any grader gave it, 15. Strictness, in other words, does not fall evenly across submissions. That same grader allowed GLM-5.3 a single solved verdict where Kimi-K3 saw 6 in the same work.

The four graders are also four of the submitters, which leaves four cells in the table where a model read its own work. GPT-5.6-Sol gave itself 7, the highest figure in its row. Kimi-K3 likewise gave itself 7 for the top of its row, work that GPT-5.6-Sol put at 3. GLM-5.3 gave itself 6, tying Kimi-K3 for the row's best. Qwen3.8-Max breaks the pattern: it gave itself 6, and Kimi-K3 and GLM-5.3 both rated the same work higher at 7. Three out of four lean their own way, then, on a sample of four cells that the paper does not analyze separately.

The ground they agree on is narrower than it looks. Of the 82 problems, 10 were judged solved by all four graders, and 8 of those came from Astra. The grades within partial spread as well. For Kimi-K3's submissions, among completed reviews not judged solved, the share rated nontrivial or breakthrough runs from 56.0% to 89.9% depending on the grader. What that shows, the authors write, is how the distinction between trivial and substantive progress varies from grader to grader.

Most measures of data quality rest on there being something to compare against. Label agreement, classification accuracy, duplicate rate: each needs a reference value to exist before it can be computed. Yet a good share of the output whose quality we actually have to ask about has no such reference. Whether a summary condensed its source faithfully, whether the reading in a client report holds up, whether a support reply was any use — none of these has a stored answer. And a model grader has already moved into that seat.

OpenProblemBench works as one example of what that arrangement needs in order to stand up. Three things stand out.

• • Break the standard into named items before grading — instead of asking whether the output is any good, set out first what this output was supposed to accomplish. Change the grader and the thing being compared against is still there
• • Do not stop at one grader — a single grader's score blends that model's disposition with the actual quality, and once blended there is no way left to pull the two apart
• • Do not read agreement as reliability — this is the paper's own warning. The four can be wrong in the same way, so a high agreement rate measures the size of the common blind spot rather than the quality

The last of the three is the one practice forgets most often. OpenProblemBench also averaged four graders' percentages into a single line, and from that line alone the fact that the runner-up changed with the grader simply disappears. What the paper did do was publish the per-grader table alongside it, so that whatever the average covered can be recovered. An internal evaluation screen that shows only the mean of its model graders offers no such route back. A design that treats disagreement as noise to be cleaned away and a design that keeps a record of it turn the same number into two quite different kinds of evidence.

The paper does not stop there either; it sets out what the next release should do. The proposal is to put a stratified sample back in front of blinded domain experts, and the part worth noticing is what it says to put in that sample. Problems the four graders unanimously judged solved go in, along with the disagreements, each level of partial progress, the overclaim findings, and the all-negative cases. A human review sample is normally drawn from the low scores or the split decisions. Including the unanimous passes in the same sample is a way of keeping open a route to finding out whether the four went wrong together.

The principle of keeping whoever produced the work apart from whoever checks it came up in an agent that cannot tell you it is finished, and the arrangement where AI grades an explanation that AI wrote showed up first on the biology side in the model that writes protein function descriptions. For the view from a company selling evaluation itself as the product, there is the firm that grades AI on real work.

Thank you for reading this far. The paper is at arXiv:2610.11118, and every figure in this article was taken from its main text and Table 3 directly. If your team is handing output with no answer key to model graders, we would like to hear how many graders you run and how you keep a record of where they disagree.

Pebblous Data Communication Team
October 17, 2026