Executive Summary

Ten of today's most capable coding agents were set against tasks drawn from 90 Nature-family papers. The instruction was not to reproduce the papers, but to actually beat the best result each one reported. This is the design of NatureBench, a benchmark built by researchers at Tsinghua, Harvard, and Peking University. And the scorecard it produced leaves anyone who works with data an unexpected question.

Even the best-performing model surpassed the existing record on only sixteen of the 90 tasks, or 17.8%. The other four-in-five could not clear the paper's bar. More telling is the character of the wins. Where an agent did prevail, it rarely invented a new scientific method; in most cases it recast an unfamiliar task as a familiar supervised-learning prediction problem and solved that instead.

The causes of failure also run against the common assumption. Cases where an agent collapsed because it misunderstood the task itself accounted for just 3.1%. Most failures came from choosing the wrong method, or from running out of compute budget before pushing the chosen method far enough. Writing code, in other words, is no longer the bottleneck.

Four numbers hold up the result: the strongest model's rate of beating the record, the share of wins reached through an engineering path, and the gap between method choice and task comprehension that separated the failures.

17.8%

Top model's rate of beating SOTA

Claude Opus 4.7, 16 of 90

82.7%

Wins via an engineering path

Supervised translation, tuning, engineering combined

45.1%

Leading cause of failure

Wrong method choice

3.1%

Failures from misunderstanding the task

Running the code is not the bottleneck

1

Ninety Nature-Grade Tasks, and Sixteen

Benchmarks that measure how well coding agents reproduce a paper already exist in number. NatureBench asks a step harder: can an agent go past reproduction and reach discovery? Put differently, can it beat the best performance a paper reported, on its own? The researchers drew 90 tasks from 90 peer-reviewed Nature-family papers, spread across six scientific domains — from single-cell omics and protein biology to physical modeling and molecular design.

Evaluation ran with web search fully disabled, so no answer could be looked up. Ten of the latest models were then run on top of three coding-agent environments (Claude Code, Codex CLI, and Gemini CLI) and pitted against one another. Each task was scored on a single question: did the run beat the best result the paper reported, or at least match it?

The strongest configuration was Claude Opus 4.7. Yet even this model clearly surpassed the existing record on only sixteen of the 90 tasks, or 17.8%. Widen the bar to tasks it beat or at least matched, and the figure is still just 47.8%, short of half. The remaining nine models scored lower, and the weakest beat the record on exactly one of the 90.

Ten Coding Agents — Surpass-SOTA Rate Ranking Claude Opus 4.7 17.8% Gemini 3.5 Flash 15.6% GPT-5.5 14.4% Claude Opus 4.6 12.2% Qwen 3.7 Max 10.0% Kimi K2.6 8.9% GPT-5.4 8.9% GLM-5.1 7.8% DeepSeek-V4-Pro 4.4% MiniMax-M2.7 1.1% Baseline: share of tasks (of 90) that clearly surpassed the paper's published record (g>0.1), web search fully disabled. The top model's average cost per task was $21.65 — three to four times the models right behind it.

Figure 1. Ranking of ten coding-agent configurations by Surpass-SOTA rate (g>0.1). Even the top scorer, Claude Opus 4.7, beat the record on only 16 of 90 (17.8%). Original Pebblous diagram. Source: reconstructed from NatureBench (arXiv:2606.24530) Table 4.

There is one more detail here. Claude Opus 4.7, the top scorer, was also the model that spent the most tokens and money. Its average cost per task was $21.65, three to four times that of the models behind it. The side that tried the most and dug the longest won, a fact that connects naturally to the failure analysis later in this article.

2

Success Was Translation, Not Invention

Read only the headline numbers and it is easy to slide into the familiar conclusion that "AI still isn't good at science." NatureBench cuts sharper by dissecting the wins themselves. The researchers classified every case that matched or beat the record by the method behind it. It turned out that 45.5% of successes came from recasting the task as a supervised-learning prediction problem. Add optimization and tuning, engineering pipelines, and pretraining scale-up — all such engineering paths together — and the figure reaches 82.7%.

By contrast, cases where an agent attempted a novel approach fitted to the scientific structure of the task, or found an alternative grounded in domain reasoning, added up to only about 17%. Most success, then, was not the invention of new science; it was moving an unfamiliar problem onto familiar machine-learning ground and tuning it well.

Breakdown of Successful (Match-SOTA) Runs by Method Engineering path — 82.7% Supervised-learning translation 45.5% Optimization & tuning 17.6% Engineering pipeline 11.0% Pretraining / scaling 8.6% Closer to invention — 17.3% Method-aligned solution 9.0% Domain-reasoned alternative 8.3% Baseline: of the 900 runs that matched or beat the record (Match-SOTA), how each success was reached. The two paths closer to real invention (method-aligned + domain-reasoned) add up to just 17.3%.

Figure 2. Breakdown of successful (Match-SOTA) runs by method. The engineering path (translation, tuning, engineering, scale-up) accounts for 82.7%; paths attempting a genuinely new method total only 17.3%. Original Pebblous diagram. Source: reconstructed from NatureBench (arXiv:2606.24530).

2.1A Win That Was Not a New Method

One case captures this character well. The task was to identify cancer-related genes. The original paper solved it with transformer-based graph representation learning, and Claude Opus 4.7 beat its record here. The way it did so was an ensemble of several graph convolutional networks: computing the normalized Laplacian, applying early stopping on a validation set, and training and combining several models while varying degree, depth, and seed.

The paper's assessment is cool-headed. The agent read the task correctly as node classification on a graph, but it did not propose a new method for identifying cancer genes. It won by grasping the structure of the task and finely tuning an appropriate existing technique — not by making a scientific discovery. Behind the word "success" on the scoreboard hides this difference in kind.

3

Failure Was Not a Comprehension Problem

So why did the tasks it could not clear collapse? Here the result flatly inverts the common assumption. When the researchers decomposed the failed runs layer by layer, cases that failed because the agent misunderstood the task itself came to just 3.1%. Most failure lay not in comprehension but in the step after it.

The largest share was method-selection failure. Choosing the wrong method to solve the problem accounted for 45.1%, the single most common cause, followed by running short of compute and time budget before pushing far enough, at 24.4%. Combine the method layer and the execution layer, and most failures pile up here. The agents almost always produced code that ran. The problem was which path they chose, and how deep they dug into it.

Layer Breakdown of Failed Runs (67.8% of All Runs) Method-layer failure 61.1% ↳ of which, wrong method choice: 45.1% Execution-layer failure 28.7% ↳ of which, insufficient compute/time budget: 24.4% Strategy-layer failure 7.0% Understanding-layer failure only 3.1% Cases that failed purely from misunderstanding the task came to just 3.1% — agents almost always produced code that ran. Most failure (89.8%) came after comprehension — in method choice and compute budget.

Figure 3. Layer breakdown of failed runs. Task-misunderstanding failure is just 3.1%; wrong method choice (45.1%) and insufficient compute budget (24.4%) account for most of it. Original Pebblous diagram. Source: reconstructed from NatureBench (arXiv:2606.24530).

The texture of failure differed by field. In relational reasoning, where the structure of the problem is comparatively clear, six in ten runs matched the record; but in biomedical modeling, where many pieces are entangled, that share fell below one in five. Cross-discipline tasks that fused two or more fields were distinctly harder than single-field ones, with eight or nine of the ten agents dropping in the same direction. The signal is that weaving different scientific contexts together to frame a problem — rather than digging deep into a single technique — is the farthest frontier for today's agents.

3.1A Wall That 258 Submissions Could Not Clear

A genomic-sequence prediction task shows this limit. It comprised nineteen sub-tasks predicting histone marks, enhancers, promoters, and splice sites. One model submitted to this task no fewer than 258 times. It cycled through every method it could reach, starting from a k-mer count model, then a linear classifier, a convolutional network on GPU, an enhancer-activity ensemble, a threshold sweep. Its best score came on the 220th attempt.

Even so, the result fell short of the existing record. The cause the paper points to was a limit of expressive power itself. However diligently the runs repeated, if the chosen method fundamentally cannot capture the task, the number of attempts does not convert into results. Diligence and outcome are not proportional; that is the lesson this case leaves behind.

To sum up: today's coding agents are already adept at understanding a task and writing a runnable solution. What holds them back sits on either side of that — the judgment to see which method to solve with, and the budget to push that method to the end. The explanation that "AI can't solve it because it doesn't understand the problem" was, at least across these 90 tasks, far from the fact.

4

The Bottleneck Is Framing, Not the Model

This result points to exactly the same spot the Pebblous blog covered three days ago. In new-materials discovery, the effort spent building the AI model was 1% of the total, with the remaining 99% going into securing the data — a Nature Materials comment we read then. Where that piece delivered a qualitative verdict — "the model is not the bottleneck" — from within a single field, materials science, NatureBench pins the same spot quantitatively, on a far wider stage of six fields, 90 tasks, and ten state-of-the-art agents.

The coordinate it marks is clear. Discovery is not slow for lack of a bigger model. The code mostly ran fine. What separated discovery was the work of deciding what problem to frame the task as and which method to aim at it (45.1% of failures), and how deep that method could be pushed (24.4%). If in materials science 99% of the effort was data, then across these 90 Nature-grade tasks the majority of failure was a matter of choice: what to solve, and how.

One distinction is worth keeping clear. The paper-code reproduction story this blog covered last month sits at a different layer from this result. That bottleneck was an infrastructure problem: getting published code to actually run against an environment and its dependencies. NatureBench's bottleneck sits above it. The code already runs; what is far rarer is using it to beat the existing record. On top of the finding that even reproduction stalls half the time, this lays a further finding: discovery beyond reproduction is rarer still.

For any organization that works with data, this result translates into a single question. If the discovery we expect from AI is slow, is the cause the absence of a bigger model, or the way we set the problem up — with which data and as which task? NatureBench's answer points toward the latter. Scores were higher when a method close to the scientific structure of the task was chosen, and most of the cases that translated the task into familiar prediction stopped at the wall.

Editor's Note

Pebblous works on diagnosing and preparing data — before it enters AI training — so that it faithfully captures the structure of the problem it is meant to solve. Put NatureBench's message in our own words, and it reads: the bottleneck of discovery lies not in the size of the model but in how the problem is framed. Which data, and which task, you use to define the problem is what ultimately decides whether the model succeeds or fails.

R

References

Academic Papers

  • 1.Wang, Y., Cheng, L., Zuo, Y., Ding, N., Zhou, B. & Zhang, K. (2026). "NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?" arXiv:2606.24530.
  • 2.Su et al. (2025). "TREE: Transformer-based graph representation learning for cancer-gene identification." (Cited as the source paper for NatureBench case study 1.)
  • 3.Dalla-Torre et al. (2025). "The Nucleotide Transformer benchmark." (Cited as the source paper for NatureBench case study 2.)

Industry & Community