Executive Summary

Key numbers

The first two show how far the best model got. The last two say how much that score moves across tasks and across models, and what the judgment rests on.

Sources: arXiv:2610.22599 · autolab-ai/EV-estimation (October 15, 2026)

Stories about AI helping with research usually turn on how well it solves the problem. MLE-bench is one of those yardsticks. It collects Kaggle competition tasks and watches how high an agent can score on them. The place the authors of this paper found empty sits one step earlier. Choosing which idea to run happens before any training does, and there was no yardstick for whether that choice was right.

The reason that place is empty is that training is slow. The space of design choices is endless and a single run costs time and money. Neither a person nor an AI can run all of them and pick afterward. Both narrow the field with a sense of which option will work and then run what is left. How often that sense is right is the number nobody had written down, and that is what the authors point to.

The usual way to measure it is to put two ideas side by side and ask which one is better. The authors did not take that route. A question about preference says which one is on top and says nothing about how far apart the two are. Instead they had the models write down a number: how much the score would move.

The procedure runs like this. Across eight MLE-bench tasks, the authors gathered 288 improvement ideas that models had proposed themselves. The ideas came from models rather than from people, which later made it possible to separate a model's own ideas from everyone else's and compare the two. The eight tasks break down into two on text (spooky author identification, toxic comment classification), four on tables (random acts of pizza, the May 2022 tabular playground series, leaf classification, New York taxi fare prediction), and two with images in them (dog breed identification, denoising dirty documents). Each idea was then written into the code, trained for real, and scored against an untouched baseline to record the change. Five models (GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1, Kimi K3, GLM-5.3) wrote their own predictions before seeing any of those results. They received the description of the idea and the code change, but not the outcome. The raw data and the predictions are public in the repository.

One thing worth noting is that the yardstick differs from task to task. Some tasks are scored by log loss, others by ROC AUC or root mean squared error. Lower is better on some and higher is better on others, and the spacing differs as well, so putting all eight on one line to add or average them means settling on a common ruler first. That is why this paper records the size of a change in units of standard deviation rather than in the task's own score. The "two standard deviations" that appears later means the move was twice the spread of the changes the ideas produced on that task.

The scores came out as a coefficient of determination, R². Results come out differently from one idea to the next, and this value says how much of that spread a model's predictions tracked. A 1 means they tracked it perfectly, and a 0 means saying the average every time would have done as well. Below are the scores by model. The slope column is taken up in the next section.

0.49 sits in an awkward place. It is clearly better than guessing with no sense at all. So is it enough to decide on its own what gets run? The authors' answer is not yet. Predictions this good already help rank which ideas to run first, but they cannot stand in for running the experiments.

A single average hides something. Split the results by task and one model's score changes wholesale from task to task. Claude Opus 5.5 climbs to 0.94 on toxic comment classification and 0.81 on the tabular task, but stays between 0.39 and 0.43 on dog breed identification and denoising dirty documents, the two tasks with images in them. That is how far the spread runs inside a single model. Change the model as well and the floor drops further. GPT-6 Astra managed 0.02 on leaf classification. A good sense in one area does not carry over to another.

The authors also tested whether more time to think changes anything. Raising the reasoning level lifted only the top three models. Claude Opus 5.5 moved from 0.35 to 0.49, Claude Fable 5.1 from 0.31 to 0.41, GPT-6 Astra from 0.28 to 0.38. Kimi K3 and GLM-5.3 barely budged. Extra context split the field the same way. Taking the code change away pushed the top three down, and adding a description of the data with twenty sample rows improved only Claude Opus 5.5.

If the coefficient of determination says how much a model got right, the regression slope says which way it got things wrong. Plot the predictions against the measured results, draw a line through them, and the slope is how steep that line is. At 1.0 the size of the change is right as well, and below 1.0 the model called it smaller than it was. The five slopes run from 0.61 down to 0.21. This is not one model's habit but a lean that shows up in the same direction in all five.

Set the two columns of the table above side by side and the order is identical. The lower a model's coefficient of determination, the lower its slope. The ones that get less right shave more off, which read the other way means both improve together as the sense sharpens. Even so, the leading model is still at 0.61, so the habit of shaving has not gone anywhere.

Putting real numbers to that slope shows how wide the gap is. For an idea that actually lifted the score by two standard deviations, the best model's written prediction lands somewhere near 1.2 standard deviations. The weakest model lands near 0.5. The direction is right, and close to half the size is gone.

Have AI propose the ideas and then have AI judge them, and an obvious worry comes attached: that a model will be generous toward what it came up with. The authors checked this separately. Once the quality of the idea itself and each model's general level of optimism were factored out, the error on a model's own ideas ran 0.01 standard deviations higher than on ideas from other models. The confidence interval runs from −0.06 to +0.08 standard deviations, which amounts to no difference at all.

They also looked at whether the result moves when someone else implements the idea. Claude Fable 5.1 and Kimi K3 swapped 66 ideas and implemented each other's, and 58 of them agreed within 0.25 standard deviations, with 25 coming out at exactly the same score. Not all of them did, though. On some ideas written loosely enough that what to do was open to reading, the interpretation of whoever implemented them decided the result. The vaguer the idea, the blurrier the line between measuring the idea and measuring the implementation.

The authors set out the limits themselves. Each idea was trained only once, so part of the recorded change may be the noise of training itself. That means the ceiling a perfect predictor could not pass is still unknown. There are also only eight tasks and all of them are small problems. Training a new model from scratch is not among them. The authors list running repeats to measure the noise, and widening the work to larger tasks and to chains of several ideas, as what comes next.

Why that limit matters shows up in what the authors are aiming at. They write that this result bears hardest on recursive self-improvement, where an AI chooses and runs the experiments that make itself better. In that setting one experiment can mean training a new model, so a wrong choice costs that much more.

Choosing what to take on first, before the expensive work starts, happens on the data side every day. Which slice to collect more of, which batch to spend the labeling budget on, which data to throw away. Collection and labeling cost as much as training does, and gathering everything first and choosing afterward is just as impossible. So here too, a sense of what something is worth moves first, before any result is in.

What this paper did is turn that sense into a number. The predictions were written down in advance and matched against measurements afterward, which is how a hit rate came out at all. Getting the same number on the data side takes the same order of operations. In practice, though, that record usually does not survive. How much a given dataset was expected to move the model is what the work gets approved on, and that estimate gets talked through in a meeting and goes no further. Once the work is done, only the result remains. A hit rate cannot be computed from predictions nobody wrote down.

For a team that has started handing collection or augmentation priorities to an AI, three things are worth checking.

• Whether the judgment is recorded together with its predicted value. If only the choice survives and not how much better it was expected to be, there is nothing left to match against later
• Whether the hit rate is checked again when the domain changes. In this paper even the best-scoring model fell from 0.94 to around 0.4 once the task changed. A record of scoring well on one kind of data is no guarantee for another
• Whether candidates with large effects are being scored small. If all five models called changes smaller than they were, a collection plan big enough to shift the field may be sliding quietly down the list, and someone needs a place to look at that separately

Thanks for reading this far. The numbers in this article were confirmed directly in arXiv paper 2610.22599 and in the repository the authors released. If you set priorities for collecting or augmenting data, we would be glad to hear what form that judgment is kept in.

Pebblous Data Communication Team
October 22, 2026