Executive Summary

Key Numbers

All four come from the main experiment on the 36 tasks. The first two say how strong the automatic baselines were; the last two say what conditions produced them and where they fell short.

Source: Ellington et al., Automatic Construction of Strong Simple Baselines (arXiv:2610.40700, October 28, 2026)

"Caught up" only means something against a scale. The researchers put three scores on every task: the score of a trivial predictor, the score of the deep learning model as reported in the original paper, and the score of the automatically built baseline. A trivial predictor is one that always returns the same answer. In a classification problem that is the most common class in the training set; in a regression problem, the average of the training targets.

Those three define simple baseline strength, or SBS. Put the trivial predictor at 0 and the deep learning model at 1, and the measure says where in between the automatic baseline stands. A value of 0 means the simple method added nothing at all. A value of 1 means it stands where the deep learning model stands. Above 1 means the simple method did better.

Across the 36 tasks, the median of that value was 0.955. The mean is lower, 0.857, because a handful of tasks sit far down the scale and pull it. Counting unscored attempts as zero drops the mean further, to 0.796, and leaves the median at 0.955. On twelve tasks the strength reached 1 or more. The paper writes "12 studies" at this point, though the unit the cohort is built from is the 36 tasks.

Where the baselines were weak is just as clear. Six tasks had a median strength below 0.5, and all six took input that was not in table form. These are tasks that work on raw sequences, structure strings or atomic coordinates, the kind of data that does not drop into rows and columns.

The authors are careful to block the obvious conclusion. Several tasks at or near a strength of 1 take the same kinds of input, so, in their words, "input type alone does not explain the low scores." All six of the weak tasks also had at least one attempt that could not be scored, and two of the six medians rest on only two scored replicates. There is not enough here to draw a line around where deep learning earns its keep.

The job this paper handed to a machine is setting up an honest comparison, which is aimed somewhere other than finding a better model. One model wrote the baselines, Claude Opus 5, at medium reasoning effort, with five independent attempts per task. Of 180 attempts, 157 produced predictions that could be scored. The rest either failed to run or returned nothing gradable.

The constraints are tight. No pretrained weights, no GPU, one script, and the run has to finish inside thirty minutes. Pretrained weights were ruled out because checkpoints trained on public data may already have seen a task's test split.

One point deserves to be pulled out. The agent was never told to build a simple model. The prompt asked it for "a reasonable baseline model, not necessarily a simple one," and training a neural network was allowed. Simplicity came from the environment rather than the instruction. Of the 157 scored scripts, 47 used PyTorch and 85 used gradient-boosted trees. Given permission to reach for a neural network, more than half of them went back to the methods that worked on table data before deep learning arrived.

So did the model simply copy the answer out of the original paper? The researchers went at that question by taking information away from the model one level at a time, starting with the full paper text and all of the training data, then cutting to the abstract, then thinning the training rows, then removing the data entirely. This experiment ran somewhere else, though. Instead of the 36 tasks it used a separate cohort of nine, and it helps to know that before reading the table.

Values measured on the nine-task cohort, as the paper reports them. Only the last three conditions differ significantly from L1.

The baseline collapses the moment the training data goes. The distance between the full paper and the abstract alone is small, 0.88 against 0.83, while taking the training rows away drops it to 0.17, and removing the test inputs as well brings it to zero. That points toward a model that received data and fit something to it, rather than one reading a paper and reproducing a score.

Whether more time helps was checked on the same nine. A single thirty-minute attempt averaged 0.88, and raising the cap to 150 minutes gave the same 0.88. Picking one of the five scored attempts by validation score reached 0.90, ten rounds of iteration reached 0.92, and a five-hour budget reached 0.96. The authors note that none of these differences is statistically significant; the smallest adjusted p-value is 0.16.

They also disclose that the arms differed in what documentation the agent could read. The single attempt and the best-of-five ran on image builds that redacted the task documentation, while the other arms saw documentation that states the reported score, and in two of the five-hour attempts the agent used that reported score to estimate its own. So the comparison does not isolate the effect of compute. Even so, their summary is that the simple single-iteration setup already captured most of the observed performance.

It sounds odd that a weak comparison clears review at the top journals in science. The authors locate the cause in the load review has to carry rather than in the diligence of any individual reviewer. Building one strong baseline means looking at the data, choosing a method and running it, task by task. Reviewers rarely have the means to do that.

Nor is this the first time a simple baseline has pulled level with deep learning. The paper's introduction gathers cases from several fields. In single-cell perturbation prediction, five foundation models and two other deep learning models were compared against simple baselines, and none of them came out ahead. In long-term time-series forecasting, one-layer linear models beat the Transformer methods under evaluation across all nine datasets. A benchmark of 45 tabular datasets found tree-based methods still stronger than the neural methods on medium-sized data, even after an extensive hyperparameter search. In recommender systems, six of seven reproducible neural approaches could often be outperformed by simpler heuristics.

The paper brings numbers for the volume as well. Springer Nature reported 3.1 million submissions in 2025, more than 30% above the year before. Machine learning venues point the same way: ICLR submissions went from 11,603 in 2025 to 19,525 in 2026, and NeurIPS submissions from 9,467 in 2020 to 21,575 in 2025.

Those figures hold up the proposal. If building a baseline is work that eats human time, and that time is what is missing, then handing the work to a machine attaches a practical and scalable check to the review process.

Turning this result into "deep learning was a bubble" says something the paper does not. The sentences in its limitations section draw the boundary first. The scope was held to supervised prediction tasks that are straightforward to evaluate and relatively cheap to compute, leaving out harder work such as representation learning and generative modeling. The cohort is 36 tasks taken from open-access publications in a small set of high-impact journals, and the model is Claude Opus 5 alone.

The comparison itself has lines drawn around it. Baseline strength is measured against the score each original paper reported, not against a rerun of the deep learning method. Nobody matched the conditions under which those reported scores were produced. Each automatic baseline was evaluated on a single fold, and for the 13 tasks the original papers evaluated across several folds, the reference it was compared against is still a single reported number. The authors add that the numerical value of baseline strength depends on the choice of trivial predictor and should be read with care.

The most candid passage concerns contamination. The model may have seen the source datasets or papers during pretraining, and the task documentation visible to the agent often states the score the original paper reported. The information-removal experiment in Section 2 takes some weight off that worry, though it too ran on the separate nine-task cohort.

Whether results split by field or by journal is still unanswered. With three to 18 tasks per journal and three to six per field, the paper states that it did not test differences between groups. No sentence about which field is sloppier can come out of this study.

When model performance gets reported inside a company, scrutiny collects around the model. People ask which architecture was used, how the hyperparameters were chosen, how many folds the validation ran. The thing the model was compared against passes quietly by. A sentence saying the new approach is some percent better than the existing one rarely gets met with a question about what the existing one actually was.

Yet as this paper shows, the size of an improvement is set as much by the comparison as by the model. The same model looks impressive against a weak opponent and modest against a strong one. Choosing what to compare against is part of the measurement.

The constraints the paper put on its baseline agent convert directly into things to check. Three of them are worth running against an internal project.

• Who did we beat? Write down, as one number, where the proposed method stands between a trivial predictor and the model currently in use
• How much went into building the baseline? If the model got weeks and the baseline got an afternoon, the difference in their scores has a difference in effort mixed into it
• Does the data fit in a table? The six tasks where the baseline was weak all took non-tabular input. Tasks with the same input reached deep learning anyway, so input type is no alibi, and when the problem does fit in a table the quickest move is to run the older methods once

To say once more what this paper automated: the comparison, not the model. Whether a method looks good because the method is good or because its opponent was feeble is a question a machine worked through on behalf of the reviewers, and on 12 of 36 tasks the answer was the second one. When we say the data proves something, what we measured it against is almost never written into that sentence. It is better to write it down.

Thank you for reading this far. The paper's title, authors and submission date; the 36 tasks and the median baseline strength of 95.5%, the mean of 85.7% and the 12 tasks; the mean of 79.6% when unscored attempts are counted as zero; the definitions of baseline strength and of a trivial predictor; Claude Opus 5, five replicates, 157 of 180 attempts scored, 47 scripts using PyTorch and 85 using gradient-boosted trees; the thirty minutes, the CPU-only rule and the ban on pretrained weights; the prompt asking for a reasonable rather than a simple baseline and permitting neural networks; the six tasks below 0.5 all taking non-tabular input together with the authors' sentence that input type alone does not explain those scores, their unscored attempts and the two medians resting on two replicates; the re-evaluations collected in the introduction (single-cell perturbation prediction, nine datasets in long-term forecasting, 45 tabular datasets, six of seven recommender methods); the values from L1 to L8 and the makeup of the nine-task cohort; the compute arms at 0.88, 0.88, 0.90, 0.92 and 0.96, the smallest adjusted p-value of 0.16 and the redacted documentation; Springer Nature's 3.1 million submissions and the ICLR and NeurIPS counts; and the limitations, including the single-fold evaluation, the 13 tasks, the three to 18 tasks per journal and three to six per field, were all confirmed in arXiv:2610.40700 itself. Section 5, which moves the building of a comparison into a company's own measurement routine, and the three checks it lists, are this article's reading. If you have a way of building baselines inside your own team, we would be glad to hear it.

Pebblous Data Communication Team
November 3, 2026

The paper and its code

• 1.Ellington, C. N., Wang, J., Addagudi, S., & Cole, E. (2026). "Automatic Construction of Strong Simple Baselines." arXiv:2610.40700 [cs.LG], October 28, 2026.
• 2.Ellington, C. N. "cnellington/automatic-baselines." GitHub. (code released with the paper)

Work in the same line

• 3."Simple Baselines are Competitive with Code Evolution." arXiv:2602.16805, February 2026. (simple baselines held their own against elaborate automated code-evolution methods in mathematical bound search, agent scaffolds and machine learning competitions)