Executive Summary

Key figures

Source: LongDS-Bench (arXiv:2605.30434)

The four numbers below capture both the size and the cause of the collapse. The first two show how large it is; the last two point to why it's a problem of state management, not memory.

Released in May 2026 by a joint team from Zhejiang University and Ant Group, LongDS-Bench has a different grain than the data-analysis benchmarks that came before it. Most existing benchmarks pose a single, self-contained question and score the answer. But real data analysis doesn't flow that way. You define a cohort, apply a filter, build an intermediate metric, quietly change that definition a few days later, then roll back to a version you froze yesterday to answer a question, all tangled across many turns.

To reproduce that flow, the researchers drew 68 tasks from real Kaggle notebooks and rebuilt them as multi-turn conversations. They span six domains (earth science, business, education, sports, social good, and community), averaging 33 turns per task, 2,225 turns in all. What matters is that these turns aren't independent. A single request depended on state from 11.3 turns earlier on average, and a single turn had to reference 2.9 prior states at once. Miss what was defined earlier, and how, and everything downstream shakes loose.

The researchers call the thing that holds this chained work together "analytical state." It isn't just the numbers produced earlier. It's the entire context of which scope, definitions, and assumptions a calculation was run under. When conditions like "the LTV of new customers last quarter, North America only, excluding repeat buyers" pile up turn after turn, the agent has to keep tracking which conditions are currently in force. The tasks capture five patterns for handling this state.

A single task held, on average, 5.8 Rollback turns and 8.6 Composition turns. In other words, going back and stacking things on top of each other is the default, not the exception. The context a human analyst leaves naturally in notes and code cells, the agent had to hold on its own every turn.

When five frontier models were run on the same 68 tasks, even first place stalled just at the halfway mark. Gemini-3.1-Pro led at 48.45% on average, followed by GPT-5.4, Claude-4.6-Sonnet, Kimi-K2.6, and DeepSeek-V4-Pro. For reference, this scoring matched human verification 93.11% of the time (Cohen's κ = 0.86), so the scores themselves are hard to doubt.

Scarier than the averages is the change within a task. Models that got things roughly right on the early turns went sharply wrong as the same task moved toward its later turns. The accuracy gap between early and late reached nearly 47pp. The very model someone might have judged "good enough" after watching only the opening was quietly collapsing at the end.

This is precisely where things get dangerous in practice. Data-analysis results usually come from the output of the last few turns: the final aggregation and conclusion. Yet the point where accuracy collapses is exactly that late stage. A pattern where the early exploration looks plausible but the final answer is wrong is the hardest kind of failure for a reviewer to catch.

The researchers dissected the failures and split the causes broadly into two camps. Traditional errors (coding, planning, domain reasoning) accounted for 31–48%, and the remaining 52–69% were "long-horizon errors" that arise only in long tasks. In other words, more than half of the failures happened not because the model lacked intelligence, but because it couldn't handle the state that carries forward. These long-horizon errors split again into three kinds.

Here an interesting paradox surfaces. What we usually worry about is that "AI forgets what came earlier." Yet that simple memory failure (Context Memory) was the smallest category. Pulling information back up was relatively easy. Where things truly broke was in applying the information correctly: judging which state is currently valid (State Management) and passing it correctly into the next calculation (Cascade).

Why the largest category, Cascade Error, is so dangerous is right there in the name. Get one definition wrong early, and every filter, aggregation, and conclusion stacked on top of it is contaminated. That's why late-stage accuracy sagged by 47pp. Not because the later parts were inherently harder, but because a state that went wrong early flowed downstream and ate away at the answer. A single subtle state drift quietly colors every turn that follows.

It's easy to think intuitively like this: if the later stage is hard, just give the agent room to think more and try more. LongDS-Bench's results rebut that expectation head-on. The researchers state flatly that "additional steps do not necessarily improve performance."

The clearest evidence is inside the leaderboard. Among the models tested, the one that used the most steps was Claude-4.6-Sonnet. Yet its score came in below Gemini-3.1-Pro, which moved far less. The model that tried harder and more often lost. When the number of attempts doesn't rise alongside accuracy, it's a signal that the bottleneck lies somewhere other than the amount of effort.

The researchers' reset experiment points to that "somewhere else." Agents tended to take fewer exploration steps as they moved into the later stage. Since a state that went wrong early had already contaminated the whole late stage, giving more steps just spins in place on a broken foundation. That's why the answer is to build a point where state gets corrected, rather than to inflate the budget.

This blog has covered the failure of long-horizon agents twice before. Once it was an infrastructure problem: a weak harness kills the agent. Another time it was a memory problem, where the secret of the agent that lasted a year wasn't cleverness but a notepad. LongDS-Bench is the third scene in that arc. This time, in the concrete task of data analysis, it nails down across 68 tasks and 2,225 turns that the cause of failure is neither too few steps nor too little memory, but the holding of state.

Read from a data point of view, the conclusion is clear. The analytical state an agent must hold (definitions and scope, assumptions and intermediate results) is ultimately data that has to be managed explicitly, outside the model. Unless it's structured so you can tell under which definition, and when, that data was updated, and which version is valid now, even the smartest model loses its way in the late stage. In the age of autonomy, the bottleneck isn't greater reasoning power. It's how well you've designed the state there is to hold.

So the question shifts to this. Is the multi-step data work your team has handed to an agent missing the contamination in the later parts, just because the earlier parts look plausible? There are three things to check. Is there a structure that explicitly stores and re-references intermediate definitions and assumptions? Have you tested Rollback, returning to a previously frozen version? And are you separately verifying the later turns, in case accuracy is only good early on? Before switching to a bigger model, it's time to put the state you need to hold in order first.

• 1.Xu, K., Lu, X., Qiao, S., Ding, Z., Xu, H., Liang, L., & Zhang, N. (2026). "LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis." arXiv:2605.30434. — A multi-turn data-analysis benchmark of 68 tasks and 2,225 turns built from real Kaggle notebooks. The top model, Gemini-3.1-Pro, averaged 48.45%, with late-turn accuracy dropping about 47pp below early turns. Of all failures, 52–69% were long-horizon errors (Cascade > State Management > Context Memory), and additional steps were found not to guarantee performance.