Executive Summary
Can you really hand an AI agent a multi-hour, real-world task and walk away? A new benchmark puts exactly that question to the test. On the previous generation, which bundled short manipulation tasks, top agents climbed to 83.5% and the problem looked essentially solved. Once the tasks were stretched to the length of actual work, agents from the same generation slumped to around 20%. This article looks at where that collapse comes from.
The 108 tasks in OSWorld-V2 take a median of about 1.6 hours. Even the top agent finished only 20.6% of them within 500 steps. Completion rates that hovered around 20% in the 45-minutes-or-less band dropped below 10% past the two-hour mark, and beyond 163 minutes no model finished a single task. Most failures happened not because the model lacked knowledge, but because it could not carry the constraints it confirmed early on and the facts that arrived midway all the way to the end.
The bottleneck, then, is state, not intelligence. Context that piles up over hours is ultimately a single dataset, and left with no schema, no freshness check, and no validation gate, it rots and breaks exactly the way a human-built pipeline does. This article argues for treating that state as data to be managed.
Key Figures
Source: OSWorld-V2 (arXiv:2606.29537), project page
20.6%
Best completion rate
Share of tasks the top agent finished within 500 steps
0%
The three-hour wall
No model finished any task running longer than 163 minutes
1.6 hrs
Median task length
About 48× longer than the prior generation (~2 min)
83.5% → 20%
Change length alone
A saturated short-task score collapses at real length
Short Tasks Are Already Solved
Agents that operate a computer directly have improved remarkably over the past two years. On OSWorld 1.0, which served as the standard yardstick, frontier agents climbed to around 80%, and according to the OSWorld team the best score reached 83.5%. In plain terms, tasks like opening a file, changing a setting, or looking something up on the web and typing it in are now handled largely without human intervention.
The question is what that 83.5% was actually measuring. The tasks in OSWorld 1.0 had a median duration of about two minutes and ran to roughly 30 steps on average, single-shot operations. Doing well at something a skilled user would finish in two minutes is clearly progress, but it is not the shape of the work we actually want to hand agents. Gathering material from several places into a table, filtering a document down to the entries that meet a condition, revising it again to reflect a requirement that changed midway through, these take hours, not minutes.
So XLang Lab, the group behind OSWorld, released a follow-up benchmark and asked: does short-task saturation mean the problem is solved? The answer was no. The benchmark's measured difficulty simply had not captured the difficulty of real work. To clear away the illusion, the tasks had to be returned to their true length.
The core point: 83.5% is a signal that agents have grown capable, and also a signal that the ruler measuring that capability was too short. Short-task saturation was not the solution to the problem but the starting point of the next question.
Measured at Real Length, Completion Falls Off a Cliff
Released in June 2026, OSWorld-V2 is built from 108 long-horizon workflows. It spans 7 professional domains and 21 subcategories, and it runs in a controlled environment that self-hosts 31 services so external service changes cannot shake the results. The median task takes about 1.6 hours, and 69.6% of the tasks run longer than an hour. Compared with the roughly two minutes of OSWorld 1.0, the axis of difficulty has shifted heavily toward duration rather than knowledge.
When frontier agents from the same generation were dropped into this environment, the scores flipped. In the best configuration, the top agent finished only 20.6% of tasks within 500 steps, and even counting partial credit it reached just 54.8%. Models that had scored 83.5% on short tasks sank to a fifth of that in the face of real length. The models did not suddenly get worse; a wall the short benchmark had kept hidden came into view.
This collapse was not one model's weakness. The agent that reached the top of the completion ranking poured out more output tokens than any other model and still stayed put, while the model that spent tokens most frugally was stuck around 13%. In other words, throwing more compute at the problem or swapping in a different frontier model did not move completion much. The wall lay not in any particular model but in task length itself.
The sharper picture is in completion by task length. Below 45 minutes it holds in the low 20s, but from the 137-to-163-minute band it drops below 10%, and past 163 minutes no model finished a single one. Performance does not decline gently. It is a cliff: it begins to break around the one-hour mark and touches bottom by three hours.
| Task length band | Completion (approx.) | Reading |
|---|---|---|
| 45 min or less | 20–24% | Even short real work is already hard |
| 137–163 min | ≤10% | Completion breaks past the one-hour mark |
| Over 163 min | 0% | By three hours, no one finishes |
Source: Yuan et al., OSWorld-V2 (arXiv:2606.29537). Figures are approximate per-band completion across several models.
The core point: completion does not tail off gently as tasks grow longer; it breaks sharply at a specific point. In a world where time is difficulty, the score in a demo and the score on real work trace entirely different curves.
What Exactly Do Agents Forget
Where the cliff comes from only becomes clear when you take the failures apart. OSWorld-V2 also introduced a methodology for classifying failure causes structurally. The hardest category was tasks that require the agent to recover state that is not visible on screen (43 tasks), followed by tasks that require stitching together information from multiple sources (46 tasks, 42.6%). What the two have in common is not a shortage of information but the need to hold on to information already in hand all the way through.
The failure story that recurred in the case studies narrows to four. The agent misses when to stop gathering information, fails to notice a change that occurred mid-task, guesses a value instead of confirming it with the user, and submits the result at the end without validating it. None of these is a knowledge problem. Each is a failure to hold on, over the course of hours, to the conditions set early, the facts that arrived midway, and the moment when something needs to be re-checked.
This is where the article both connects to and diverges from our two earlier pieces. Pebblous has argued that in a year-long simulated business, what separated survival from ruin was not intelligence but the habit of writing things down, and before that that what breaks an agent is not a weak model but a weak harness. After the structure that a harness provides and the act of writing things down, this piece steps one level deeper to look at how the thing written down itself goes wrong.
The core point: what agents forget is not knowledge but state. The constraints they fixed, the information that arrived midway, the moment to re-check: the instant these three, which must be held for hours, quietly leak away, even a capable model spins its wheels.
It Is a Data Quality Problem, Not a Memory One
The prescription that writing things down solves it is only half right. Context that accumulates over hours is ultimately a single dataset the agent keeps re-reading, and a dataset rots if it is not managed. Translate the failures OSWorld-V2 observed into the language of a data pipeline, and they line up exactly with failures we already know from human data quality.
| What the agent did | In data-quality terms |
|---|---|
| Loses a constraint set early on | Schema drift — a fixed condition silently lost from the state record |
| Fails to reflect information that arrived mid-task | Missing increments — referencing only the initial snapshot, never absorbing updates |
| Guesses instead of confirming | Unvalidated insert — writing a value of unknown origin into state as if it were fact |
| Skips the final check | No QA gate — committing the output without validation |
There is a further twist here. When Chroma Research tested 18 frontier models, performance degraded non-uniformly well before the context limit was reached, and cleanly structured input sometimes triggered the degradation faster. The paradox of data trust, that the tidier a log looks the less anyone re-validates it, whether human or agent, shows up in an agent's long-lived state just the same.
The core point: an agent fails to finish hours-long work not for want of a smarter model but because the state accumulated over those hours was never managed as a single dataset. Data with no schema, no freshness check, and no validation gate will break a pipeline even when a human builds it.
Not a Bigger Model, but State as Data
The natural rebuttal is this: won't a wider context window fix it? The earlier results answer. A wide window is not the same as a well-used one. Models hold on to the beginning and the end of a long input while letting the middle slip by, and as the window grows they lose track of where to pay attention. A bigger model and a longer window do not resolve this bottleneck on their own.
What needs to happen is to promote state into data to be managed. First, have the agent record the constraints and decisions it must hold not as free-form prose but as structured state. Separating what is fixed from what is still open into distinct fields reduces the chance of a quiet reversal later. Second, split the task along a time axis with explicit checkpoints, and at each one make the agent re-check the initial constraints against the latest state. Third, require the output to pass a validation step before it is submitted. The schema checks and QA gates we place in human pipelines belong in the agent's state too.
So the question changes. To put your agent on real, hours-long work rather than a demo, will you wait for a bigger model, or will you design the state it carries over those hours like a data contract? Split a pilot into 30-minute, one-hour, and two-hour bands and measure completion, and it soon shows exactly where your agent starts leaking state. That is where the real work begins.
To close: the power to finish long tasks comes not from a bigger brain but from well-managed state. The context an agent carries over hours is data. How you handle that data's schema, freshness, and validation is the real edge in the age of autonomy.
References
Academic Paper
- 1.Yuan, M. et al. (2026). "OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks." arXiv:2606.29537. — 108 long-horizon desktop workflows. Median 1.6 hours, top-agent binary completion of 20.6%, and 0% across all models for tasks over 163 minutes.
Primary Sources
- 2.XLang Lab. (2026). "OSWorld-V2 Project Page." — Task-length distribution, domain composition, per-model leaderboard, and a summary of the exposure-attribution methodology. Source of the 83.5% saturation comparison.
- 3.XLang Lab. (2026). "xlang-ai/OSWorld-V2." GitHub. — Benchmark code, tasks, and evaluation harness. External variability is controlled via 31 self-hosted services.
Industry Report
- 4.Chroma Research. (2025). "Context Rot: How Increasing Input Tokens Impacts LLM Performance." Chroma. — An experiment across 18 models. Performance degrades non-uniformly as context grows, and well-structured input can even trigger degradation faster.