Executive Summary

Key Numbers

Source: Nguyen et al., Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials, arXiv:2611.04073 (2026)

A computer-use agent is an AI that looks at a screen and moves a mouse and keyboard the way a person would until the job is done. It sorts files, fixes the formatting in a spreadsheet, changes a setting in an installed program. Because it sees one screen and plays one move, an agent that cannot look a few moves ahead walks into the wrong menu and never finds its way back.

The prescription for that is the world model. Before the agent presses a button, it pictures what the screen will look like afterwards, and it picks the action whose imagined future looks more plausible. The idea runs like a chess player reading a few moves ahead, and the last few years have produced several reports of the method raising scores.

Those reports, though, did not measure the world model on its own. Two other things usually ride along in the same system. One retrieves a tutorial that describes how the task is done and reads it alongside the screen; the other generates several candidate actions and appoints a judge to pick one. Scores rose, and nobody could say whether the imagination, the tutorial or the eye for choosing had done it. That is where the paper starts.

To take the pieces apart, the team needed a task suite with the tutorials already in place, and none existed. So they built two. OSWorld-Tutorial holds 360 Linux desktop tasks, 110 of them carrying human-verified online tutorials and the remaining 250 carrying tutorials written by Gemini 3.8 Flash. OSWorld underneath it is the scoreboard cited most often for computer-use agents, and Pebblous looked at the race to climb it in Agents Are Data. ScienceBoard-Tutorial holds 169 tasks in science software, where the programs covered (Celestia, ChimeraX, GrassGIS, KAlgebra, Lean, TeXstudio) have few enough users that usable online tutorials barely exist. All 169 of those tutorials are generated.

The comparison baseline built on top is RAG-PolicyJudge, RAG-PJ for short. The policy takes the current screen, the history so far, the task instruction and the tutorial, and emits three candidate actions at once; the same model puts on the judge's hat and picks one of them. Turning the chosen action into real screen coordinates falls to UI-Venus-1.5. The judge sees the three candidates, the task instruction, the trajectory so far and the tutorial.

Two ways of adding a forecast sit on top of this. The first describes the next state in words, as earlier work does. Instead of rendering a picture, the model writes a sentence along the lines of "pressing this button opens this window," and the sentence goes to the judge with everything else. The team took two published methods of this kind, WebDreamer and R-WoM, as they are, and had each look one step and then two steps ahead.

The second is the oracle condition at the heart of the experiment. The agent predicts nothing; it simply receives the real screenshot that the action produces. For each candidate action the team saves the state of the virtual machine hosting the environment, presses once, and rolls back to the saved point. No world model can ever be more accurate than that, and not a cent of compute goes into the forecast itself. In exchange, the condition cannot exist in the field, because seeing the future means living it first. If scores fail to rise even here, one conclusion is left. The bottleneck is not in the accuracy of the forecast.

The table below holds the success rate on OSWorld-Tutorial. Read it top to bottom, in the order the pieces were stacked. It starts from the policy alone with no tutorial, then adds the tutorial, then the judge, then ground-truth screens one step to three steps ahead.

Success rate across all 360 OSWorld-Tutorial tasks. Policy and judge are both Qwen3-VL-32B, averaged over three runs. The 15 and the 50 are the actions allowed per task. Source: Table 6, Appendix A of the paper.

The tutorial lifts 5.3 points and the judge adds 3.9 more, for 9.2 together. The ground-truth screens laid over that are worth minus 0.1 points at 15 steps and plus 0.3 at 50. Either figure is smaller than the spread between three runs of the same setup: the row with the judge attached carries a standard deviation of 1.4 points at 15 steps and 3.6 at 50. Better forecasting would not fix it either, since what produced the result above was already a perfect forecast. Free ground truth did nothing.

The values in the table average all 360 tasks together. The same table also carries columns that split the tasks five ways, and two more things show up there. The first is that the 9.2 points did not land evenly. At 15 steps, operating-system tasks went from 65.3% to 91.7% and professional-tool tasks from 60.5% to 83.7%, while everyday-app tasks only reached 66.6% from 63.5% and tasks that cross several apps barely moved, 21.0% to 25.0%. Tutorials and judges work best where the procedure is spelled out. The second is that the ground-truth screens still added nothing in any of the five. Operating-system tasks held at 91.7% to the decimal, and the other four scattered between a 3.4-point loss and a 1.0-point gain. No gain was hiding behind the average.

Science software shows the same pattern. On the 169 ScienceBoard-Tutorial tasks run with Qwen3.8-27B, the policy alone scored 29.0%, the tutorial took it to 44.2%, and the judge to 48.3%. Two steps of ground-truth screens gave 49.3%. The Qwen3-VL-32B column of the same table has the same shape at 40.2% and 40.4%. The paper states that the three rows using ground-truth screens do not differ significantly from RAG-PJ. The environment changes, the model changes, and the conclusion follows along.

The text forecasts did not merely fail to help. On ScienceBoard, adding one to a setup that already had a tutorial and a judge pulled 48.3% down into the 42.6% to 44.8% range. When the imagined future is wrong, an action chosen on the strength of it can be worse than doing nothing at all.

Up to here it reads as though the whole world-model line of research was wasted, and the paper does not say that. The team ran the same OSWorld-Tutorial tasks again with only the tutorial removed. Policy and judge with no tutorial score 43.8% at 15 steps. Two steps of ground-truth screens on top bring that to 48.0%. At 50 steps, 47.4% rises to 50.6%. The gain that had vanished comes back once the tutorial is gone.

Before taking the right-hand bars at face value, look at the condition behind them. The 4.2 points came from ground-truth screens, which is to say from screenshots obtainable only by actually running the action. The numbers change once a world model that can really be built goes into the same slot. At 15 steps, R-WoM gained 0.4 points looking one step ahead and 1.1 points looking two. WebDreamer, from the same text-forecast family, took away 2.4 and 3.7 points instead. Even where no tutorial stands and the world model's slot is open, a forecast written in words changes sign depending on which method goes into it.

With the two tables side by side, the world model's place becomes visible. Where a tutorial is present, perfect foresight has nothing to add; where none is present, perfect foresight is the only thing that adds. The world model served as a substitute for a missing tutorial rather than a bonus stacked on top of one, and as a substitute it currently falls short of a single page of instructions. Next to the 9.2 points from tutorial and judge sits a 4.2-point gain that exists only where a virtual machine can be rolled back.

A failure that arrives after a look at the true future cannot be blamed on the forecast. So the team examined not every failed trajectory, but the ones that failed while holding ground-truth screens. On OSWorld-Tutorial they collected 2,116 such trajectories and had Gemini 3.8 Flash sort the reasons into nine categories, tagging every trajectory with one main reason and any secondary ones. Since the count covers only the failures that survived an exact view of the future, the reasons left in the pile are the ones forecast quality does not explain.

The largest category was a failure to recognize that the task had succeeded. Neither the policy nor the judge can tell whether the state on screen satisfies the instruction. The second largest sits with the judge: handed information about the future, it still cannot work out what is finished and what remains, and it reaches again for an action that already failed. The paper puts no exact percentages for these two into its text and shows them only in a figure, so the ranking is as far as this goes.

The authors drew their own lines around the work. Every measurement ran on Linux virtual machines, and generalization to other operating systems went unchecked. The comparison between human-verified and generated tutorials ran on 110 tasks rather than the full 360. Policy and judge are both untrained Qwen-family models, so a model trained specifically to judge, or a larger commercial model, could land somewhere else. The failure categories are labels from a model rather than from people. The generated tutorials came from a model with web search available, so in an environment that can only search internal documents, tutorial quality may be lower still, as the team notes. 'Perfect Score Without Solving Anything' made the same case for naming conditions one at a time. Under a single benchmark name, what gets handed to the agent moves the number by nearly 10 points, and the spread sits right there in this paper's table.

The paper stops here. Read the measurement from the side of a team weighing workplace automation and two things remain.

First, what changed the agent's score was the document beside it and the eye that read and chose, not the model's imagination. Tutorial and judge made 9.2 points, and dropping foresight into the same slot for nothing left the number unchanged. The first question when bringing an agent in-house tends to be which model, but something comes before it: whether our work procedures are written down as documents a machine can look up. A procedure living only in someone's head, a wiki of screenshots with no words, a manual written against screens from three years ago, none of these have anywhere to go once retrieved.

Second, the most common failure was not recognizing that the job was done. That is a question about the model's reasoning, and before that a question about completion conditions written down nowhere. A task instruction ending at "tidy up the report" leaves no ground for judging what counts as finished. People paper over the gap with intuition; an agent does not know where to stop. Handing work to an agent means preparing the completion criteria with the same weight as the procedure document, and those criteria have to be a condition a machine can check, not a sentence a person reads.

Both are questions of data preparation rather than model choice. Procedures written to be read and completion conditions written to be checked are the training data of the agent era. Three questions follow.

• Where are our work procedures written down? In someone's experience, or in a document that can be retrieved?
• Does that document match the screen as it stands now? A stale tutorial has the same effect as an imagined future that is wrong.
• Who judges "done," and against what? With no condition in writing, an agent knows neither how to finish nor when to stop.

Thanks for reading this far. The paper is at arXiv:2611.04073, and both datasets and the code are on GitHub. If your team has already put its internal work documents into a form an agent can read, we would like to hear how you wrote the completion conditions.