Executive Summary

Key figures

Source: ParaRecover (arXiv:2609.12345, 11 Sep 2026)

Agent evaluation mostly comes down to one question. Did the agent get the answer, and did it reach the end? Right answer, 1. Wrong answer, 0. The appeal is obvious: nobody has to make a judgment call, and the result collapses into a single number. That number also says nothing whatsoever about what the agent did on its way to the answer.

The paper's introduction puts the gap in one sentence.

“However, successful task completion does not necessarily imply reliable agent behavior. An agent may still complete the task through redundant calls, repeated trial and error, or incorrect recovery strategies, leading to high cost in real systems.”

Redundant calls, repeated trial and error, and botched recovery strategies all still finish the task. They finish it, and a real system pays for it. Tokens go out, external APIs get hit more often, the clock runs longer. A completion metric records none of that cost.

Other groups have tried to look at process. In the comparison table the paper assembles, ToolSandbox and TRAJECT-Bench already cover structured trajectories and parallel calls, and BFCL covers parallel calls too. Those benchmarks stop at whether a tool call succeeded and whether its format held. A separate line of reward-model benchmarks asks which of two whole trajectories is better, but that exercise ranks trajectories that mostly went right. No one had handed an agent a broken trajectory and started the clock there. In the table, exactly one row is ticked for tracing an error back to its source and for deciding to replan, and that row is ParaRecover.

The tasks come from the BUTTON dataset, and tool execution runs in a simulation rather than against real APIs. The authors explain the choice: the target is judgment under a mid-run error state, not the reliability of an external API, so they wanted a controlled environment. That buys reproducibility and the ability to inject errors, and it gives up the latency and shifting state of a live system. The authors list this first among their limitations.

An error taxonomy sits at the bottom of the benchmark. The agent's execution is modeled as a directed acyclic graph, where nodes are subtasks and tool calls and edges are the dependencies between them. On top of that model the authors carve out fourteen ways things can go wrong. Alongside the tool-call-level mistakes earlier work already covered, they mined large volumes of execution logs for the mistakes that only appear when work runs in parallel across several rounds.

The paper also shows that these fourteen boxes were not invented at a desk. The authors took real execution logs from GPT-4o-mini, GPT-4o and GPT-5.1 and counted how the errors fell out. The most frequent were wrong parameter types, invalid values, redundant calls, and calls that needed to happen but did not. All fourteen types showed up naturally in real runs, and models with more parameters and stronger reasoning made fewer errors.

There are 10,626 cases. LEVEL-1 holds 4,033 of them, where the error is still confined to the round just before. LEVEL-2 holds 6,593, where the error slips through undetected into the next round or stacks up with fresh errors on top. Of the whole set, 2,812 cases were picked by hand out of real execution logs, about one in four, and the remaining 7,814 were made by planting one of the fourteen types into a clean trajectory.

Of the fourteen boxes, the one that filled up fastest in LEVEL-1 was wrong serialization, at 10.09% of the 4,033 cases and a clear margin ahead of second place. This is the mistake of lining up two nodes one behind the other when neither needs anything from the other. It did not happen for lack of a rule. The prompt the authors gave each model spells it out: add a dependency only when the return value of the earlier task fills an argument of the later one, and otherwise attach the task to the root node. Handed the rule in writing, the models still drew edges that nothing required.

Even with the errors sorted into types, no score comes out until there is a marking scheme. The yardstick the authors built has three axes with five items each, fifteen items in all. They call it the SDE rubric, after the initials.

• Structural Integrity (S) asks whether the plan holds up as a form. Does the reasoning match the graph, do nodes placed side by side secretly depend on each other, can the execution engine read the graph at all.
• Diagnostic Reasoning (D) asks how deeply the agent read the failure. Did it reach the root cause rather than the visible symptom, did it work out how far downstream the error had spread, did it guess without evidence.
• Evolutionary Strategy (E) asks whether the rewritten plan is any good. Did the agent change only what needed changing, did it stay on the original goal, does each new step have a reason to exist.

Thirteen of the fifteen items take one of three values: 1, 0.5, or 0. The authors cite prior work showing that finer items with fewer options raise both the accuracy and the consistency of marking. The remaining two items never meet a judge at all. Whether the execution engine can read the submitted graph is settled by a script with a pass-or-fail flag, and the last item, E5, is a formula. It divides the optimal number of rounds the reference solution needs by the number of rounds actually spent, caps the ratio at 1, and multiplies by whether the task succeeded. An agent that reaches the same answer after ten rounds instead of six loses points.

The thirteen judgment items are too many for human marking at scale, so another language model served as judge. The authors drew 400 cases per item and compared the judge against human labels: exact agreement 0.8067, mean absolute error 0.1208. Because the judge model belongs to the same family as models near the top of the leaderboard, the authors added a cross-check with a separately calibrated GPT-4o judge on five models. The LEVEL-1 ordering did not move. In LEVEL-2 the top three swapped places and first went to a DeepSeek model. The judge is still a model either way, and the authors note that bias can creep in on complex or ambiguous cases.

That same agreement sample carries something else. Item by item, the lowest human average among the thirteen belongs to Recovery Closure (E2), at 0.2350 out of 1, under half the highest-scoring item. Agents patch the point that blew up and get almost no credit for closing out the nodes, states and responses hanging beneath it. The marking prompt did instruct judges to be strict, which is worth allowing for. Even so, the overall human average of 0.4736 came in slightly below the judge's 0.4813. It is hard to lay the low scores on a harsh model judge.

Sixteen models from six families ran on the same prompts: OpenAI, Anthropic, Google, Alibaba, Zhipu and DeepSeek. The authors measured DeepSeek flash separately in thinking mode, so the table carries seventeen rows. The highest average belongs to Claude Opus 4.6, with 69.23 on LEVEL-1 and 66.85 on LEVEL-2. The scale runs to 100, and not one row clears 70.

Where the points go missing is visible in the leading model's own results.

This shape is not peculiar to the leader. Across all seventeen rows in the table, strategy revision is the lowest of the three axes. Models more or less locate where the error happened, and all of them come apart when they have to rewrite the plan precisely and efficiently. Every model also scored lower on LEVEL-2 than on LEVEL-1, which says that tracing an error back gets much harder once it has spread past a single round. That holds without exception in the averages, while axis by axis four models nudge their strategy revision score slightly higher on LEVEL-2.

One scene of points draining away appears in an appendix. Node 3 had the wrong argument type and came back empty, on a task that was not especially complicated. Rather than fixing that call and retrying it, the agent pinned the failure on an upstream task, spun up node 7, and restarted the whole workflow. A cascade of errors followed. The reference solution needs six rounds; the run took eighteen, and round efficiency (E5) fell to 0.33. The run still counts as complete. Completion asks only that the agent reach the response node inside a cap of forty rounds, so a six-round task finished in eighteen rounds earns a 1 in the table.

The gap widens on smaller models. GPT-4o mini drops from 36.17 to 30.05 in round efficiency on LEVEL-2, where errors propagate across rounds, while its completion rate barely moves, 89.32% to 88.84%. The process got worse and the outcome metric did not follow it down.

The split across error types is just as clear. Parameter mistakes, which break execution visibly, get caught reasonably well. The hard families are structure and tools, and within those the lowest diagnostic scores for both models landed on redundant calls. The paper gives the reason.

“Unlike other errors, it typically does not cause explicit execution failures but manifest as “plausible yet ineffective” implicit errors, making them difficult to detect—a common challenge for agents.”

A redundant call leaves no failure signal. The call succeeds and a value comes back. It is plausible, and useless. On redundant-call tasks DeepSeek flash earned a structural score of 83.44, the best of the fourteen types, and a diagnostic score of 69.31 on those same tasks, the worst. The form of the plan is flawless, and that is why nobody catches it. The error log stays empty, so the model walks past it and so does a human reviewer. This is exactly the class of mistake that completion-only evaluation is blindest to.

The part worth attention is not the leaderboard but the paper's final experiment. The authors kept only the trajectory pairs more than five points apart, assembled 3,187 preference pairs out of them, and trained Qwen3-8B on that set.

A prompt-only comparison ran alongside it. Attaching three worked examples to the prompt moved the average from 43.86 to 44.47. That is 0.61 points. The version trained on the error labels landed at 49.10, a gain of 5.24. The hardware was a single server with eight RTX 3090 cards, the method was LoRA, and the preference stage ran for one epoch.

An obvious suspicion attaches to this. If the training data was built with their own rubric, of course the rubric score goes up, and that is circular. The authors raise the same worry and remeasure with metrics no judge model touches, four of them computed straight from execution logs and environment results.

The share of unexecutable plans fell by nearly half, and completion rose along with it. The point of the table is that the numbers rose on a yardstick no judge model holds. The columns, taken one at a time, also show which stage changed what. Unexecutable plans were already down to 6.71% after supervised fine-tuning, and the preference stage opened up a different column, tool-use efficiency, which jumps from 21.29 to 26.20. A name tag on each failure did not stop at diagnosis. It repaired the model. This is one confirmation on one model, and the labeling side took several passes of human work.

The question Pebblous has held onto for a long time is where data came from and what it passed through to become the value it is now. An agent's failure records stand in front of the same question. What produced the gain in this paper was not a bigger model and not a longer prompt. It was 10,626 lines, each one annotated with what kind of failure it was. Label design turned out to be the performance.

So a reader who takes this work as news of one more benchmark has read only half of it. Four things are worth checking against your own team's agent pipeline.

• What shape are last week's failed jobs sitting in right now? If they survive as a single cell reading "failed", you can count them but you cannot fix them.
• If you do split failures into types, how many boxes do you need? A taxonomy that looks only at tool names and parameters has nowhere to put the dependency mistakes that parallel execution creates.
• How many jobs marked successful got there through redundant calls? This mistake never reaches an error log, so it stays invisible unless someone counts it separately.
• Who attaches the labels? A label a person adds once becomes training data the next time around, and records nobody labels are just storage.

Thank you for reading this far. Every figure and quotation in this article can be checked at the source, in the paper posted on arXiv and the repository the authors opened. We would be glad to hear how many boxes your organization uses to write down an agent's failures.