Executive Summary

Key Figures

The first two numbers are the gap that opened when one model ran the same tasks twenty times. The last two describe how the failed trials looked from the outside.

Source: arXiv 2608.19741 (2026-08-20) · 507 business tasks, 14 models, 20 repeated trials per task

Benchmarks that grade agents inside executable environments have clustered around code. A patch can be run against a test suite, and a function call can be checked against a schema. But changing a booking, processing a refund, or updating an insurance claim does not reduce to a single answer. The agent has to take several turns to pull out the information the user left out, follow company policy, and leave the backend in exactly the right state at the end.

Thinkingbox is an execution environment built for that setting. Every task comes with a resettable backend database, an MCP-compatible tool session, a domain policy document, a knowledge base, and a simulated user standing in for the person on the other side. The user side holds a separate set of facts that are not volunteered at the start; they come out only when the agent asks. That mirrors how real requests arrive, missing the pieces the agent needs.

The paper lays out the difference from earlier benchmarks along five axes: whether tools and APIs are used, whether the agent talks with a user, whether there is a stateful backend, whether unrequested side effects are checked, and whether tasks are exposed through MCP servers. SWE-bench works on repository state but has neither tool calls nor user dialogue. BFCL and ToolBench have tool calls but leave no persistent state. AppWorld has state and collateral-change checks but no back-and-forth with a user, and MCP-Atlas runs against real MCP servers without a stateful backend. The closest neighbors are τ-bench and τ²-bench, which pair dialogue with state and check side effects partially. The only row with all five axes filled is Thinkingbox-bench.

The benchmark itself holds 507 tasks split across five domains. The length of the policy document and the number of actions one task requires differ by domain.

Grading looks at the final state of the backend rather than the closing message. Each task carries a set of executable checks, and it passes only when every one of them is true. The checks ask whether the required state change happened, whether the values are right, and whether any change nobody asked for came along with it. Touching the wrong customer record or applying an unauthorized update is caught here. The order of the tool calls is not scored, so a different path still passes as long as the world ends in the same state.

Of the 507 tasks, 477 are decided by those backend checks alone. Only the remaining 30 add a requirement on the final response: not disclosing a hotel's confidential information, or clearly telling the user about an outcome that policy requires the agent to refuse. These are conditions the database state cannot confirm. They are not there to score writing quality. They are an exception carved out for the requirements state checks cannot reach.

Fourteen models went into the same sandbox and ran each task 20 times independently, under the same system prompt, the same tool definitions, and the same policies. GPT-5.4, the best of them, averaged 65.36% pass@1. On its own, that number describes a model that gets it right six or seven times out of ten.

Independent here means each attempt resets the backend to its initial state and opens a fresh tool session. The authors are explicit about why: two attempts at the same task must not share database rows, cached tool state, or side effects, or the numbers measured across repeated trials stop being trustworthy. The twenty runs are not the same conversation replayed either. The model playing the user answers at temperature 0.3, so the user's wording changes from attempt to attempt. So the share of tasks passed in all twenty runs is not a score on one fixed dialogue rerun twenty times. It measures whether the agent lands the same result when the request comes in worded a little differently each time.

Counting the same model's results two other ways changes the picture. Tasks it succeeded on at least once out of twenty came to 91.12%. Tasks it passed in all twenty came to 25.25%. The first number says a successful path exists somewhere. The second says the agent finds that path again every time. A gap this wide means one successful run is no guarantee about the next one.

Counting by task makes the band sharper. GPT-5.4 passed 128 of the 507 tasks all twenty times and failed 45 of them all twenty times. That leaves 334 tasks, two thirds of the set, where the outcome flipped from attempt to attempt. This is both the easiest band to cherry-pick for a demo and the band that causes trouble once the agent is in production and the result stops reproducing.

Line the models up and the ranking flips depending on the metric. On average success rate DeepSeek-V4-Pro is ahead of Claude Opus 4.6; on the share of tasks passed in all twenty runs, the order reverses.

Claude Opus 4.6 never got anywhere on 152 tasks, nearly twice as many as DeepSeek-V4-Pro. Yet on the tasks it could do, it completed all twenty runs almost four times as often. One model works across a narrow range but holds steady inside it; the other reaches wider and cannot repeat itself. Comparing the two on a single average success rate can flip an adoption decision.

Model size does not explain the ordering either. Mistral-Large-3 has hundreds of billions of total parameters and still scored below much smaller Qwen models in several domains. Going the other way, moving from 9B to 27B inside the Qwen family lifted average success from 5.41% to 32.94%. The authors read this as a sign that tool-call formatting, agentic post-training, reasoning mode, and robustness to the environment matter at least as much as nominal parameter count.

Scores also swing hard by domain. Averaged across models, retail sits around 52% while auto insurance sits near 23%. Only GPT-5.4 and Claude Sonnet 4.6 stayed above 50% in all five domains, and even those two differ in character. Sonnet 4.6 held between 51.14% and 68.93% across domains, a narrow spread, while GPT-5.4 drops from 76.33% on retail to 54.60% on consulting. Claude Opus 4.6 took 74.90% on retail and fell to 14.65% on auto insurance, and GPT-5.2 slid from 70.20% on retail to 22.40% on auto insurance. A single aggregate score cannot tell you whether a model is usable for your workflow.

The authors also put their own grading method on trial. Out of 121,680 valid trials from 12 models, they took the 79,853 judged as failures and re-scored them with weaker criteria that never look at backend state. The three criteria tighten in turn: first, whether the conversation closed normally; then, whether at least one state-changing tool call was made; and last, whether the final tool response came back without an error.

Clean termination covered 84.86% of failed trials, and 80.88% went further and made a tool call that actually changed the database. For 67.24% there was not even an error marker on the last tool response. The authors write that a response-only or tool-call-only view would make many incorrect executions appear complete.

Look at the same 79,853 trials through executable state checks and a different scene appears. A database hash mismatch caught 98.95% of them, 77.61% had a wrong field value, and 43.30% left behind an extra change nobody asked for. Another 25.36% had gone the other way and skipped a required change or side effect. A quiet log and a correct state turned out to be two separate facts.

Sorting failures by type shows where things come apart. The authors assigned each failed trajectory one dominant failure signature.

One auto insurance case in the appendix shows the second type in full. A customer whose payment is seven days overdue asks for an extension. The agent looks up the customer, checks the policy, and completes identity verification. Then it pulls the payment arrangement history and gets back a value of two arrangements in the last twelve months. Two is the ceiling for this customer's tier. The agent writes that very number into the ticket description, then grants a third extension.

The closing message reads like this: identity has been fully verified, the due date has been extended, and the account now shows three arrangements in the past twelve months. Then it marks the ticket solved. Every tool call executed without an error and the conversation wrapped up smoothly. The golden state was to grant no extension and deny the request. The agent stated the number that made the action a policy violation, and violated the policy right next to it.

Carry these results into an adoption review and three things change. The first is how many times you run it. Instead of resting on one success, run the same scenario repeatedly and record how many of how many passed. The paper's twenty runs are a research setting, but even with a lower repeat count in an internal review, simply not stopping at one exposes much of that 334-task band.

The second is where the verification data comes from. Response text and tool-call logs are easy to produce and easy to collect, which is how they end up as the default QA signal. What this paper shows is that the same data makes a weak basis for a verdict. Deciding a pass means capturing the backend state before and after the work, comparing it against the correct state, and counting whether any unrequested change came with it. It also means most of the preparation goes into a resettable test backend and a defined correct state for every task.

That preparation is not spent on evaluation alone. In the paper's design, the checks that decide a pass double as the training reward. The same task world can be reused to measure a different agent, and the individual checks can serve as signals for tuning a model. Defining the correct state is not a cost that a single evaluation uses up.

The third is which work you measure on. When the same model takes 74.90% on retail and falls to 14.65% on auto insurance, a public leaderboard ranking cannot stand in for a score on your own workflow. The longer the policy document and the more tangled the eligibility rules, the wider the gap.

There are conditions to read alongside the results. The benchmark's tasks are synthetic reconstructions of a non-public collection of enterprise cases, and the authors do not claim they represent the distribution of real work. Only tasks with a single correct terminal state were kept, so workflows with several defensible resolutions are excluded by construction. The user role is one fixed GPT-5.4-mini deployment, so there is no counterpart who changes their story or stops cooperating. The same GPT-5.4-mini also judges the 30 tasks with response requirements. What happens at a real service desk is harder than this, not easier.

The paper is a preprint posted on August 20 and has not been peer reviewed. The authors are researchers from the University of Pittsburgh, Northwestern University, and UC Irvine together with researchers at Microsoft, and a footnote records that the equal-contribution first authors did the work during a Microsoft internship. The framework is published at microsoft/thinkingbox, with the scenarios and tool servers kept separately at microsoft/thinkingbox-data.

The Pebblous blog has covered a benchmark that catches agents which finish the task and break the rules and a record of 3,607 field failures sorted into types. This paper adds repeated trials as another axis to that discussion. The original is at arXiv 2608.19741.

Pebblous Data Communication Team
August 22, 2026

Academic Papers

• 1.Li, Z. et al. (2026). "One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows." arXiv:2608.19741.
• 2.Yao, S., Shinn, N., Razavi, P., Narasimhan, K. (2024). "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv:2406.12045.
• 3.Barres, V., Dong, H., Ray, S., Si, X., Narasimhan, K. (2025). "τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment." arXiv:2506.07982.

Code & Data Repositories

• 4.Microsoft. microsoft/thinkingbox. GitHub. — Sandbox framework (tb CLI, MCP session proxy, evaluation harness)
• 5.Microsoft. microsoft/thinkingbox-data. GitHub. — 507 task scenarios and tool server package