Executive Summary

Key numbers

Source: Wongchamcharoen et al., arXiv:2608.18554 (2026-08-19)

The benchmark takes its name from the centaur, the old shorthand for a human and a machine bound into one team. Last year Andreas Haupt and Erik Brynjolfsson argued that model evaluation was stuck in an imitation game of autonomous task completion, and that it should move toward centaur evaluations measuring how models augment the performance of others. This study is that call turned into a measurement procedure.

The intuition that the best player is not the best coach came out of research on people first. Experts, having lost the memory of not knowing, misjudge where a novice will get stuck (Hinds 1999), and the curse of knowledge leads them to overestimate what the other party already holds (Camerer et al. 1989). The reproducibility repository the authors released carries the same thought in its name: best-player-not-best-coach.

What a benchmark usually measures is a model's solo ability: reading the prompt, holding the constraints, producing the final deliverable by itself. The paper calls this the automation mode. The design adds a second mode on top of it. In augmentation mode the model under evaluation never touches the deliverable. It writes only a text telling another model how to approach the task.

The format of that text is forced to be identical for every model. It runs 200 to 250 words and has to cover three stages: a requirements check, a plan for structure, and a self-review checklist to run before submitting. Supplying solution content, or any sentence the worker could paste straight into the final answer, is prohibited. The deliverable itself is written by GPT-3.5-Turbo. Because the worker is pinned to a single model, the only thing that changes from condition to condition is the quality of the guidance, not the strength of the team as a whole.

The diagram below sets the two modes against each other. On the top row the model under evaluation produces the answer; on the bottom row the same model writes guidance only, and the fixed worker produces the answer.

There are seven tasks. Menu planning, travel planning, and market trends analysis come from GDPval; operations research analysis, counseling, and tutoring session planning from the Anthropic Economic Index. Tax preparation the authors wrote themselves, to include one task whose answer is settled by rules. Every task carries its own rubric, and five dimensions are scored across all of them: instruction following, accuracy, practical usefulness, organization, and tone.

The scoring is done by four judge models. They are shown two anonymized outputs and pick the better one, and no judge evaluates an output from its own model family. How trustworthy those verdicts are is measured separately. In 99.7% of non-tied comparisons, the response the judge picked was also the one that scored higher on the judge's own rubric. Across 6,265 comparisons seen by at least two eligible judges, the judges agreed on the winner 71.0% of the time, which breaks down into 74.5% in automation and 67.8% in augmentation. Telling good coaching from bad, in other words, was harder than telling good output from bad.

The automation results hold few surprises. GPT-5-Mini takes the best average rank, Claude-Opus-4.8 comes next, and GPT-O4-Mini and Claude-Sonnet-4.6 form the tier below. GPT-3.5-Turbo, the model used as the worker, sits at the bottom. Task by task the rankings stay fairly concentrated as well. It is the familiar picture in which the strong models mostly win.

Carrying that table anywhere else calls for reading the conditions alongside it. What was measured here are single-turn, bounded professional deliverables completed without tool usage. Twice over, the paper attaches the caveat that model orderings from agentic benchmarks, where models take multiple steps, call tools, and interact with an external environment, need not transfer to this setting. The automation ranking is an ordering for this shape of task, not a universal ranking of model capability.

Move to augmentation and that concentration scatters. No assistant model holds first place across the task set. Counseling went to GPT-4.1, market trends to GPT-O4-Mini, menu planning and tutoring to GPT-5-Mini. On operations research, tax preparation, and travel planning, no assistant model took first place at all. The table below sets the winner in each mode side by side.

Even scattered, the winners have a grain to them. GPT-4.1, ahead in counseling, was strong at framing a careful response; GPT-O4-Mini, ahead in market trends, matched a rubric that rewards concise synthesis and actionable interpretation. GPT-5-Mini, which won menu planning and tutoring and was the best assistant on tax preparation, was strong where a task demands tracking constraints to the end and explaining in structure. The two tasks the unaided condition took still had a leader among the assistant models. In operations research that was GPT-OSS-120B, rewarded for optimization framing and trade-off reasoning; in travel planning, Claude-Sonnet-4.6, rewarded for sequencing and budget sense. Different models, in short, are good at different kinds of guidance.

Following a single model makes the split sharper still. Claude-Opus-4.8 was the strongest at writing market trends analysis directly, at a mean rank of 2.05; asked to coach the worker on the same task, it fell to 8.15. GPT-4.1 runs in exactly the opposite direction. Writing counseling responses itself it sits at 7.40, near the bottom; put in the coaching seat it reaches 3.80, the best of the assisted conditions. These are averages over ten independent runs, so neither is a lucky draw.

Set the two modes against each other using the mean ranks of the nine models and the Spearman coefficient comes out at 0.48. The two-sided p is 0.187, short of conventional significance. That is no basis for declaring the two abilities unrelated, but it is no basis either for reading one ranking off the other. In the authors' phrasing, automation ability is an incomplete proxy for assistance quality.

Which way a model leans is fixed per model. Spread the nine across mean ranks and four of them, Claude-Opus-4.8, Claude-Sonnet-4.6, Gemini-3.1-Pro, and GPT-O4-Mini, are stronger in automation than in augmentation. GPT-4.1 stands alone on the other side, and GPT-OSS-120B sits close to the diagonal where the two axes roughly agree. The one model with the best average rank among assistants in both modes is GPT-5-Mini. Averaged across all seven tasks, that is to say, the top of both tables is the same model; what diverges is everything under it and the task-by-task placement.

The possibility that rubric bias produced the divergence was checked too. Across the five general dimensions, from instruction following through tone, most models received similar relative scores rather than spiking on one axis and dipping on another. Nobody won by scoring high on organization. What differed was how the model guided the worker.

The candidate pool in augmentation mode holds one entry beyond the nine assistant models: GPT-3.5-Turbo doing the task alone, with no guidance text at all. Keeping that condition in the same table makes it possible to read directly whether the help helped. On operations research analysis, tax preparation, and travel planning, the unaided condition ranked first. No model's guidance, on those tasks, beat working alone.

Flattening the seven tasks into an average does not change the picture much. The unaided condition lands at a mean rank of 3.79. Exactly one assistant model finished ahead of it, GPT-5-Mini at 3.66. The remaining eight sat, on average, below the condition with nothing attached.

If that sounds strange, research on people has been saying it for a while. Vaccaro, Almaatouq, and Malone, in a 2024 meta-analysis of 106 experiments, found that human-AI combinations frequently underperform the better of the human-alone and AI-alone conditions. MathTutorBench, which evaluates the pedagogical quality of LLM tutors, likewise found that stronger problem-solving ability does not automatically imply better teaching. What this study adds is that same observation moved into the space between two models, with a ranking attached to how much each model's guidance is actually worth.

Before these numbers travel into a real workplace, the scope of the design has to travel with them. The guidance is delivered once and the worker cannot ask back. The assistant may not supply solution content, only shape the process. The worker is pinned to GPT-3.5-Turbo. The authors call these conditions a conservative lower bound on augmentation, and leave widening them into multi-turn workflows as the next step.

How far the two rankings come apart varies by task. The paper computes, for each of the seven tasks, the correlation between the automation and augmentation ranks of the nine models. The values run from 0.85 down to -0.04.

Only one of the seven is statistically distinguishable from zero: tax preparation. It survives even after tightening the threshold for having run seven comparisons. And that is the task the authors built themselves, the one governed by rules, where finding a discrepancy settles the matter as right or wrong. Where there is a single path to the answer, the model that solves well also tends to guide well.

Travel planning sits at the other end. At -0.04, effectively no relationship survives between the two rankings. Fitting a budget, sequencing a route, deciding how to handle what is uncertain: none of that has a single correct answer. On tasks like these, writing well yourself and guiding someone else to write well point in different directions. Market trends at 0.10, operations research at 0.17, and counseling at 0.25 sit in much the same band.

Translated into an adoption decision, it comes out like this. For rule-bound work, seating a top model from the existing leaderboards in the coach's chair will not go far wrong. The more a task turns on judgment and composition, the less that table supports the decision, and the less there is to do but measure against your own work.

A ranking says who guided well; it does not say what guiding well consists of. Across all seven tasks, the authors put the guidance texts behind the two highest-ranked outputs next to those behind the two lowest-ranked outputs and read them against each other. The frame for judging came from the education literature on scaffolding. The two sets came apart in three places.

Scaffolding is the term Wood and colleagues coined in 1976 for the help that lets a learner reach a task beyond their unassisted efforts. The point is that the expert guides the learner through the task rather than completing it for them. Van de Pol and colleagues added three properties in 2010: support has to be calibrated to the learner's current level, withdrawn as competence grows, and ultimately handed back so that responsibility rests with the learner. Because guidance here is delivered once, fading cannot apply, which leaves calibration and transfer of responsibility as the two usable criteria. The three splits below are precisely a check on those two.

5.1Does it avoid repeating what the prompt already said?

On market trends analysis, the first-ranked guidance instructs the worker to plan "2 to 3 succinct bullets" per trend pairing "core observation, primary driver(s), and potential market implication," with "a short preface about methodology and data vintage" up front. The ninth-ranked guidance on the same task recites the list of factors already enumerated in the prompt: supply, demand, weather, storage, LNG exports, infrastructure, policy. The worker is reading that prompt too, so the repetition adds nothing.

5.2Does it set the order, or hand the ordering back?

On travel planning, the third-ranked guidance forces assumptions and clarifying questions "before the provisional itinerary" and nails down a named sequence: introduction, budget breakdown, day-by-day itinerary, conclusion. The ninth-ranked guidance says to "determine sequence and hierarchy of information for clarity and flow." It reads like an instruction, but the decision travels straight back to the worker. It then adds that each section should be populated "with neutral placeholders."

5.3Does it avoid forbidding what the task requires?

The most harmful case came out of the operations research task, which explicitly requires two practical solutions presented with their trade-offs. The ninth-ranked guidance instructed the worker to establish a workflow for integrating indicators and risks "without delving into specific solutions or analysis." It prohibited the task's core deliverable. The third-ranked guidance on the same task does the opposite: "Present two distinct, practical strategies. For each, explicitly state the expected benefits, potential drawbacks, and associated risks." What the scaffolding literature calls direction maintenance, the job of pulling a learner back when they drift off the objective, ran here in reverse.

Put the three together and the common feature of bad guidance surfaces. Taken as sentences, all three are impeccable. Determine sequence and hierarchy, weigh balanced trade-offs: either would pass unremarked in any meeting note. The trouble is that those sentences repeat what the worker already knows, hand decisions back, or forbid what the task asked for. It doubles as a yardstick to hold against your own internal prompts and agent instructions.

Why the distinction matters socially, the paper notes separately. AI deployed to replace and AI deployed to assist leave different marks on the labor market, and automating one worker's tasks can create room to assist elsewhere in the same production process. In a nationwide survey of 1,500 U.S. workers across 104 occupations, equal partnership with AI rather than replacement was the dominant preference in nearly half of the occupational categories studied. Yet most of the rankings currently in circulation measure only one of those two axes. Hence the authors' request to evaluation bodies and model developers to carry both. Any evaluation that tests only one, as they put it, is incomplete by design.

The paper writes down its own limits as well. Final verdicts came from models rather than people, which makes this simulation evidence at scale rather than expert judgment, and the more expertise a task carries, counseling or tax among them, the more human validation it needs. There are only seven tasks and a single fixed worker. The authors call the work an initial pilot application rather than a definitive map. The paper is at arXiv:2608.18554, and the per-task judgments can be inspected in the interactive dashboard the authors released.

Primary Source

• 1.Wongchamcharoen, P. K., Gulati, K., Fong, M. M., & Nagaraj, A. (2026). "CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks." arXiv:2608.18554.

Cited Prior Research

• 2.Haupt, A., & Brynjolfsson, E. (2025). "Position: AI Should Not Be an Imitation Game: Centaur Evaluations." Proceedings of the 42nd International Conference on Machine Learning.
• 3.Hinds, P. J. (1999). "The Curse of Expertise: The Effects of Expertise and Debiasing Methods on Predictions of Novice Performance." Journal of Experimental Psychology: Applied, 5(2), 205–221. doi:10.1037/1076-898X.5.2.205
• 4.Camerer, C., Loewenstein, G., & Weber, M. (1989). "The Curse of Knowledge in Economic Settings: An Experimental Analysis." Journal of Political Economy, 97(5), 1232–1254. doi:10.1086/261651
• 5.Vaccaro, M., Almaatouq, A., & Malone, T. (2024). "When Combinations of Humans and AI Are Useful: A Systematic Review and Meta-Analysis." Nature Human Behaviour, 8. doi:10.1038/s41562-024-02024-1
• 6.Macina, J., Daheim, N., Hakimi, I., Kapur, M., Gurevych, I., & Sachan, M. (2025). "MathTutorBench: A Benchmark for Measuring Open-Ended Pedagogical Capabilities of LLM Tutors." arXiv:2502.18940.
• 7.Wood, D., Bruner, J. S., & Ross, G. (1976). "The Role of Tutoring in Problem Solving." Journal of Child Psychology and Psychiatry, 17(2). doi:10.1111/j.1469-7610.1976.tb00381.x
• 8.Van de Pol, J., Volman, M., & Beishuizen, J. (2010). "Scaffolding in Teacher–Student Interaction: A Decade of Research." Educational Psychology Review, 22. doi:10.1007/s10648-010-9127-6