Executive Summary

Multi-agent systems are already deployed everywhere, yet it was never clear when collaboration among several agents actually beats one strong single agent. A joint team from Google DeepMind and MIT answered the question with a controlled experiment that held the prompt, the tools, and the compute budget fixed while varying only the coordination structure and the model's capability. This article reads what they found.

The variable that best separated collaboration's gains from its losses was neither the architecture nor the number of tools. It was how well a single agent solved the task on its own in the first place — its baseline performance. Once baseline accuracy climbed past roughly 45%, adding agents brought almost no improvement, and the very same collaboration that lifted decomposable financial reasoning sharply pulled down sequential planning, where each step leans on the decision before it.

So "how many agents should we add" turns out not to be a matter of architectural taste but a measurable decision. This article lays out what to measure first in order to make that call.

~45%

Capability-saturation threshold

Above this baseline, adding agents stops paying off

+80.8%

Financial reasoning

Gain from collaboration on a decomposable task

−70.0%

Sequential planning

Drop from collaboration on a step-dependent task

94%

Gain-sign prediction

Rate at which the baseline rule called the sign of the gain

1

Agents Added on Instinct

For the past two years, systems that wire several agents together have carried something close to an article of faith: two are better than one, and one orchestrator makes it better still. Benchmark reports seemed to back the intuition up, since results kept showing scores rising as more agents were added.

The trouble is that the source of those higher scores was tangled together. Adding agents usually means more tool calls, more compute budget, and more total tokens as well. So it was never clear whether the score went up because of the collaboration structure itself or simply because more resources had been burned. The claim that "multi-agent is better" hardened without ever controlling for that confound.

The study Google DeepMind and MIT published in Nature Machine Intelligence aims squarely at this point. The team fixed the task prompt, the available tools, and the compute budget, leaving only the coordination structure and the model's capability as variables. Hold the resources still and change only the structure, and any difference in score belongs entirely to the way agents collaborate.

That a Big Tech lab (Google) and academia (MIT) put their names on the work together, and that its conclusion actually pumps the brakes on the multi-agent marketing those same companies sell, adds weight when you read the result.

2

The Predictor 260 Experiments Found

The experiment spanned 260 configurations. Across six benchmarks — web browsing, financial reasoning, planning, work automation, software engineering, and terminal tasks — the team crossed five coordination structures with three LLM families. The coordination structures were a single agent (the baseline), parallel independent runs with no communication, a centralized orchestrator-led setup, a fully connected decentralized setup where everyone talks to everyone, and a hybrid that mixes hierarchy with limited peer communication.

Single Independent Centralized Decentralized Hybrid
▲ The 5 coordination structures compared in the study — original Pebblous diagram (Fig. 1 reinterpretation)

Filling the three LLM families with OpenAI, Google, and Anthropic models was a safeguard against any single vendor's quirks skewing the conclusion. Only by putting all three companies' models into the same experimental frame can you say the finding reflects a property of the collaboration structure itself, rather than the habits of one model.

Across this wide grid, the variable that most robustly predicted collaboration's gains and losses was surprisingly simple. It was neither the type of architecture nor the number of tools. It was how well a single agent solved that task on its own — that baseline performance explained the success or failure of collaboration better than anything else. Whatever structure you layered on top, the skill of the single agent you started from had already decided much of the outcome.

When we design collaboration, the question we usually ask first is "which architecture should we use." This study says there is a more important question in front of it: "how well does a single agent do this task alone?"

3

Past 45%, Adding Agents Stops Paying Off

If baseline performance is the predictor, the next question follows naturally: at what point does collaboration stop mattering? The empirical answer the team pulled from the data is about 45%. Once a single agent's baseline performance crossed that line, adding more agents produced almost no improvement. The team calls this the capability-saturation threshold.

The mechanism is one of marginal returns. For a model that is still shaky, another agent's verification and correction create real room to improve. For a model that already does well, collaboration has little gap left to fill. In that state, adding agents starts to cost more in mutual coordination than it returns in gains. As capability rises, the marginal benefit of collaboration approaches zero while the coordination cost stays put.

0 Threshold ~45% Single-agent baseline performance (%) 0% 45% 100% Collaboration performance change
▲ Single-agent baseline performance vs. collaboration gain — original Pebblous diagram (Fig. 2 reinterpretation)

The threshold was not just an observation. A decision rule combining baseline performance with team size correctly called the sign — positive or negative — of the multi-agent gain in 94% of the 16 model-by-benchmark combinations across SWE-bench Verified and Terminal-Bench. That is a signal that you can gauge in advance whether collaboration is worth adding.

4

Tasks That Work, Tasks That Fail

If baseline performance decides whether collaboration succeeds, the structure of the task decides which direction that success runs. The same coordination structure lifted performance sharply on some tasks and collapsed it on others. The per-domain figures in the preprint show the gap in sharp relief.

Task type Collaboration effect Why
Financial reasoning (decomposable) +80.8% Splits into independent sub-problems, so parallel agents can each take one
Sequential planning (step-dependent) −70.0% Each step leans on the previous decision, so coordination cost and error propagation swallow the gain

The crux of the difference is decomposability. When a task can be split into mutually independent pieces, agents take the pieces in parallel and collaboration turns straight into a gain. When one step's output becomes the next step's input — a sequential-dependency task — adding agents raises the cost of keeping them aligned, and a single error travels down the chain.

Pebblous's earlier piece on how 100 agents pushed vLLM inference speed up 5× reads differently through this frame. That success came not from piling on agents for their own sake, but because the inference bottleneck was a decomposable problem that many agents could work in parallel. Attach the same collaboration to sequential planning and the result would likely have run the other way.

The second most significant predictor, the number of tools, follows the same grain. The more tools a benchmark involves, the heavier the burden on agents to coordinate tool use, and the more that shaves off collaboration's net gain. Likewise, freely communicating decentralized structures had no central verification and suffered worse error propagation — a hint that connecting everyone is not always best.

5

How Much to Trust the Prediction

For the proposal to decide collaboration by measurement to be persuasive, it has to be honest about how trustworthy that prediction is. The team reports that a regression-based prediction model picked the optimal architecture correctly in 87% of held-out configurations within the same domain. Inside familiar territory, in other words, you can compute "how many to add" fairly reliably.

That 87%, though, is the accuracy of relative ranking — which structure is better — and nothing more. Even within the same domain, the power to predict the absolute score itself sat at a coefficient of determination (R²) of only about 0.37 to 0.41. Put another way, this method is stronger at sorting the direction of "is collaboration worth adding or not" than at precisely predicting "what score you will get." And direction is usually what practice needs.

The boundary shows up when you cross domains. When one dataset was held out entirely from training and the model was asked to predict absolute performance in that unfamiliar domain, R² fell as low as −2.09 — worse than simply guessing the mean. Transplant a threshold and a rule learned in one domain directly into a completely different one, and they can break.

Architecture-selection accuracy 100% 0% 87% In-domain (held-out configs) Absolute-performance explanatory power (R²) 0 R²≈0.4 In-domain R²=−2.09 Cross-domain
▲ Prediction reliability: 87% in-domain accuracy / R² 0.37–0.41 vs. −2.09 cross-domain — original Pebblous diagram (Fig. 3 reinterpretation)

So the right way to put this study to work is not "plug the number 45% straight into our domain." It is "measure our own single-agent baseline on our own task and model, and re-confirm the threshold within our own domain." The method transfers; the numbers have to be re-measured for each domain.

6

Measuring the Value of Collaboration

What this study leaves the practitioner is not skepticism. It is an order of operations for deciding whether to add collaboration by data rather than by taste or trend. When you design a new agent system, three things come first to measure.

First, the single-agent baseline. Before layering on a collaboration structure, measure how well a strong single agent solves the task. If it already solves it well, start by questioning why you would add more agents at all.

Second, the task's decomposability. Tasks that split into independent sub-problems reward parallel collaboration, while strongly step-dependent tasks get dragged down by coordination cost and error propagation as agents are added.

Third, re-measurement in your own domain. Do not borrow another domain's threshold; re-confirm it with your own data and your own model. The method transfers, but the numbers do not.

In the end, "how many agents should we add" was never a matter of drawing an architecture diagram. It was a matter of measurement — of the baseline, the decomposability, and the domain data. The value of collaboration can be estimated to some degree even before you attach it. When Gartner analyzed 107 agentic-AI cases, the observation that the few projects delivering ROI all touched "execution" rather than "proposal" points the same way. Whether collaboration or autonomy, value is decided not by the elegance of the structure but by what it actually improved.

Editor's Note

This is why Pebblous has always read agents and autonomy as a data problem. Whether collaboration is a gain or a loss is settled not by architectural taste but by measurable data — whether the task is decomposable and whether the single agent already sits above the threshold. If your team runs a multi-agent pipeline, these three measurements are worth using to take a fresh look at your own structure.

R

References

Academic papers

Industry & press