Executive Summary

Tool-calling agents are usually evaluated on execution traces: a static dataset holding the user's instructions, the agent's responses, and the tool calls exchanged in between serves as the test to grade against. But real production data is hard to use as a test. It holds sensitive or proprietary information, and there is rarely enough of it to support comprehensive pre-deployment testing. So the practice of replacing or augmenting real data with synthetic data to build evaluation benchmarks has taken hold.

That is where the core difficulty arises. How do you confirm, in numbers, the relationship between this synthetic data and the real data it stands in for? SynAE is a framework built to answer exactly that question. It looks at synthetic data across four axes: task instructions and intermediate responses, tool calls, final output, and the downstream performance you observe when you actually grade agents with the test.

Each axis is then measured on three properties. Validity asks whether the synthesized instruction-response plausibly accomplishes the task goal; fidelity asks how faithfully it reproduces the distributions and patterns of the real data; diversity asks how wide a range it covers. Validity answers "does this make sense," fidelity answers "does this resemble reality," and diversity answers "is this wide enough."

▲ Original Pebblous diagram (reconstruction of the SynAE evaluation structure)

The researchers deliberately injected representative synthetic-data generation methods to see what SynAE would catch. What stands out most is that two generation methods move in opposite directions. Blank Filling masks some of the original tokens and lets the model fill them in; as the masking ratio climbs, it drops fidelity but actually raises diversity. Oversampling does the reverse. It duplicates a sequence to inflate the set, and as the duplication ratio climbs, fidelity and diversity collapse together.

In other words, even within the same "synthetic data generation," two properties can move in the same direction or in opposite ones depending on the method. If you take diversity alone as your standard for a good test, you will catch Oversampling's diversity collapse but wave Blank Filling's fidelity collapse right through. One metric cannot see where the other one breaks.

▲ Original Pebblous diagram (reconstruction of SynAE's per-method trade-offs)

The divergence does not appear only between different properties. Even within fidelity, the individual metrics disagree. In-Context Generation produces new data from a few examples, and as the example count grew it improved one fidelity metric (KNN-precision) while another (KNN-recall) got worse. "Fidelity is high" cannot be lumped into one phrase; the conclusion flips depending on which metric you looked at.

What it concretely looks like for a synthetic test to drift away from reality shows up in a case SynAE caught. Swap the original data's "recommend art landmarks" task for "sports landmarks" to boost diversity cheaply, and the surface form stays intact while the instruction, the tool calls, and the final output no longer line up—so the task itself fails to hold together. You cannot spot that mismatch from the surface of the test; only measuring several properties together brings it out.

The NeMo case from the Executive Summary captures the whole story. The validity of two tests built with the same tool is effectively equal at 0.98 and 0.99, yet fidelity splits to 0.71 and 0.94. Had you measured only validity and moved on, you would have missed that the test built with GPT-4o-mini is far closer to real production patterns.

SynAE's fourth axis, downstream performance, produces an intriguing paradox. Even when you push Blank Filling's masking high enough to wreck the tool-call metrics, the ranking you get by grading several models with that test barely moves. Because a high-performing model still outperforms a low-performing one even when tool-use patterns are distorted, the relative ordering among models holds.

That result reveals that two questions are, in fact, different questions. "Can this test rank the models?" and "Does this test faithfully resemble the real world?" are separate, and they yield separate answers. A test that is flawless for ranking can be a failing grade on the standard of reproducing reality. The reverse holds too.

▲ Original Pebblous diagram (reconstruction of SynAE's ranking-fidelity paradox)

This blog recently covered a benchmark that grades data-analysis agents skill by skill (cleaning, joining, anomaly detection), arguing that a single total score erases an agent's strengths and weaknesses, so the grading rubric should be broken into finer pieces. SynAE aims one layer deeper. No matter how finely you break the rubric, if the test data it stands on is flimsy, that fineness becomes an illusion. The question underneath remains: how faithfully, and how widely, does the skill-labeled task data itself capture reality?

It is a different register from the recent piece on coding agents and scientific reproduction. That piece asked a question about capability: why the agent can't solve the problem. This one asks a question about the evaluation tool, whether the test that posed the problem resembles the real world at all. The first question faces the model; the second faces the data.

Diagnosing data quality has always been treated as a multidimensional problem. That you must weigh several yardsticks together (accuracy, completeness, consistency) is a principle data-quality research has long emphasized. What SynAE newly did is apply that multidimensionality to a concrete and urgent case: the synthetic tests used to evaluate agents. Requiring that validity, fidelity, and diversity be viewed together across four axes is the latest edition of the old principle that data quality must not be compressed into a single number.

So the first question an organization considering a synthetic benchmark should ask changes. Not "what score does our model get on this benchmark," but first: is this benchmark's test valid, does it faithfully reproduce our reality, is it diverse enough? Trusting a model's score begins with diagnosing the test.

• 1.Wang, S., Maddi, A., Lin, Z., Fanti, G. (2026). "SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations." arXiv:2605.22564. — Carnegie Mellon University / Microsoft Research; a framework measuring synthetic data across four axes and three properties.