Executive Summary

When we decide whether to put a data analysis agent into production, we usually look at one number on a leaderboard: the success rate. But data work doesn't break in only one place. An agent can read the schema correctly yet botch the join, clean the data cleanly yet miss the outliers, write code that runs yet ignores a business constraint. This article looks at AgenticDataBench (arXiv 2607.01647), a benchmark aimed squarely at that gap.

The key move is changing the unit of scoring. Instead of measuring whether a task was completed as a single total score, the benchmark attaches ground-truth labels to individual data skills—cleaning, joining, anomaly detection, visualization, schema exploration—and grades each one. It mines 433 representative skills from 6,510 high-quality Stack Overflow solutions via hierarchical clustering, then combines them into 344 tasks. With each task requiring an average of 23.5 skills, two agents can share the same total score yet reveal very different strengths and weak spots.

The question this design raises lands directly on anyone evaluating adoption. When we trust an agent, what are we actually verifying—the outcome, or the ability?

433

Representative data skills

Extracted from 6,510 Stack Overflow solutions via hierarchical clustering

344

Benchmark tasks

Averaging 23.5 skills combined per task

15

Industry domains

Including 5 real B2B fintech use cases

38%

Best-model pass@1

Adjacent evaluation (arXiv 2603.20576) — the totals themselves are still low

1

Solved Is Not the Same as Trusted

A leaderboard success rate is a convenient number. It compresses how many of 100 tasks an agent finished into a single figure, which makes it easy to line models up and pick one. The trouble is that the number can't tell "finished" apart from "did it right." In a data pipeline, an error early on propagates quietly downstream, and the final output gets dressed up in a plausible table or chart. If the success verdict only inspects the shape of the deliverable, a join that went sideways or a missing value that slipped through never shows up in the score.

For an organization, that distinction is anything but trivial. Say an agent cleaned some sales data, aggregated it by department, and produced a report. A total-score benchmark confirms only that "a report came out." Whether that report double-counted duplicate records, mixed currency units, or waved an anomalous transaction through as normal is a separate matter. An agent with a high success rate can still slip repeatedly at one specific point, and that slip stays hidden behind the total.

So the question we need is this: what exactly does the number we call a success rate guarantee, and what does it miss?

Ingest Clean Join Analyze Report ! ✓ hidden error marked success An error at the join stage passes downstream, yet the final report still reads as a success

▲ Pebblous original diagram (reinterpreted from AgenticDataBench's problem framing)

2

How to Score a Skill

AgenticDataBench answers by raising the resolution of the scoring. Rather than checking only whether a task passed or failed, it defines the recurring manipulation patterns that make up data science work as units called "skills" and gives each one a ground-truth label. Cleaning, joining, anomaly detection, visualization, schema exploration, business-context reasoning—each becomes a separate object of scoring. Whether an agent reads the schema well but breaks down at the join, or cleans fine but chooses visualizations poorly, becomes visible skill by skill.

The skill list wasn't set arbitrarily. The authors clustered 6,510 high-quality data science solutions accumulated on Stack Overflow with hierarchical clustering to extract 433 representative skills—reverse-engineering, from the data itself, the moves that real developers have repeated in handling data. They then combined those skills into 344 tasks, designed to maximize the diversity of combinations, so that a single task touches 23.5 skills on average.

The scope stays close to real practice, too. Alongside Kaggle, UCI, government, and academic data, the authors gathered 97 real datasets across 15 industry domains, including one large fintech company's actual B2B use cases. The data a single task handles approaches 500MB on average, and the reference solution code runs past 100 lines, so these are field-scale jobs rather than toy examples handed over on a plate. The benchmark and test bed are released under Apache-2.0, so you can inspect the skill-by-skill map of strengths and weaknesses directly rather than a leaderboard total.

6,510 high-quality Stack Overflow solutions hierarchical clustering 433 representative skills combined 344 tasks 23.5 skills per task on average 15 industry domains · 97 real datasets including 5 fintech B2B use cases

▲ Pebblous original diagram (reinterpreted from AgenticDataBench's construction pipeline)

The core shift is changing the scoring question from "how many did it get right" to "what can it do, and where does it break down." A total gives you a ranking; skill-level labels give you a diagnosis.

3

What a Total Score Misses

The evidence that a single total score is hard to judge by has already piled up in adjacent benchmarks. On DA-Code (arXiv 2410.07331), which evaluates data science code generation, even the top model reached only 30.5 points, and on the same tasks the completion rate ranged from 38.7% to 99.5% depending on the agent framework. In other words, the total alone makes it hard even to decide which framework to pick.

Another evaluation on data question answering (arXiv 2603.20576) tells a similar story. Among five LLM-based agents, the best performance stopped at 38% pass@1. A commercial data agent scored 7 percentage points better than the same model's baseline approach, yet both approaches failed entirely on queries requiring value extraction from unstructured text. It's a concrete case that a slightly higher total does not make one weak skill disappear.

Completion rate by framework (same tasks) 38.7% 99.5% DA-Code top model total score 30.5 0 50 100 On the same tasks, completion rate still swings wildly by framework

▲ Pebblous original diagram (figures reconstructed from DA-Code, arXiv 2410.07331)

This is where skill-level scoring earns its worth. The sentence "this agent scores 82" doesn't help an adoption decision. The sentence "cleaning and joining are stable, but it breaks down repeatedly on unstructured extraction and anomaly detection" tells you where a human needs to stand.

4

Can We Trust the Ruler Itself?

There's another layer to the problem: audit results showing that the very benchmarks measuring these totals are shaky. A study checking the annotation error rates of datasets widely used to evaluate data agents reported error rates of 52.8% on BIRD Mini-Dev and 62.8% on Spider 2.0-Snow. If more than half the ground-truth labels are suspect, how far can the totals scored on top of them be trusted?

An audit in the same vein found that 7 of 10 agentic benchmarks misestimate the relative performance of models by as much as 100%. That means leaderboard rankings can flip. The whole evaluation ecosystem, which has leaned on a single number called the total, has become a subject for re-examination.

Annotation error rate in data benchmarks 52.8% BIRD Mini-Dev 62.8% Spider 2.0-Snow 7 of 10 agentic benchmarks misestimate relative performance by up to 100%

▲ Pebblous original diagram (reconstructed from benchmark annotation-error audit findings)

AgenticDataBench's design of attaching fine-grained ground-truth labels per skill reads as a structural response to this crisis. Splitting the scoring unit into smaller pieces makes it easier to verify and fix which label is wrong locally, and reduces the chance that one error contaminates the entire total. Raising the resolution isn't only for diagnosis; it's also a way to restore trust in the benchmark itself.

5

Outcome, or Ability?

What skill-level scoring is ultimately aiming at comes down to one thing. Data quality is decided not by the size of the model but at the fingertips that actually handle the data. One row misaligned in cleaning, one key wrongly matched in a join, one missed outlier can bring down the trust of the final result. A success rate hides those fingertips wholesale; skill-level labels mark the breaking point on a set of coordinates.

So the question an organization must ask before entrusting data to an agent changes. Not "what's this agent's total score," but "which task will recur in our data—cleaning, joining, or anomaly detection—and how good is this agent at exactly that skill." When the object of verification moves from outcome to ability, the adoption decision is made on a skill map rather than a leaderboard ranking.

AgenticDataBench is one of the first rulers to draw that map. The remaining question rests not with the tool but with us. When we trust an agent, are we looking at the outcome, or the ability?

Editor's Note. The concern Pebblous has returned to again and again points at the same place. Whether data is AI-Ready is settled not by swapping the model but at the fingertips of processing—cleaning, reconciliation, anomaly checks. Skill-level scoring is an attempt to quantify those fingertips, and a thread for moving the language an organization uses to verify an agent from total score to ability.

R

References

Core Papers

Benchmark Reliability Audits