Executive Summary

Benchmarks that measure how well we pick a model are everywhere. Reasoning, coding, math, safety — the leaderboards are dense. But the seat that measures how well the model's raw material, its training data, was prepared has stood empty for a long time. Data prep is not glamorous. Cleaning source documents, pairing questions with answers, stripping out noise — none of it climbs a leaderboard. And yet this is the very step that moves the outcome most.

DataPrep-Bench, released by Peking University together with the Shanghai institute for advanced algorithms and others, confronts this gap head-on. The paper's premise is simple. Even though training-data quality fundamentally decides an LLM's ability, there was no unified benchmark for how well LLMs and agents actually prepare training data. Why not? The paper points to four reasons.

• • Fragmented methodology: research on "constructing" data and research on "evaluating" data quality grew up in separate literatures, leaving no shared evaluation infrastructure.
• • Inconsistent protocols: source data, base models, and downstream benchmarks differed from study to study, making it impossible to compare which construction strategy actually produces useful data.
• • Weakly validated metrics: existing quality metrics mostly confirmed correlation on a single model and a single domain, or settled for the indirect approach of filtering a top few.
• • Cost barrier: measuring the full pipeline from construction through fine-tuning to downstream evaluation is compute-heavy, so prior work tended to validate only on the domains and models that favored it.

The skill DataPrep-Bench measures comes in two tracks. The first is Data Construction, the ability to turn raw material into supervised training data. The second is Data Quality Evaluation, the ability to predict a candidate dataset's training value before running downstream training at all. The paper treats neither as a sidebar to the other; it measures both as equal goals under a single protocol. It puts the hand that makes the material and the eye that appraises it on the same scale.

What "quality" means here matters. It is not a surface property like whether the prose reads smoothly or whether the spelling is clean. It is training usefulness — how much downstream performance actually rises when a model trains on that data. So the two tracks score differently. The construction track fine-tunes a base model on the produced data and grades it by domain-benchmark scores; the quality-evaluation track grades by the Pearson correlation between an evaluator's score and actual fine-tuning performance.

The construction track's scoring is one layer more concrete. The domain data each method produces is mixed into the general-purpose instruction dataset Dolly-15k, used to fine-tune two base models (Qwen2.5-7B and Llama-3.1-8B), and the resulting model's domain-benchmark score is compared against a model trained on Dolly alone. So each method's score points to how much more useful that method made the data, measured on the common floor that Dolly provides. Figures like "15.1 → 34.2" that appear later all read against this floor.

There are six domains: General Text, Math, Science, Medical, Finance, and Law. The mix is deliberate. Some are dense with tightly compressed knowledge; others demand open-ended reasoning; still others are tangled with specialized terminology and regulation. Only by watching where the same prep method holds and where it collapses can you weigh what "AI prepares data well" actually amounts to.

In the construction track, several methods stepped into the same ring: DataFlow, an expert-designed multi-stage workflow; DataFlow-Skill, where GPT-4o selects and composes its operators; direct requests to a frontier commercial model for QA-pair generation; an agentic approach running in a Claude Code environment in ReAct format; and the paper's newly released Data-Construction-Skill. That last method, instead of spelling out every instruction in a prompt, guides the agent with a reusable "skill layer" that packs task instructions, output schema, quality constraints, and validation rules.

The most striking gain came in finance. Fine-tuning Llama-3.1-8B on Dolly-15k alone puts the finance-domain score at 15.1; adding the data produced by Data-Construction-Skill jumped it to 34.2 — an absolute lift of nearly 20 points. It is a vivid demonstration of how strong structured extraction can be in a knowledge-dense domain. That said, in the same finance domain DataFlow-Skill edged slightly higher at 36.5, making it clear that the skill layer is no master key that dominates every method.

The real lesson lies in the failures. The finding the paper stresses most is that synthetic domain data often does harm rather than help. Several generation methods scored below the Dolly-alone baseline across multiple domains, and this backfire was especially pronounced in Llama-3.1-8B's Science domain. In Qwen2.5-7B's finance, Dolly alone scored 57.8, yet adding Data-Construction-Skill actually pulled it down to 55.5. You add more data, and performance is shaved off.

This is where the second track becomes especially interesting for Pebblous readers. If you can only tell whether data is good or bad after building it and running a fine-tune, the compute is already spent. The quality-evaluation track asks the opposite question: before training, can you predict how useful this dataset will be from a single scalar score? The paper lined up 18 evaluators against the same candidate pool — 9 quality-based, 6 diversity-based, and 1 newly released distribution-based — and compared them.

The one that worked best was the paper's newly proposed Distributional Alignment Score (DAS). The idea comes from domain-adaptation theory: the closer a candidate's distribution sits to the target domain's distribution, the more likely it is to help in training. The computation goes like this. Encode the candidate dataset and a per-domain high-quality corpus (a proxy) with a fixed text encoder, then measure the distance between the two distributions (Maximum Mean Discrepancy, MMD). The closer they are, the higher the score. It looks at the alignment of distributions, not surface text.

The result was clear. DAS held the strongest correlation across four of the six domains even as the model was swapped, and above all it was the only metric to clear a correlation of 0.70 simultaneously in Math, Science, and Medical. In Math it rose from 0.72 to 0.93 depending on the model. The QuRating family that looked at surface quality and the Vendi family that looked at diversity did not clear this bar with it. It means that, before running a fine-tune, you can gauge fairly reliably whether the data will help just by looking at its distribution.

An honest benchmark does not report only its successes. If DAS shone in Math, Science, and Medical, in finance and law every metric, DAS included, struggled together. In finance, DAS's correlation was weakest at around 0.42 to 0.50, and other metrics were unstable or even turned negative. This is not the weakness of one particular metric; the paper plainly acknowledges that the domain itself is hard.

The reasons run several layers deep. First, the candidate pool was asymmetric across domains. General Text had 14 candidates while finance and law had only 8 to 10 — a statistical condition under which the Pearson correlation is hard to pin down stably. Add to that the domain's character, where specialized terminology and regulatory context are tightly interwoven. In a field where a subtle shift in wording can swing the meaning sharply, gauging training value from distributional distance alone becomes that much trickier.

The paper flags one more limitation. When a per-sample quality metric is scaled up to the dataset level, it tends to degrade into the simple heuristic of averaging or picking a top few. A gap also remains between the answer format humans want and the format models are good at producing. Turning data readiness into a score has begun, but in half the domains that score is not yet trustworthy.

In the AI-Ready Data that Pebblous talks about, the hardest word is "ready." The judgment that data is prepared has long leaned on the instinct of a seasoned practitioner. The call that this dataset is fit to train on was usually an experienced engineer's intuition, not a measured number. DataPrep-Bench, and DAS within it, is the first attempt to put that intuition up on the scoreboard.

This piece reads alongside two earlier ones. The article arguing that the cost savings of a custom SLM ultimately come from the quality of data curation (The Data Quality That Decides Custom SLM Cost) claimed that "quality decides cost." DAS is the first case of turning that claim into a score you can predict before training. And if the article that first named data readiness as a ceiling (AI May Know the Answer but Not When to Trust It) spoke of that readiness existing, this benchmark measures the party that produces it — the AI's own skill.

The practical question resolves into two layers. Does the data we mean to train on actually resemble the distribution of the domain we want to predict? And can we confirm the degree of that resemblance as a number before burning compute? In a domain where DAS works, the answer moves closer to "yes"; in finance or law, the honest answer remains "human eyes are still needed." Either way, one thing does not change: the basis of judgment has begun to move from feel to score.

• 1.Liang, H., Cai, Q., Lin, Y., Du, J., Xia, Q., Qiu, S., et al. (2026). "DataPrep-Bench: Benchmarking LLMs as Training Data Preparators." arXiv:2607.20465.
• 2.DataPrep-Bench Project Page. "DataPrep-Bench." Datasets, leaderboard, and evaluator list.