Executive Summary

In a data-centric AI pipeline, data quality profiling acts as a gate. Before data goes into training or serving, you measure things like missing rate, duplicate rate, outlier rate, and functional-dependency violation rate, then block what fails and let the rest through. The catch is speed. Scanning all 5 million rows to compute those metrics takes tens of seconds to a few minutes on a single machine. When data refreshes daily and pipelines run constantly, that cost becomes the wall that blocks real-time monitoring.

The standard alternative is progressive sampling. You draw only a fraction of the whole, a budget, and estimate overall quality from it. A 5% budget means you look at 250,000 of 5 million rows and defer the rest. The question that matters is which sampling strategy strikes the best balance between accuracy and cost.

Here an old assumption enters. The expectation is that smart sampling, which exploits correlations between columns or schema structure to draw from "where problems are likely," should beat random sampling that pulls any row uniformly. Prior work, which found that quality errors cluster along data dependencies, has propped up that expectation. Laure Berti-Équille's benchmark tests the assumption with the first systematic measurement across nine sampling strategies.

The benchmark used eight datasets, from controlled synthetic data to real administrative records to IoT sensor streams. The sharpest scene comes from 500,000 rows of New York City 311 complaints. With a 5% budget, that is 25,000 sampled rows, used to estimate four quality metrics, random sampling posted a mean relative error of 0.49%. Seeing just 5% of the data instead of scanning all of it, it pinned overall quality to under 1% error.

On the same data and the same 5% budget, DAG-guided sampling had an error of 19.5%. The two differ by roughly 40 times. That is, the cleverly chosen sample was 40 times less accurate than the random one. What matters is that this was not a fluke of one dataset. Across real administrative data, smart sampling ran 11x to 49x worse than random, and the gap was statistically significant under a signed-rank test.

The more telling comparison is cluster sampling. Cluster merely splits the data into √N blocks and draws blocks at random; it uses no error proxy and no schema at all. Yet that same cluster method reached nearly the same accuracy as random. Smart sampling armed with a proxy lost, and sampling with no proxy that merely split evenly by position won. That is where the conclusion comes from: the variable that decided accuracy was representativeness, not cleverness.

Why did it lose? The crux is what smart sampling looks at to decide "where problems are likely." DAG and Metropolis methods assign each row an error-proxy score and bias the draw toward high-scoring rows. But that proxy looks only at numeric-column outliers (values outside the interquartile range) and missing rates. It reacts to extreme latitudes and longitudes, or a sudden jump in an amount, yet it is structurally unable to see quality defects in categorical or string columns.

The problem is where defects actually live in real data. In NYC 311, NYPD arrest records, and the UCI census, all three real datasets, quality defects sat mostly in categorical and string columns: a missing agency code, a state value that does not line up, a classification label that is mixed up. IQR-family proxies are congenitally blind to defects like these. So smart sampling herds its sample toward the numeric outliers, where there are in fact no defects, and leaves the categorical side, where the defects really are, empty.

Stratified and importance sampling suffer from the same illness. They only change the allocation or the weighting; because they use the same IQR proxy, they inherit the exact failure mode of DAG and Metropolis. Only cluster, which uses no proxy at all, holds on to random-level accuracy. A controlled experiment nails the cause too. Leave DAG's graph proposal in place and swap the IQR-based weights for uniform weights, and the error drops from 19.5% to 3.8%. The prime suspect for the bias was that proxy-weighting step itself.

Even the device meant to undo the bias later betrays you on this data. Proxy-guided sampling draws its sample, then reweights each row by the inverse of its proxy score to correct the bias after the fact. But in string-heavy data most rows' scores cluster near zero, and the inverse of that can shoot up to a million-fold. Even after you cap and clip it, a handful of over-weighted rows drag the whole estimate right back. Trying to fix a cleverly drawn sample cleverly, the correction device ends up inheriting the same proxy's blind spot.

The categorical blind spot is the first trap. But in numeric-dominated data the failure takes a different, worse shape. On 2.31 million rows of real IoT sensor data from the Intel Berkeley Research Lab, the true outlier rate was 29.6%. Random sampling hit that value to within 0.3 percentage points at a 5% budget. DAG-guided sampling, on the other hand, no matter how far you raised the budget, even at 100% of the data, stayed fixed on an outlier estimate badly off from the truth.

The reason lies in how smart sampling works. DAG concentrates the sample on extreme-value rows. That pushes the quartile boundaries (Q1, Q3) outward within the sample, and the sample's IQR widens. Once the IQR widens, almost no row falls outside that range, and nothing inside the sample gets flagged as an outlier. As a result the estimate of an outlier rate that is truly 29.6% collapses to essentially zero. Drawing more of a sample does not fix it, because the estimator itself is structurally biased.

The two failures point in opposite directions but share a root. One misses because the defects sit where the proxy cannot see (categorical); the other misses because the place the proxy concentrates on (numeric extremes) is exactly what breaks the estimator. Either way, once a biased sample loses representativeness, the quality estimate computed on top of it can no longer be trusted.

Accuracy is not the only problem. As data grows, random sampling's work rises almost linearly, but DAG-guided sampling rises superlinearly because it recomputes the entire candidate pool every batch. At 5 million rows, random took 14.3 seconds and DAG 169, about 12 times slower; at the 7.41 million-row scale the gap reached 28x, and on 5.98 million rows of taxi data it stretched to 47x. Even under injected errors, random's robustness led by roughly 5.6x. You end up piling schema-maintenance cost on top of sampling that is slower, less accurate, and less robust.

The practical conclusion is plain. If you are designing a real-time data quality gauge, standing up random or cluster sampling at a 5–10% budget, before you painstakingly build schema metadata or dependency graphs, is for now the best starting point. The more heterogeneous your sources and the more your schema keeps changing in production, the higher the cost of finely choosing "where to look," and its benefit did not hold up on any real dataset.

This result also connects directly to the view Pebblous has built up in real-time quality monitoring. In drift monitoring, a stable estimation error under 1% lets you trigger retraining only on meaningful shifts in missing or duplicate rate. When the estimate instead swings by several percentage points from run to run, retraining spins its wheels while the data is fine, and compute is wasted. The practical translation of this benchmark is that a quality gauge's reliability comes from representativeness, not from clever sampling.

• 1.Berti-Équille, L. (2026). "Data Quality Profiling at Scale with Progressive Sampling: A Benchmark for Data-Centric AI Pipelines." arXiv:2607.25356.
• 2.Berti-Équille, L. (2026). "progressive-profiling." GitHub / Zenodo (DOI: 10.5281/zenodo.21628028). Reproduction code for the 9 sampling strategies and datasets.
• 3.Intel Berkeley Research Lab. (2004). "Intel Berkeley Research Lab Sensor Data." The benchmark's real IoT dataset (2.3M readings, 29.6% outlier rate).