Executive Summary
Say you run into one strange value. There is no clean original to confirm whether it is an error or a rare but genuine observation. Reference-free automated cleaning means acting on the data in that state, and most working data sits exactly there. One researcher injected controlled corruption into financial, clinical, and environmental datasets and ran seven agent configurations 126 times. The best error finder turned out to be deterministic profiling that never once called a language model, with the most heavily equipped configuration behind it. The real finding, though, was not the leaderboard but what sat underneath it. Practically none of the seven configurations managed to replace a wrong value with the right one.
The best finder was not the safest one. The deterministic profiler that led on detection also led on the rate at which it touched healthy values, because it treated any statistical outlier as an error on the spot. The most sophisticated configuration, by contrast, logged zero safety violations — not out of caution, but because it never issued a single repair decision. The paper records this as conservative non-repair rather than an achievement. Finding well and refraining from careless fixes were separate axes, and no configuration won both at once.
That changes the question an organization should ask before automating cleaning. Not how well does it find, but does a record survive of what was changed and why. In this study every reversible script that ran succeeded and provenance logs were written in structured form, yet the overlap between a decision and the evidence it cited was almost nil. Of the three boxes a clinical audit demands, original-value preservation and attribution of the change were filled and only the reason was left blank. Retrieving evidence, ranking it, and logging it is one job; anchoring a decision in that evidence is another.
0.561 / 0.421
Detection F1: deterministic rules vs. the best LLM configuration
The configuration that never called a model finished first
0.028
The only non-zero expected-repair match rate among seven configurations
The other six were all 0.000
0.041
Lexical overlap between a decision and the evidence sentence it cited
The highest value among configurations running citation-alignment checks
72 / 72
Reversible scripts that executed successfully
The ability to undo landed before the ability to fix
No one can tell you the value is wrong
Data cleaning research has long been conducted with the answer key on the table. You take a clean original, inject errors into it, and score whether the system restores the original. Practice does not work that way. When a radiation sensor reports a reading ten times its usual level, no file anywhere tells you whether the instrument failed or an event actually occurred. The same holds for an extreme number in a clinical record that may be a typing error or a rare outcome. Between the fact that a value looks strange and the judgment that the value is wrong lies a gap nobody crosses on your behalf.
An experimental study posted to arXiv on 14 August 2026 by Hadi Fadlallah of the Lebanese University of Science and Arts turns that gap into the experimental condition. Under it, the paper redefines cleaning as making evidence-based decisions rather than changing values. What the system must produce is not repaired data but decisions plus the record that supports them. That redefinition drives the design of every metric that follows.
Limits worth knowing before you read on
Before carrying any conclusion out of this study, it is more honest to state what it is not. Every number below holds inside these conditions, and most of them become false claims if the conditions travel separately.
- It is a single-author preprint that has not been peer reviewed. v1 dated 14 August 2026, CC BY 4.0. Code and artifacts are currently available only through an anonymized link.
- Exactly one language model was used. Every configuration ran
openai/gpt-oss-20bthrough OpenRouter — a 20-billion-parameter open-weight model — at temperature 0.2, with a 3,000-token output cap and a sample of at most 10 records. No frontier model appears anywhere in the study. Nothing here should be read as "LLMs in general lost to rules." - The evidence base is 17 fixed local documents. The evidence retrieval described later is not a web search; it searches 17 documents placed in the project in advance.
- There are no statistical tests. No p-values, no confidence intervals, no significance testing. The author states that three repetitions are too few to infer variability.
- Injected errors may be more regular than naturally occurring ones. The paper's internal-validity section concedes this directly.
What still makes the study worth reading is its metric design. Instead of scoring a cleaning system on a single accuracy figure, it measures detection, repair, evidence, restraint, reproducibility, and cost separately — and the picture that emerges is that no configuration wins all six axes at once. The table below is the skeleton of the experiment.
| Item | Value |
|---|---|
| Runs | 3 datasets × 2 modes × 7 configurations × 3 repetitions = 126 runs. Zero terminal failures |
| Datasets | German Credit (finance, UCI) · EHR (clinical, Kaggle) · Radiation (environmental monitoring) |
| Two modes | Controlled synthetic corruption (ground truth available, so scoring is possible) / raw data (no ground truth, descriptive reporting only) |
| Injection scale | 30 · 33 · 30 errors per dataset per repetition, plus 5 valid signals each. A fresh corrupted file is generated for every repetition |
| Model settings | openai/gpt-oss-20b, temperature 0.2, 3,000-token output cap, sample of at most 10 records |
Source: Fadlallah (2026), arXiv:2608.14765, Tables 7 and 8
The choice is not just fix or don't fix
The paper defines six classes of decision a system can reach when there is no ground truth, and requires it to choose one for every issue it finds. This list is the single most portable asset in this report. What your in-house cleaning pipeline should output, and what you should demand of a vendor, is all contained in it.
| Decision class | When to use it | The paper's example |
|---|---|---|
| Safe repair | The error is unambiguous and the correction is directly supported by a rule, documentation, or an executable check | Standardizing timestamp formats; converting a documented placeholder code to null |
| Conditional repair | The correction is plausible but rests on an explicit assumption | A unit conversion where the source unit is strongly inferred but not documented |
| Flag only | The value is suspicious but there is not enough evidence to change it | Sudden spikes, unusual transactions, unexpected sensor readings |
| Preserve | The value is unusual but may be a meaningful rare event | Rare clinical outcomes, extreme environmental readings, genuine financial shocks |
| Human review | The case is high-risk, ambiguous, or unresolved | Conflicting evidence, or uncertain domain interpretation |
| Reject repair | A proposed fix is insufficiently supported or violates policy | Trying to replace an outlier merely because it is statistically extreme |
Source: same paper, Table 6. Hold on to that last row and the twist in Section 3 reads far more sharply
Seven configurations, one capability at a time
The comparison runs across seven configurations given the same task. Two are control baselines; the other five form a series in which each adds one capability to the one before it. To read which addition improved what, you need this lineage in hand first.
| Configuration | Capabilities |
|---|---|
| A0 | Language-model baseline given only the schema and a sample |
| A1 | Deterministic profiling and checks baseline. No model calls at all |
| A2 | Adds a profiling summary to the language model |
| A3 | Adds executable Python checks and generation of reversible scripts |
| A4 | Adds controlled local evidence retrieval |
| A5 | Adds source ranking and citation-alignment checking |
| A6 | Adds a conservative repair policy. Everything switched on |
A0 and A1 are control baselines, not consecutive steps in a single-factor ablation. Capability-level comparisons hold only across A2→A3 (tools), A3→A4 (evidence retrieval), A4→A5 (ranking and alignment), and A5→A6 (conservative policy)
The capability lineage across the seven configurations looks like this. A0 and A1 sit side by side as control baselines; from A2 on, each configuration stacks one capability on top of the one before it.
Reconstructed from the paper's Table 4 configuration definitions, ordered by capability accumulation. A6 adds a conservative repair policy on top of everything before it
This lineage is the scale against which every later table is read, because which axis improves, which stays flat, and which moves backward when a capability is added can only be seen in the differences between configurations. And the axes that improved and the axes that got worse did not move together along the series. They split within a single configuration too.
On detection, the rules came out ahead
The first axis is detection: how many injected errors were found, and how accurate those calls were. In synthetic-corruption mode the highest detection F1 belonged to A1 — 0.561, with precision 0.687 and recall 0.484. A1 never calls a language model. It is classical profiling: compute column statistics, then catch missing values, range violations, and format breaches with rules. A6, with every capability switched on, came second at 0.421.
Reading that ranking as a defeat for language models misreads the study. The more important number sits at the other end. A0, given only the schema and 10 sample records, scored an F1 of 0.013 — precision 0.074 against recall 0.007. That combination does not mean it gave many wrong answers; it means it barely spoke at all. A0 raised an average of 3.222 issues per run against A1's 22.556. Eyeballing a table turned out to identify almost nothing as anomalous.
Hand the same model a profiling summary and F1 climbed from 0.013 to 0.307 — a factor of 23. Add executable Python checks on top and it reached 0.402. Model reasoning worked only when it sat on deterministic signal, and that is the most reproducible observation in the study. Past that point, however, adding capabilities did not improve detection monotonically. A4, with evidence retrieval added, held roughly flat at 0.407, and A5, with ranking added, actually slipped to 0.376.
The figure below places the detection F1 of all seven configurations side by side. The orange bar is A1, which uses no language model; the gray bars are the model-based configurations.
A0 and A1 are control baselines; A2 through A6 form the cumulative-capability series. Data: paper Table 13
Look at precision alone and the picture shifts again. The most accurate caller was A4 at 0.928, followed by A6 at 0.885 and A3 at 0.884. The model-based configurations said little but were mostly right when they spoke; A1 said a great deal and missed on roughly three calls in ten. Recall runs the other way, and nothing came close to A1's 0.484. Even on detection alone, then, the practical lesson is to ask which side of the precision–recall trade-off a tool has been tuned for.
Datasets differed too. Average F1 was highest on the clinical records at 0.405, followed by the financial data at 0.380, with the environmental radiation data lowest at 0.280. The paper does not explain why the radiation data was hard. It notes that the data mixes temporal patterns with rare events and that the 10-record sample cap may have been too small for long time series, but records both as untested hypotheses.
This result only half-overlaps the thesis in our earlier report It's the Weak Harness, Not the Weak Model, That Kills Your Agent. Detection rising 23-fold once profiling — a harness — was attached points the same way as that thesis. But improvement stalling, and once reversing, as more harness went on marks the thesis's boundary. Saying the harness matters more than the model is not saying that more harness is always better.
Finding and fixing were different jobs
The detection leaderboard is this study's surface. What lies beneath it is the finding that prompted this report. Practically none of the seven configurations restored a wrong value to the correct one. Only one, A3 at 0.028, posted a non-zero expected-repair match rate; the other six were all 0.000. They knew where to fix. They did not know what to fix it to.
Hitting the target versus hitting the value
Understanding that result means separating two metrics that look alike. Here is what the paper defines in Table 10.
- Safe repair rate is the share of the system's repair decisions that aimed at a spot where repair was legitimate. Repairs applied to valid signals or non-repairable cases, and repairs with no corresponding ground-truth issue, are excluded from the count.
- Expected-repair match rate is the share of those well-aimed repairs whose proposed value actually matched the ground-truth correction.
A1's two figures show the difference starkly: a safe repair rate of 0.770 against an expected-repair match rate of 0.000. Roughly eight in ten of the spots it chose to touch were the right spots, and not one of the values it wrote there was right. The paper nails this down explicitly, noting that a high safe repair rate does not mean the proposed corrections were accurate and that this configuration's expected-repair match rate was 0.000. That is why, when a vendor says "our tool only makes safe repairs," you have to ask which of the two metrics that sentence refers to.
The best finder touched the most healthy values
The reversal shows up in the safety metrics. A1, first on detection, recorded an unsafe repair rate of 0.064 and an unnecessary repair rate of 0.167. Only two of the seven configurations — A0 and A1 — actually touched healthy values; from A3 through A6 both metrics are 0.000. That runs exactly counter to the received wisdom that rule-based systems are safe and language models are risky.
The reason lies in what A1 counts as an error. Table 11 of the paper lists among A1's expected limitations that it may treat statistical anomalies as errors without semantics, context, or conservative reasoning. If you kept the last row of the decision-class table from Section 1, that is precisely the behavior it forbids: replacing a value merely because it is statistically extreme. A rule does not ask why the value is extreme.
The same A1 scored 1.000 on valid-signal preservation. It looks contradictory, but the two are sides of one property. With no model in the loop, A1 defaults to flagging rather than acting, so it erased none of the injected genuine signals — and that same property, seen from the other side, is what makes it flag outliers as errors. One more footnote is mandatory when reading the table below. The 0.000 valid-signal preservation of A3 through A6 does not mean they damaged signals. The metric is defined only over detected valid-signal cases, so it also reads 0 when no valid signal was detected at all.
The table below puts all three axes so far in one place, by configuration. Read across and you get the character of a single configuration; read down and you see what disappeared as capabilities accumulated. Three cells deserve attention: the row with the highest detection F1 is also the only row with a non-zero unsafe repair rate; the expected-repair match column is zero everywhere but one cell; and as you move down, the safety metrics fill with zeros not through better judgment but because the number of repair decisions itself shrank.
| Config | Detection F1 | Expected-repair match | Safe repair | Unsafe repair | Unnecessary repair |
|---|---|---|---|---|---|
| A0 | 0.013 | 0.000 | 0.000 | 0.000 | 0.222 |
| A1 | 0.561 | 0.000 | 0.770 | 0.064 | 0.167 |
| A2 | 0.307 | 0.000 | 0.000 | 0.000 | 0.111 |
| A3 | 0.402 | 0.028 | 0.444 | 0.000 | 0.000 |
| A4 | 0.407 | 0.000 | 0.556 | 0.000 | 0.000 |
| A5 | 0.376 | 0.000 | 0.222 | 0.000 | 0.000 |
| A6 | 0.421 | 0.000 | 0.000 | 0.000 | 0.000 |
Source: paper Tables 13 and 14 (synthetic-corruption mode). Valid-signal preservation is left out of this table: from A3 onward its 0.000 comes from non-detection rather than preservation failure, and placing it alongside these columns invites misreading
A6's zero violations are not an achievement
That A6, with everything switched on, logged zero safety violations reads at first like a win for the conservative policy. Read the table down and it isn't. Unsafe and unnecessary repairs were already 0.000 from A3 onward, so this is not something A6 created. And in synthetic-corruption mode A6 issued no repair-class decisions at all. In the paper's own framing it is better read as conservative non-repair. The author writes in the discussion that the hypothesis that a conservative policy improves safety outcomes is not clearly supported by the A5-versus-A6 comparison.
Where the restraint happened is on record too. A6 raised 12.778 issues and issued only 3.444 decisions. A5 raised 12.667 and narrowed to 4.667. The conservative policy worked by finding and then declining to act, not by finding less. How those 3.444 decisions split among flag, preserve, human review, and reject-repair is not reported for synthetic mode. A partial distribution is visible only in raw-data mode, where A6's conservative-flag rate is 0.556 and its human-review rate is 0.111.
Would a frontier model have changed this?
A reader who has come this far is entitled to push back: the study used a 20-billion-parameter open-weight model, so of course it lost. There is an independent experiment that ran the same family of task on frontier models. Bendinelli, Dox, and Holz, in a 2025 ICLR workshop paper, gave gpt-4o, o3-mini, claude-3-5-sonnet, and gemini-2.0-flash function calling and a stateful IPython session, had them repair injected errors in three Kaggle datasets, and scored the result by performance on a clean held-out test set.
The direction of the result is the same. No model reached the improvement attainable under perfect correction, and on the hardest dataset every model came in under 1% improvement. A human data scientist working for an hour on that same dataset also got 0%. The quality of the failures is more telling. claude-3-5-sonnet correctly detected the implausible fish-consumption figures for landlocked Afghanistan and Nepal and then applied an arbitrary scaling factor instead of a statistical method such as quantiles. For missing country codes it invented a new category called Other rather than inferring from earlier years. It is the same zero expected-repair match rate in a different lab. Distribution shifts, trends, and biases spanning multiple rows went undetected by every model.
There is a line to hold when placing the two studies side by side: they score on different axes. One measures the improvement in downstream model performance; the other measures F1 and match rate against injected errors. Percentages and F1 scores cannot be compared in the same breath. What overlaps is the direction of the conclusion, not the numbers. That direction reads as follows. The failure of picking an arbitrary replacement after spotting an anomaly did not disappear as models got larger. There is, as yet, no evidence that scaling the model solves it.
Citations attached, decisions unanchored
Without ground truth, the only way to justify a decision is to cite something. From A4 on, the system searched a local evidence corpus and attached documents to decisions; from A5 on, it ranked sources and checked whether citations aligned with decisions. Attaching evidence worked well on its own terms. The rate of rules generated with no supporting evidence was lowest at 0.037 for both A5 and A6, and in raw-data mode A4 was lowest at 0.028. The model-only configuration A0 scored 0.667 on the same metric. Adding profiling alone dropped it to 0.167, so the tendency to invent unsupported rules fell sharply from deterministic signal alone.
The problem sits one layer up. Measured per decision, citation alignment is 0.036 for A5 and 0.041 for A6. The remaining configurations score 0 because they run no alignment check at all. How to read that number needs stating precisely. Citation alignment is the maximum Jaccard similarity between a decision statement and the evidence sentences it cites, computed per decision and averaged. It does not mean 4 decisions in 100 carried evidence; it means the lexical overlap between decision text and evidence text averaged around 4%. Read as a proportion, it misleads.
Set evidence coverage beside it and the shape of the gap gets clearer. That metric is the fraction of decisions referencing at least one evidence record, and evidence here includes profiling results and tool output, not only retrieved documents. In synthetic-corruption mode the highest value was A1's 0.870 — a configuration with no model and no retrieval that logged its own statistical profile as the evidence record. Behind it come A4 at 0.778 and A2 at 0.722, while A5 and A6, which add ranking and alignment checking on top of retrieval, fall to 0.630 and 0.519. Yet those two configurations, as we just saw, also had the lowest rate of unsupported rules. Invented rules went down while the share of decisions carrying an evidence record went down with them.
The paper assesses its own system this way: in its current form it is more accurately described as an evidence-controlled, provenance-aware system than as a mature evidence-grounded autonomous repair system. Having retrieved evidence and having anchored a decision in evidence are two different claims.
What "ranked" actually resolved
One implementation detail captures the character of this gap. Evidence ranking is a weighted sum over five axes: authority 0.30, relevance 0.25, specificity 0.20, consistency 0.15, and temporal validity 0.10. But the local corpus carries no consistency annotation, so every piece of evidence received a fixed 0.7. An axis holding 15% of the weight ran without distinguishing anything at all. It is hard to find a cleaner illustration of the distance between a feature described as ranking and what that ranking actually separated.
Fairness requires the other side too. The fundamentals of auditability did hold. All 72 reversible scripts that executed succeeded. Provenance logs recorded prompt context, sources, tool output, applied rules, affected records, and original values in structured form. All 126 final outputs passed schema validation. Of "what was changed and why," only the why was missing; what was touched, when, and how survived.
The standards stack has no box for it
To check whether this hole belongs to this experiment alone, look at the industry standards. The figure below shows what the two standards for recording data lineage do and do not hold.
Compiled from the OpenLineage column-lineage facet spec and W3C PROV-DM
OpenLineage's column-lineage facet is fairly sophisticated. It attaches a transformation to each input field, distinguishes direct derivation from indirect influence, and even records whether a value was masked. But the question it answers is which column fed which column. Why a single value became that value is outside the spec. W3C PROV leaves a machine-queryable record of what was produced by what through its entity–activity–agent triple, yet it is deliberately domain-neutral and does not describe value-level causality in its core vocabulary. Hence the steady stream of domain extensions, and hence tuple-level provenance remaining a research topic.
Citation alignment of 0.041 can be read as the first quantified observation of that empty box. The same shape of gap our earlier report The Proof Was Right. How It Got There Wasn't Recorded observed in mathematical proof recurs in data cleaning: the output can be verified while the reasoning that produced it cannot.
Paying more did not buy accuracy
The last thing anyone evaluating adoption asks is the price. This study logged runtime and token counts for every run, and the two moved differently. The table below averages nine runs per configuration in raw-data mode.
| Config | Runtime (sec) | Total tokens |
|---|---|---|
| A1 | 0.081 | 0 |
| A0 | 25.209 | 4,841 |
| A2 | 52.616 | 9,346 |
| A3 | 61.958 | 9,392 |
| A4 | 43.521 | 9,918 |
| A5 | 34.356 | 10,133 |
| A6 | 138.938 | 14,803 |
Source: paper Table 16 (raw-data mode). Averaged across all runs, A6 comes to 77.466 seconds and 12,916 tokens
Token counts rose monotonically from A2 to A6. That is expected: switching on capabilities lengthens both prompts and tool output. Runtime did not. It sits at 61.958 seconds for A3, falls to 43.521 for A4 and 34.356 for A5, then jumps to 138.938 for A6. The paper's reading is that cost depends on reasoning behavior, retrieval, and tool execution rather than on the number of capabilities enabled. The practical implication is that budgeting by feature count gets it wrong.
The widest gap is between the two baselines. In the same raw-data mode A6 took roughly 1,715 times as long as A1 and still scored lower on detection F1. Tokens admit no ratio at all, since A1's count is 0 and the comparison does not hold. The figure below puts the relationship on one plane.
Runtime is the raw-data mode average (Table 16); F1 requires ground truth and therefore comes from synthetic-corruption mode (Table 13). Read with the caveat that two different modes share one plane
The trap in converting cost to money
The temptation to price the tokens is strong, but these numbers are weak for that purpose. At the model's published rates, a single A6 run comes to well under a cent. The paper does not report the input–output split, so only a range can be given: with output capped at 3,000 tokens, at least 79% of the 14,803 tokens are input. The real cost therefore sits toward the bottom of that range.
Multiplying that figure by a million records to produce an annual cost is not advisable. The system looks at a sample of at most 10 records plus a profiling summary rather than the whole dataset, so the assumption that tokens scale with record count does not hold in the first place. What to take from this study is a sense of direction, not an absolute price. There is a two-order-of-magnitude difference in unit price between a small open-weight model and a frontier API, and within a single task there is a three-order-of-magnitude difference in runtime across configurations. And that spread was never repaid in accuracy.
Does the same input produce the same answer?
Reproducibility was measured as the rate of exact decision-signature matches across repeated runs. The highest was A1 at 0.500, with A5 and A6 at 0.444. That A1, a deterministic pipeline, is not at 1.000 catches the eye. The paper does not explain the cause. The only clue it offers is the fragility of the metric: a match requires the issue identifier and decision class to be exactly identical, so semantically equivalent outputs can be counted as different.
Two other facts in the paper allow this reading. A fresh corrupted file is generated for every dataset and every repetition, so in synthetic mode even a deterministic pipeline sees different input each time and cannot produce matching signatures. Raw-data mode, by contrast, holds input identical across repetitions and should yield exact matches. Average the two modes and you get 0.500. This is this report's reading, however, not the paper's explanation. The paper does not answer the question.
The execution audit record is worth a look alongside it. There were 4 retries, 30 JSON repairs, 31 outputs whose content changed in post-processing, and 25 schema-invalid outputs before normalization — 0 after, leaving all 126 final outputs valid. It is also hard to write off wobbly structured output as a pathology of small models. In the frontier-model study cited earlier, invalid submission rates were 13.53% for claude-3-5-sonnet, 13.10% for gemini-2.0-flash, 10.87% for o3-mini, and 7.94% for gpt-4o. The two metrics are defined differently and cannot be compared on the same scale, but the direction — that unstable structured output is not solely a question of model size — comes through.
Cost and reproducibility look like the items tacked on at the end of an evaluation, yet they only become decision-grade when placed in the same table as the other four axes. The cheapest and fastest configuration in this study led on detection and was simultaneously one of the two configurations that actually touched healthy values. The most expensive one was safe by way of touching nothing and scored lower on detection. Isolate any single axis and both look good.
No single score can pick the tool for you
Everything above reduces to one line: no configuration won all six axes at once. So the questions to ask when choosing a tool split six ways too. The table below converts what this study measured into questions to ask in procurement and internal adoption. It is written so it can be lifted straight into a vendor evaluation document.
| Axis | What the study measured | What to ask before adopting | The warning the study leaves |
|---|---|---|---|
| Detection | Precision, recall, F1 | What does it treat as anomalous? Is it tuned for recall or for precision? | Deterministic rules beat the whole 20B model series |
| Repair | Expected-repair match rate | Does "it fixed it" mean it wrote the correct value, or that it identified the right spot? | One non-zero value across seven configurations: 0.028 |
| Evidence | Citation alignment, unsupported-rule rate | Can you point to the sentence in the document that supported this decision? | The logs survive, but decision–evidence overlap runs around 4% |
| Restraint | Unsafe repairs, unnecessary repairs, preservation rate | What did it leave alone? What share went to a human? | The best finder also touched the most healthy values |
| Reproducibility | Exact decision-signature match | Does the same input give the same answer? If not, what is moving? | The best was 0.500, and the cause was never established |
| Cost | Runtime, tokens | What does a million rows cost, and how many hours does it take? | A 1,715× spread under identical conditions, and the expensive side was not more accurate |
The winner changes from axis to axis, as the figure below shows. A1 led on three axes, but the other three went to different configurations.
A1 (orange) topped three axes, but a different configuration led on repair, evidence, and restraint. Data: paper Tables 13, 14, 16
Most tools on the market stop at detection
Hold these six axes up to the market and the positions sort themselves out. The widely used open-source data quality tools are all detection tools. They draw the line at design time: neither repair nor rollback is their job. This is where A1 in the study maps onto practice.
| Tool | Detection | Repair | Audit and rollback |
|---|---|---|---|
| Great Expectations | Schema, range, distribution, and custom checks | None | Auto-generated Data Docs reports and test history |
| Soda Core | Declarative YAML checks | None | Trend visualization of results |
| dbt tests | Warehouse-native tests | None | Failing rows surface in the output, and little more |
| AWS Deequ | Spark-based constraints and statistical metrics | None | Metrics repository |
| Monte Carlo | Machine-learning anomaly detection | Detection-centric | Traces upstream causes through lineage |
| Bigeye | Anomaly detection | Suggests resolutions; execution stays with people | Metric- and owner-based alerting |
| Informatica CLAIRE | Metadata-driven rule generation | Claims automatic issue resolution | Lineage and governance. Rollback absent from the announcement |
Compiled from 2025–2026 comparative reviews and vendor announcements. "None" under repair is not a defect; these tools are detection-only by design
Scan the 2026 agentic data quality product announcements and one box stays empty. No announcement documents the ability to roll back what an agent automatically repaired. Platform-level table time travel rewinds a snapshot; it is not the ability to say this value was changed on this evidence, so undo this decision alone. Absence from an announcement does not mean absence of the feature — it means nothing was confirmed within what has been published. Still, the contrast stands: a research prototype landed 72 out of 72 rollbacks, and commercial announcements have yet to mention theirs.
Overlay the audit requirements and one box stays blank
Among these six axes, some are already demanded by regulation. Clinical data is the clearest case. 21 CFR 11.10(e) requires audit trails to be secure, computer-generated, time-stamped, and independent of the data they cover, and states flatly that changes must not obscure previously recorded information. Reason-for-change does not appear verbatim in Part 11, but EU GMP Annex 11 requires it explicitly and ALCOA+ expectations make it effectively mandatory. Regulators have stated plainly that automated corrections must also be attributable to an identifiable actor, carry a recorded reason, and never be applied silently.
Map those requirements onto the study's outputs box by box and the empty one appears immediately. The requirement not to obscure the original value maps to the reversible script; who changed it and when maps to the provenance log. What remains is the reason, and the metric that corresponds to that box is the citation alignment from the previous section.
| What a clinical audit requires | The corresponding output in the study | Did it hold? |
|---|---|---|
| Changes must not obscure the original value | Reversible script | Held. 72 of 72 executions succeeded |
| Attribution of who changed it and when | Provenance log | Held. Recorded in structured form |
| Reason for the change | Per-decision citation alignment | Did not hold. 0.041 |
21 CFR 11.10(e) and ALCOA+ requirements mapped onto the paper's outputs. Even with evidence attached to the reason box, an audit does not count it as filled if that evidence does not support the decision
Financial requirements point the same way. The European Central Bank's May 2024 guide on risk data aggregation and risk reporting requires complete and up-to-date lineage at the data-attribute level from collection through final reporting, plus a separate register of quality issues and limitations. Practice lags well behind. In Deloitte's 2024 benchmark survey, 72% of banks had defined a data quality risk appetite while only 17% had operationalized it. And even attribute-level lineage stops at where this value came from. Why this value changed is still not a standard field.
Article 10 of the EU AI Act requires that training, validation, and testing data for high-risk systems be as free of errors as possible in view of the intended purpose and, at the same time, be subject to governance covering collection processes and origin. The demand to fix it and the demand to record what was changed and how sit in the same article. The timeline recently slipped. The AI digital omnibus regulation, published in the Official Journal on 24 July 2026 and in force from 27 July, pushed the application date for Annex III high-risk obligations to 2 December 2027 and Annex I embedded systems to 2 August 2028. Plenty of material still says the rules apply from August 2026, so cite carefully. A later deadline does not mean the requirement went away; it means the preparation window got longer.
Of the three boxes regulation demands, the automated cleaning pipeline filled two, and the most important one — why — stayed blank. There is no standard slot for that box in the lineage stack either. For an organization about to adopt cleaning automation, the combination says something plain. Today's safeguard comes not from how well a system fixes things but from whether it can leave a record of what it changed and why.
Why This Matters to Pebblous
DataLens, the diagnostic engine inside Pebblous's DataClinic, moved from a purely neural design to a neurosymbolic one. Deterministic components catch what rules and statistics can catch, and only judgments requiring semantics and context go to the model. This study offers an independent outside observation of that choice. Strip out the deterministic components and the model-only configuration managed a detection F1 of 0.013 — effectively saying nothing — while adding a profiling summary lifted it to 0.307. Neural reasoning worked only on top of deterministic signal. That said, the paper never tested DataClinic. It corroborates a design decision pointing the same way; it does not validate our product.
From a data quality standpoint the more important result is that detection and repair are separate jobs. It substantiates why an AI-Ready Data pipeline keeps the diagnostic stage and the improvement stage apart. An expected-repair match rate near zero across every configuration means the space for automatic repair is still narrow, and the fact that reversible scripts and provenance logs held first means auditability should be automation's first deliverable. On the path from training data into a model's internal representations, if no record survives of what changed and on what grounds, no quality claim downstream of it can be verified.
Translated into customer and partner practice, it becomes a list of questions. What a manufacturing, healthcare, or financial customer should ask when evaluating cleaning automation is not what the accuracy percentage is but: can it be undone, on what evidence was it changed, what did it leave alone, and what share went to a human. In domains where a wrong repair costs more than a missed detection, the restraint of flagging, preserving, and escalating instead of repairing may well be the right answer. The six-axis table in Section 6 is a document for starting that conversation.
Finally, the study leaves a design premise on Pebblous's roadmap. Building retrieval, ranking, and logging is one job; anchoring decisions in evidence is another. The first three already worked in this prototype and the fourth did not. And no lineage standard has a place to hold that fourth one. That is why Data Greenhouse's autonomous cleaning roadmap has to take the gap as a design premise — a description of a box nobody has filled yet, rather than a claim of competitive advantage.
Editor's Note. Pebblous works on data quality diagnostics and lineage design, so we have an interest in this subject. This report is not written to recommend a product; it is an attempt to put on record that an outside controlled study independently showed what goes unseen when cleaning automation is scored on detection rate alone. The paper cited is a single-author preprint that has not been peer reviewed, and it used exactly one language model. The honest use of this article is to take the structure of the axes rather than the absolute numbers.
References
Primary sources
- 1.Fadlallah, H. (2026). Agentic Data Cleaning Without a Clean Reference: An Experimental Study of Capabilities and Trade-offs. arXiv:2608.14765v1 [cs.AI], 14 Aug 2026, CC BY 4.0. (Most figures in this article come from Tables 6, 7, 8, 10, and 12–16 and §9–§11 of this paper. Single-author preprint, not peer reviewed)
- 2.Bendinelli, T., Dox, A., Holz, C. (2025). Exploring LLM Agents for Cleaning Tabular Machine Learning Datasets. arXiv:2503.06664. ICLR 2025 Workshop on Foundation Models in the Wild. (Four frontier models compared. Different scoring axis, so the numbers are not directly comparable)
Academic lineage
- 3.Wang, R. Y., Strong, D. M. (1996). Beyond accuracy: what data quality means to data consumers. Journal of Management Information Systems 12(4), 5–33. (The multidimensionality of quality; the root of the six-axis decomposition)
- 4.Abedjan, Z. et al. (2016). Detecting data errors: where are we and what needs to be done? PVLDB 9(12), 993–1004. (The starting point that no single technique catches every error)
- 5.Rekatsinas, T., Chu, X., Ilyas, I. F., Ré, C. (2017). HoloClean: holistic data repairs with probabilistic inference. PVLDB 10(11), 1190–1201. (The probabilistic cleaning line of work)
- 6.Naeem, Z. A. et al. (2024). RetClean: retrieval-based data cleaning using LLMs and data lakes. PVLDB 17(12), 4421–4424. (The lineage of the evidence-retrieval configurations)
- 7.Rashkin, H. et al. (2023). Measuring attribution in natural language generation models. Computational Linguistics 49(4), 777–840. (The root of the citation-alignment metric)
- 8.Li, L., Fang, L., Ludäscher, B., Torvik, V. I. (2025). AutoDCWorkflow: LLM-based data cleaning workflow auto-generation and benchmark. Findings of EMNLP 2025, 7766–7780.
Standards and regulation
- 9.W3C (2013). PROV-DM: The PROV Data Model. W3C Recommendation. (The entity–activity–agent triple; value-level causality sits outside the core vocabulary)
- 10.OpenLineage. Column Level Lineage Dataset Facet. (DIRECT vs. INDIRECT transformation types and the masking field)
- 11.European Union. Regulation (EU) 2024/1689 (AI Act), Article 10. (Data and data governance requirements for high-risk systems)
- 12.Gibson Dunn (2026). EU AI Act Omnibus: postponed high-risk deadlines. (Regulation (EU) 2026/1744, in force 27 Jul 2026. Annex III application date 2 Dec 2027)
- 13.Certivo (2026). Electronic signature and audit trail requirements. (21 CFR 11.10(e), ALCOA+, and the attribution and reason-recording requirements for automated corrections)
- 14.European Central Bank (2024). Guide on effective risk data aggregation and risk reporting. (Attribute-level lineage and the quality issue register requirement)
- 15.Deloitte (2024). BCBS 239 benchmark survey 2024. (72% had defined a quality risk appetite against 17% who had operationalized it)
Tool landscape
- 16.Integrate.io (2026). Agentic AI data quality tools. (A roundup of nine products; none confirmed to offer autonomous repair together with rollback)
- 17.Informatica (2026). CLAIRE headless multi-agent announcement. 20 May 2026.
- 18.DQOps. Data quality and audit trails. (Why unstructured logs are insufficient as audit evidence, and the case for a structured store)
Earlier Pebblous reports
- 19.Pebblous (2026). The Proof Was Right. How It Got There Wasn't Recorded. (Companion piece on the gap where output is verified and reasoning is not)
- 20.Pebblous (2026). It's the Weak Harness, Not the Weak Model, That Kills Your Agent. (The harness-first thesis and its tension with this study)
- 21.Pebblous (2026). Great Expectations Deep Dive — The First Line of Defense for ML Pipeline Data Quality and Its Limits.
- 22.Pebblous (2026). PrismaDV Reads Downstream Code to Write Data Validation Rules.
- 23.Pebblous (2026). hermes-agent's Self-Learning Loop: How Data Quality Degrades.