Executive Summary

Key Numbers

The four numbers come from two different benchmarks. The false alarms, the F1 score, and the misses are results from EIDBench, an end-to-end benchmark that attaches 60 tasks to five real datasets and scores 1,500 pass-or-block verdicts. The 48 out of 63 comes from ICDBench, which measures nothing but the ability to find constraints hidden in code, across 63 cases.

The first three numbers are what reading the task code bought. The last one is the hole left behind in exchange.

Source: arXiv:2604.21765, Tables 2 and 3

The paper opens on a scene that is common after an acquisition. A large travel agency buys the startup behind a booking app, and the parent company's DevOps team wires the startup's services into the corporate data lake. Every night an ETL pipeline pushes booking data in, and three consumers read it: a batch job that computes discounts and sends notification emails, a daily report that aggregates bookings in progress, and a training run for a model that predicts whether a booking completes. The table has six columns: name, email, country, customer category, revenue, and status.

Tired of weekend incidents, the team adds data unit tests. Hand a sample of the existing data to Deequ, run its constraint suggestion, and profiling plus heuristics produce constraints on per-column missing rates and value ranges. The pipeline now runs those tests on every new batch to decide whether it passes. What happened over the next three nights is where the paper states its problem.

On the first night the tests passed the batch and the batch job died. The cause was a condition that lived only inside the code. Records whose status column reads COMPLETED must carry a valid value in the email column, and a constraint suggestion that had seen nothing but the data distribution could not see that conditional dependency. The second night it was wrong in the other direction. The tests read GER in the country column and customer category 3 as unfamiliar values and quarantined the batch, but the startup's engineers explained that GER is a value the legacy booking system emits now and then and that 3 is a rare but valid premium tier. On the third night a human decided to let the batch through, and this time model training died. The preprocessing code normalizes revenue by dividing it by its standard deviation, and that batch had a standard deviation of zero, so NaNs entered the training data.

What the three incidents have in common is not on the data side. The answer to the first was in a branch, to the second in the allowed values the code already handled, and to the third in a single division. The question of whether data is clean does not close until you settle what the data is for. The same batch can be safe for the report job and fatal for model training, so a single test sitting in the middle of the pipeline cannot answer that question in the first place.

PrismaDV is not a matter of asking one large model to please write the tests. It takes a data sample and the task code as input and produces constraints by passing through modules with separate jobs. Ordinary code handles profiling and post-processing, and a language model is called only in the middle stages that read the code.

• Profiling. Basic statistics such as schema and value distributions are computed from the sample. This is the stage whose starting point matches existing tools.
• Column access detection. The system finds which columns the task code actually reads and which combinations it reads together. Constraints on unused columns produce nothing but false alarms, so the scope of validation gets narrowed first.
• Data flow analysis. It follows where those columns are compared, computed, and branched on to infer implicit assumptions. A column used in a division, a column guarded by a conditional, and a column passed straight into a library function each leave a different requirement behind.
• Constraint code generation and pre-checking. Inferred assumptions are turned into executable constraints in PyDeequ syntax, and constraints that break syntactically or fail to run are thrown away.

The pipeline in existing tools effectively ends at the first stage. Profile a sample, suggest constraints with heuristics, and that is the output. The two stages PrismaDV adds behind it are where the paths diverge. It checks which columns the task code reads so the validation scope narrows first, then follows how those columns are used inside the code to settle what the constraints say. Feed in the same sample and the constraints come out different once the code arrives as input too.

Going back to the booking data makes each stage concrete. Finding that the status and email columns are read together in the batch job code raises a conditional constraint over the pair as a candidate, and finding the line where revenue is divided by its standard deviation creates the requirement that the batch's standard deviation not be zero. Conversely, when the allowed values for the country column are written into the code as a whitelist, GER gets handled as a valid value rather than an anomaly.

The numbers also record that generating many constraints is not the goal. On the constraint discovery benchmark Deequ produced 1.9 constraints per case on average and PrismaDV 2.6, while simply handing over the code and asking produced between 3.1 and 3.5. The approaches that produced more also produced far more false alarms, and some of their constraints were syntactically broken and never ran at all. PrismaDV puts a pre-check stage in the way, so every constraint left at test time is in an executable state.

Inside the system a data-code assumption graph remains. It connects columns with the assumptions inferred from the code, so picking up one generated constraint lets you walk back to the code location behind it and to the assumption written in natural language. Ask why the constraint that revenue must not be zero was attached, and the graph points at the line that normalizes it.

One side of the graph holds the input columns the task actually reads, and the other holds the implicit assumptions inferred from the task code. There are only those two kinds of nodes and nodes of the same kind never connect, which makes the structure bipartite. Start from a column and every assumption attached to it comes out. Start from an assumption and out come the code location that produced it and the constraint synthesized from it.

This structure matters in practice because validation rules end up passing through human hands. For a team to trust an automatically generated constraint, it has to be possible to tell where to fix it when it turns out wrong. A constraint that cannot state its grounds gets switched off the first time it raises a false alarm, and a rule switched off never gets switched back on. Because natural-language assumptions sit in the middle as an intermediate representation, the paper also leaves room for carrying the same assumption into other syntaxes such as Great Expectations or plain Python assert statements.

The CIKM 2026 demo is the version that puts this graph in a web interface for hands-on use. Attendees run the system over five real datasets and 60 tasks to produce assumptions and constraints, walk the graph and edit assumptions, and run task-aware and task-agnostic tests side by side on batches seeded with errors. It draws a picture where reviewing rules on a data team becomes checking grounds rather than checking a list.

Keeping the two papers apart makes them easier to look up. The demo paper carries its question as its title, "Will This Data Break My Task?" - Interactive Synthesis of Task-Aware Data Unit Tests (arXiv:2608.09376, posted August 10, 2026), and the main paper that covers the method and the benchmarks in detail came four months earlier as PrismaDV: Automated Task-Aware Data Unit Test Generation (arXiv:2604.21765). Every number in this piece comes from the latter. The implementation is public in the deem-data/PrismaDV repository, so the prompts and post-processing that actually drive the four stages can be inspected directly.

The false alarm counts and the miss counts both come out of two benchmarks the authors built for this. They differ in character, and one measures nothing but the ability to find an assumption hidden in code. Each of its 63 cases comes with a data sample, the code that processes it, a natural-language description of the hidden assumption, and a ground-truth constraint, along with data that should pass and data that should be rejected prepared separately. The domains range across payment processing, cricket match rules, and in-game auctions, and the cases run from easy ones such as an explicit assert to ones that need column dependencies scattered through control flow and knowledge of how a machine learning library behaves.

On this benchmark the results break one way. PrismaDV passed 61 of the 63 samples that should pass and blocked 48 of the 63 that should be rejected. Deequ managed 44 and 27, and TensorFlow Data Validation 32 and 32. The outlier detection family blocked nearly everything or passed everything, which amounts to not deciding at all.

The other one is an end-to-end benchmark. Five datasets carry 60 tasks between them, and each dataset gets 25 batches with errors injected. A given batch is not harmful to every task, so of the 1,500 verdicts, 827 should be a pass and 673 should be a block. The assert blocks that establish the ground truth are stripped out of the task code, each method is then asked to produce tests, and the pass-or-block decisions of those tests are compared against the ground truth.

Reading the false alarm column against the block column shows what existing tools were actually doing. Deequ blocked 560 of the 673 harmful batches. On blocking alone that beats PrismaDV's 368. The same tests also blocked 693 of the 827 safe batches. A test that cries wolf more than six times out of eight gets switched off or ignored within days, and the block rate of a test that has been switched off stays a number on paper.

The numbers on PrismaDV's side are not all boast either. The configuration with GPT-5 as the backbone cut false alarms to 112 and passed 305 harmful batches in exchange. Switch to Gemini 2.5 Pro and misses fall to 195 while false alarms rise to 230. The authors themselves describe this point as a preference that shifts with the backbone, and they write that the model should be chosen according to which one you can tolerate less, false alarms or misses. No configuration drives both errors to zero at once.

A mechanism for refining tests over time is proposed alongside. SIFTA is an optimization scheme that picks the informative cases out of the sparse success and failure signals obtained by actually running the tests and the tasks, then updates the prompts of the modules. In the scenario of adapting to new data, its average F1 across five datasets reached 72.8%, ahead of 67.7% for hand-written prompts and roughly 65% for general-purpose prompt optimization. In the new-task scenario, though, the gap narrows to 69.8% against 67.3%, and dataset by dataset there are cells where the hand-written prompts win.

What counts as an informative case is written into the name. SIFTA, short for Selective Informative Feedback for Task Adaptation, computes per-column failure precision, meaning the share of times the task itself actually failed when that column's constraint failed. Task and column pairs with a low value are where false alarms originate, so the optimization is poured into that spot. The direction is not to make the rules denser but to re-measure the grounds of the alarms that are already ringing.

The results from removing the components one at a time should be read carefully instead. Dropping multi-column constraints costs 0.92 percentage points of F1, dropping data flow analysis 0.48, and dropping assumption inference 0.17. Each module contributes, yet most of the gap comes from the fact that the task code was put into the input rather than from any individual module.

The limitations the authors state are not short either. The prototype is built for tasks that use one file and one table, so data flow spread across several scripts and derived tables produced by joins are out of reach for now. SIFTA needs training batches with observed errors, which are not easy to keep around in real operations, and the authors put automating error injection to get past that cold start on the list of next steps. There is also the fact that the task code in the end-to-end benchmark is itself language model output reviewed by humans. Extending the benchmark to real industrial codebases sits near the front of their future work.

From here on this is not the paper's claim but a reading added from the side that has worked on data quality. What this system changes is not validation technique alone. What a quality score attaches to changes with it. Score a dataset on missing rates, uniqueness, and value ranges and that score is a property of the dataset. Start asking whether this batch breaks this task and the score attaches to a pair of data and task. The same table can be ready for the report and not ready for model training.

The party doing the judging moves just as far. Quality used to be defined by the rule list the data team set. Now the code at the point of use states the requirements. This is where AI-Ready starts reading as a relation that asks what a dataset is ready for, rather than a grade stamped on the dataset. The readiness discussion Pebblous covered in It Starts With the Data, Not the Model and the rule-authoring burden noted in the Great Expectations deep dive meet on the same axis. If those pieces dealt with the problem that a person has to write the rules by hand, this paper answers where those rules should be read from.

With the prototype confined to one file and one table, this system cannot be bolted onto a pipeline today. Moving where the judgment is made, though, starts without swapping tools. Reading the rules already running once from the task side is enough. Three things are worth checking first in practice.

• How many of the data validation rules running right now aim at columns that are actually read? A missing-rate alarm on a column nobody reads spends nothing but false alarm budget.
• Can you answer why a rule came to exist? A rule that cannot state its grounds goes off at the first false alarm, and the fact that it went off gets recorded nowhere.
• Does the pipeline reflect the fact that requirements differ across the tasks using the same table? A design that tries to guard several tasks with one gate produces misses and false alarms at the same time.

• 1.Chen, H., Phani, A., Schelter, S. (2026). "Will This Data Break My Task?" — Interactive Synthesis of Task-Aware Data Unit Tests. CIKM 2026 (Demo Track).
• 2.Chen, H., Phani, A., Schelter, S. (2026). PrismaDV: Automated Task-Aware Data Unit Test Generation. arXiv preprint.
• 3.DEEM Group. deem-data/PrismaDV. GitHub repository (reference implementation).