Executive Summary

Key Numbers

Of the four numbers below, the first three are the terms of the trade the paper measured. The speed you gain, the recall you give up, and the extra loss that shows up in real deployments. The last one comes from a separate line of research and shows why the duplicates you miss matter later.

Sources: SieveIVF (arXiv:2608.03199), Lee et al. 2022

Deduplication is the step nobody watches in a data cleaning pipeline. It takes nowhere near the human labor that labeling does, and it does not change the output as dramatically as filtering. But once a training corpus reaches hundreds of millions of records, this is the step that holds the cluster longest, because every one of those vectors has to be checked against the others for near neighbors.

The standard answer is IVF, the inverted file index. Vectors are sorted into thousands of partitions in advance, and when a query arrives, only the few most promising partitions get searched. The problem is that those few are handed out identically to every query. Deduplication only needs neighbors above a similarity threshold, and fixed-probe IVF allocates its budget with no knowledge of that condition. An easy query finds its answer in the first partition and keeps digging through the rest anyway; a hard query runs out of budget without finding anything.

When the authors took apart four production Hunyuan workloads, the scale of the waste became visible. The exact top-10 neighbors that satisfied the threshold were scattered as deep as rank 16 at the 90th percentile, yet between 93.76% and 99.95% of them already sat inside the first eight partitions. For most queries, eight partitions were enough.

The subjects of that analysis were four indexes of 10 million 768-dimensional vectors each, with the vectors spread across 2,400 partitions. The similarity threshold for calling something a duplicate ranged from 0.85 to 0.93 depending on the character of the workload. At corpus scale the same structure simply grows. A few wasted partitions per query, repeated a hundred million times.

What makes the waste quiet is that it never surfaces as a failure. The search completes normally and the duplicates get filtered properly. The excess just drains into the cloud bill and the cleaning lead time. It is a cost nobody opens a ticket for.

The rule SieveIVF introduces fits in one sentence. If W partitions in a row produce no candidate that satisfies the threshold, the query stops searching. Any qualifying result along the way resets the counter to zero and the search continues. Because each query now stops at a different point, partition access scatters, so the authors preserved cache locality by regrouping queries on the fly whenever they were ready to visit the same partition.

Ending a search early is not a new idea in itself. Earlier work trained a separate predictor to guess when a given query should stop. SieveIVF removes that training. No training queries, no predictor, just a count of how many partitions in a row came back empty. Adoption costs almost nothing, but the basis for the decision is thin in exactly the same proportion. All the information behind the stop is whatever has already been searched.

The diagram below shows how fixed probing and early stopping handle the same query differently. The top row is the conventional approach burning through its entire partition budget. The bottom row is a query that satisfies the threshold at the third partition, then hits eight empty partitions in a row and stops at the eleventh.

With W set to 8, the four 10-million-vector Hunyuan workloads ran between 4.1 and 7.6 times faster, and recall loss landed between 0.03 and 1.13 percentage points. On the two public 100-million-vector datasets the speedups came out between 6.1x and 8.4x. The number in the headline is the far end of that range, DEEP-100M, which ran 8.38 times faster and lost 2.29 points. Under identical conditions LAION-100M came in at 6.11x and 1.43 points.

What the table is really saying is not in any single figure but in the fact that both columns move together. Widen the window from 8 to 12 and the loss drops below half while the speed gain shrinks with it. The authors left the index structure, the centroids, and the in-partition search logic untouched. W is the only parameter tuned at run time. So the question you have to answer first narrows to one: on this dataset, how much can you afford to miss? The appendix rests on the same premise. It proposes running 2% of a batch as a sample and picking the W that meets a target loss of 0.5% automatically.

What the paper measures is search quality. The 2.29 points are the share of neighbors that should have been found and were not, not a measure of how much worse the trained model gets. That link sits outside the paper's scope. Yet it is exactly the link that matters to anyone operating the data, because the missed duplicates are not deleted. They stay in the training corpus.

What happens when duplicates stay is comparatively well measured. The study Lee and colleagues presented at ACL 2022 reported that removing repeated strings from training data cut how often a model emits training sentences verbatim by roughly a factor of ten, from over 1% of output tokens down to around 0.1%. The same study also produced field measurements: 6.7% of C4 and 18.6% of RealNews were duplicates.

More importantly, that risk is not evenly distributed. Follow-up work from Carlini and colleagues showed that the degree of memorization scales log-linearly with the number of times a sequence appears in training data. Kandpal and colleagues confirmed that the same structure carries through to the success rate of data extraction attacks. So how dangerous the missing 2.29 points actually are depends not on the percentage but on how often the missed items repeat inside the corpus. Missing 2.29 points of items that appear once each is not the same number as missing a document cluster that repeats hundreds of times.

The risk does not pile up inside the model alone. The same Lee study reported that in standard datasets, more than 4% of the validation set overlapped with the training set. When duplicates survive, evaluation scores come out higher than the model's real ability, and the judgments made on those scores go on to steer the next round of training. Missed duplicates contaminate the ruler as well as the model.

Which kind of duplicate early stopping misses more often is not in the paper. Given the nature of the rule, the missed neighbors are not random: they cluster among queries whose matches sit near the threshold and deep in the partition ranking. How that distribution relates to duplication frequency is something nobody has measured yet. That is why a single recall number cannot stand in for downstream risk.

This is where the governance problem starts. A decision has been made that affects model quality, and there is no place that records it. Dataset cards and datasheets increasingly ask for provenance, collection method, licensing, even safety filtering. The documentation around C4 and RedPajama serves as a de facto template. Nowhere on that list is a field for how early the approximate search was stopped.

Even the fields that are on the list get filled in poorly. A study that audited widely used medical imaging datasets against documentation standards found only 20% to 39% of the required items satisfied, with provenance and quality the biggest gaps. An approximation parameter from the cleaning pipeline never made it onto the required list in the first place, so it is not even a candidate for that compliance rate.

The paper's automatic W selection makes the gap sharper rather than smaller. If each batch runs a sample and picks the W that meets a target loss, then W is no longer a constant written down once somewhere in the pipeline. It becomes a runtime value that differs from batch to batch. A value chosen that way may survive in the logs, but it does not make it into the documents that describe the dataset. Months later, when something turns up in the model, there is no way to reconstruct how aggressively that batch was cleaned.

Regulation is inching in this direction. The EU AI Act and the NIST AI RMF both push toward explicit documentation of training data and models. Cases where the required items reach down to approximation parameters in the cleaning pipeline are still rare. There is no reason to wait for regulation to get there. The first reason to keep this record is not audit but self-tracking.

There is not much a team running deduplication can do about this today beyond adding a paragraph to the dataset card. But five things have to appear in that paragraph as numbers for it to be useful later.

• • The similarity threshold used to declare a duplicate, and the embedding model behind it
• • How early the search was stopped, meaning the early-stop window or the probe budget
• • Whether centroid assignment was computed exactly or approximated
• • The measured recall loss and the method used to measure it, sample size included
• • Where the per-batch values are kept, if the setting varies from batch to batch

Five lines will do. With them, when a model later reproduces training sentences verbatim or an evaluation set looks contaminated, the cleaning stage can be put on the list of suspects. Without them, that stage drops out of the investigation entirely. The 2.29 points of recall are not a defect. They are a calculated trade, and the paper disclosed it honestly. The problem is that the party making the trade does not write down its terms.

Primary Source

• 1.Hu, Z. et al. (2026). SieveIVF: Threshold-Aware IVF Execution for Large-Scale Training Data Deduplication. arXiv:2608.03199

Academic Papers

• 2.Lee, K. et al. (2022). Deduplicating Training Data Makes Language Models Better. ACL 2022. arXiv:2107.06499
• 3.Carlini, N. et al. (2022). Quantifying Memorization Across Neural Language Models. ICLR 2023. arXiv:2202.07646
• 4.Kandpal, N., Wallace, E., & Raffel, C. (2022). Deduplicating Training Data Mitigates Privacy Risks in Language Models. ICML 2022. arXiv:2202.06539
• 5.Wang, Z. et al. (2023). Data Management For Large Language Models: A Survey. arXiv:2312.01700