Executive Summary
Someone drops a few fabricated sentences into the comment section under a news article they don't own. Those few lines get picked up by a crawler, pass through deduplication, language filters, and a quality classifier in turn, and end up sitting inside the pretraining data of a large model. In July 2026, researchers at the University of Washington and Ai2 measured the survival rate of that path for the first time. Their result reopens the word "cleaned": what does that certification actually guarantee, and what does it leave out?
The final probability that comment poisoning survives curation and reaches training data was 0.13%. It sounds small, but at the scale of Common Crawl, which makes up most of this data, it affects more documents than all of Wikipedia does (0.067%). The reason quality filters miss this poisoning is not weak performance; it is that the filters were designed to look at something else entirely.
What curation verifies is a content signal: how plausible a document looks. It is not a source signal, the record of whose hands the text passed through to get there. That gap is itself a data provenance problem, and in AI-Ready Data practice it is why you still have to ask where a document came from after curation is done.
0.13%
Final inclusion rate
Share of comment poisoning that survives curation to reach training data
+18.6%
Model preference shift
Bias toward a target entity from poisoning just 0.1% of tokens
47.1%
Quality-filter survival
Poisoned comments that passed the educational-value classifier stage
65.3%
Survival after rewriting
Belief-manipulation poisoning that persists even after AI rewriting
What Curation Was Built to Remove
Text scraped from the web can't be used as-is. So modern pretraining corpora (Dolma, FineWeb, DataComp-LM) run it through three layers of curation: deduplication to stop the same document from entering many times, language filters to remove unwanted languages, and a quality classifier that scores how usable each document is.
Of these, the quality classifier has been the quiet driver of recent performance gains. FineWeb-Edu's educational-value classifier is the canonical example: a lightweight regression model trained on synthetic labels generated by Llama-3-70B. Training on data filtered through it raised MMLU by 12% and ARC by 24%. Letting a machine grade "is this document as useful as a textbook?" actually worked.
There's one easily missed fact here. What these filters grade is always the whole document. "Is this page Wikipedia-like?" and "does it have educational value?" are judged at the document level. A well-written news article passes as high quality. The problem is that the comments attached below that article come along with it.
The Comment Section Blind Spot, Where Poisoning Survives Three Stages
An attacker doesn't need access to the model's training pipeline. They plant manipulated text in the comment section or an open message board of a web page they don't even own. It's the same method social propaganda campaigns use. Of the pages the researchers scanned, 3.4% had a comment platform, and many of those allowed immediate posting with no authentication. WordPress alone accounted for 85.2% of that infrastructure.
The researchers decomposed the probability of poisoning reaching training data into the product of three stages: whether a comment section exists so injection is possible, whether the crawler actually captures the comment, and whether it passes the curation filters. The survival rate at each gate is below.
| Stage | Gate | Survival |
|---|---|---|
| S1 | Pages with a comment section open to injection | 3.4% |
| S2 | Crawler (Resiliparse) actually captures the comment | 71.9% |
| S3 | Passes the triple curation filter (heuristic 28.8% → language 40.4% → quality 47.1%) | 5.5% |
| Final inclusion rate (S1 × S2 × S3) | 0.13% | |
What stands out is that the quality filter's survival rate, 47.1%, is the highest of the three. The gate you'd expect to catch poisoning best is the one that lets it through most. Section 3 explains why.
250 Documents Is Enough, and How Poisoning Sticks in a Model
Why 0.13% is not a number you can wave away becomes clear from how poisoning actually sticks in a model. The researchers poisoned 0.1% of the tokens for OLMo-3-family models from 65M to 1.3B parameters, then measured preference toward real competing brands (Citroën vs. Renault, Boeing vs. Airbus, Pfizer vs. Moderna). Preference shifted 18.6–19.0% toward the side the poisoning pushed, and the bias held constant regardless of model size.
More troubling is that the poisoning doesn't wash out easily. After supervised fine-tuning (SFT), the 65M model retained 40% of the effect, and the 1.3B model didn't lose it entirely either. Even the common curation practice of rewriting documents with AI to sanitize them (WRAP-style) left 65.3% of belief-manipulation poisoning alive. Changing the injection format to conversational, Q&A, or unlabeled made no difference; all three came through strongly poisoned.
Prior work (Souly et al., 2025) showed that as few as 250 poisoned documents can plant a backdoor. Work backward from the 0.13% inclusion rate and you can secure those 250 by commenting on 100,000 to 1,000,000 pages, a scale an automated comment bot reaches in a day. A small probability meeting a large scale is no longer small.
There's an interesting contrast. Programmatic advertising could not be poisoned the same way. Because 92.9% of ad slots are empty placeholders filled in later by JavaScript, a static crawler never captures them. The risk isn't all web data; it's the open surfaces where people leave text directly: comments, forums, reviews.
Quality Filters See What Was Written, Not Who Wrote It
Now back to why the quality filter had the highest survival rate. A quality classifier looks at the statistical plausibility of the whole document. But poisoning doesn't arrive as a low-quality document. It's a foreign object hitching a ride on a news article or blog post that already passed as high quality. From the filter's point of view, that document is still one of high educational value, and a few lines of comments barely move the verdict. Poisoning isn't the filter's target; it's in its blind spot.
Deduplication is no different. When the same comment repeats across many pages, it can even look to the filter like content confirmed by multiple sources, which helps it survive. Repeated posting could be a clue for bot detection, but current pipelines have no dedicated stage that checks for it.
In the end the problem isn't performance, it's perspective. The curation pipeline inspects only the content signal: "what does this text say?" The source signal, "who posted this text, through what path of trust?", was never part of the design. The gap between those two is exactly the hole this paper exposed.
The three mitigations the paper proposes — separating comments from body text for a dedicated check, provenance-aware filtering that down-weights pages with unauthenticated open posting forms, and a temporal-consistency check that catches abrupt content changes between crawl snapshots — all point in one direction: move the basis of filtering from content signal to source and submission-path signal.
Mitigation isn't confined to the curation pipeline either. As noted, WordPress alone accounts for 85.2% of comment infrastructure, so if a few platforms adopt authenticated posting, rate limits, or anomaly-pattern moderation, the possibility of injection itself drops sharply. It's an approach that turns a problem curation has to filter after the fact back into a problem that's hard to plant in the first place.
Questions to Ask When You're Told the Data Is "Cleaned"
For a practitioner who buys or builds data, this paper redefines one concept. A "cleaned" certification means the data passed a document quality score, not a guarantee that every piece of text inside it arrived through a trustworthy path. Curation is closer to a plausibility check than a pass/fail verification.
The paper itself is clear that this figure is not a finished diagnosis. It used Common Crawl as a proxy for production crawlers, based the estimate on a single pipeline (Dolma 3), and validated in a sandbox rather than through live injection. So the number 0.13% is less an answer than a reason to check for yourself where your data came from.
So when you're handed a vendor's "curated and verified" line, it's worth raising the level of the question by one notch. Below are prompts that can serve as a starting point.
- Is this dataset's quality score assigned per document, or does it distinguish segments within a document (body, comments, quotes)?
- How much text comes from open comments, forums, and reviews, and is that origin preserved as a label?
- When the same sentence repeats across multiple sources, is that treated as grounds for trust or as a clue of bot activity?
- Is there a temporal-consistency check that filters out pages whose content changed abruptly between crawl snapshots?
These questions all converge on one axis, data provenance: beyond what was written, can you trace where it came from and whose hands it passed through? The problem that remains after curation lives right here, and so does the reason AI-Ready Data has to address not only content quality but source trust. Calling data "clean" becomes a verifiable claim only when you can say where that data came from.