Executive Summary

For the past few years there has been a quiet consensus among the people who build large language models. To make a good model you need a lot of data; to gather a lot of data you have to scrape the whole web; and if you scrape the whole web, copyrighted content inevitably comes along with it. So "rights-cleared data" came to mean "data that gives up scale." The logic runs in three steps: clearing rights shrinks the usable data, less data means less training, and less training means weaker performance. That is how the assumption was built — that clean data comes with a "performance tax" attached.

The assumption was plausible because real numbers backed it. FineWeb, a large dataset refined from web crawls, runs to about 15 trillion tokens; the Allen Institute's Dolma reaches roughly 3 trillion. Corpora that filtered for license from the outset, by contrast, stalled at a fifth to a seventh of that. The scale gap was undeniable, and the inference — "so the performance gap will be just as wide" — looked natural. The bars below show exactly that gap: the size difference between license-agnostic web-crawl datasets and rights-cleared legal corpora.

But there is a trap in this picture. The equation that a scale gap is a performance gap was an assumption that had never been tested. Does five times more data make a model five times better? Or is there a point past which "what you put in" overtakes "how much you put in"? To answer that head-on, there is no substitute for actually training models on a small-but-fully-rights-cleared dataset and standing them on the same benchmark as models trained on far larger data.

That is precisely the axis of comparison this report takes up. Instead of the familiar "open versus closed data" framing, we narrow it one notch further and ask about performance parity across regimes. On one side, a model trained only on rights-cleared open data; on the other, a model trained on unlicensed data with no regard for permissions. Watching how — and how far — these two regimes diverge on the same scale is the most honest way to test whether the assumption is true. The protagonist of that evidence is Common Corpus.

The answer to "what made this possible" lay not in chasing scale but in stacking sources selectively. Rather than scraping the web indiscriminately and filtering after the fact, Common Corpus gathered only content whose rights status had been verified, sorted into six collections from the start. Each collection was filled solely with material whose copyright had lapsed (public domain) or that was distributed under an open license, and the result came to roughly 2 trillion tokens. The goal was never scale; it was license purity, and 2 trillion tokens accumulated as a by-product.

The makeup of the six collections reveals the dataset's character directly. Nearly half is public-domain books and newspapers (OpenCulture), followed by legal and administrative documents (OpenGovernment). High-quality code, open scholarship, wikis and forums, and structured knowledge across 300-plus languages fill the rest. There is almost no casual chatter or social-media narrative; instead, formal written prose with orderly sentence structure dominates. That skew turns out to be the key to understanding the benchmark results we will see later.

As important as scale is the breadth of language. Common Corpus has 33 languages above 1 billion tokens, 8 of them above 10 billion, and more than 40% of the whole is non-English — which led Pleias to describe it as "more multilingual open content than any other open dataset." This is where it parts ways with the English-dominated Dolma and FineWeb. As books and legal documents flow in across many languages in balance, the dataset gains a different axis — language diversity — in exchange for giving up scale. One thing, though, must be split out honestly. The 300-plus languages that OpenSemantic advertises in the distribution above are Wikidata-based structured knowledge, not natural-language pretraining text. The weight of actual sentence learning is carried by the 33 languages above 1 billion tokens and the 8 above 10 billion; the number 300 should be read as the structured-coverage layer sitting on top of that.

2 trillion or 2.27 trillion? The honest arithmetic of a growing dataset

When Common Corpus is cited, the token count appears differently from source to source. The first announcement in November 2024 gave 2.003 trillion tokens (2,003,039,184,047); the June 2025 paper reports about 2 trillion (1,998,647,168,282); the dataset card distributed today points to 2.267 trillion (2,267,302,720,836). This difference is not a matter of differing deduplication. It is because this is a growing dataset whose collections have kept being updated and expanded after publication. So this report does not blur the two numbers together. The snapshot used in the paper's experiments is about 2 trillion tokens; the release you can download now is 2.27 trillion and still climbing.

Now let us put them on the scale. The paper pitted two models trained on Common Corpus (PleIAs 350M and 1.2B) against eight models of similar or far larger size across three multilingual benchmarks. MultiBLiMP asks whether a sentence is grammatically correct using minimal pairs; XStoryCloze and XCOPA probe world knowledge and commonsense causality. Before reading the table, here is the conclusion up front: it won on grammar and trailed on commonsense reasoning. That boundary is the whole of this evidence.

Source: Common Corpus paper, Table 2 (arXiv:2506.01732v2), LM Evaluation Harness. Orange highlights mark the cells where PleIAs leads on MultiBLiMP.

The lead on grammar, and the single strongest line

Read the MultiBLiMP column top to bottom and you can see where the assumption gives way. PleIAs 350M (0.774) outscored not only its own size class (270M–560M) but XGLM 1.7B (0.710) and OLMo 1B (0.699), models close to three times its size. Within the table there is no model the 350M lost to on the grammar benchmark. The 1.2B model (0.797) trailed only Gemma 3 1B (0.799), by 0.002 — well within noise — and beat everyone else. To be honest about it, the 1.2B and Gemma 3 1B are effectively on par. The argument this report leans on is not "the 1.2B won," but "the 350M punched above its weight class."

The shortfall on commonsense reasoning — a boundary not hidden

The two right-hand columns of the same table point the other way. On XStoryCloze and XCOPA, PleIAs 350M and 1.2B mostly trail Gemma 3 and XGLM. On these benchmarks, which probe world knowledge and narrative causality, the models trained on closed data scored better. (Against OLMo 1B, the results are similar or narrowly ahead.) This shortfall is not something to hide but a clue to be explained — and the explanation runs straight back to the collection composition we saw earlier.

Common Corpus is heavily weighted toward orderly written prose — books, law, science, code — with almost no everyday conversation or social-media narrative. That is superb material for learning grammatical regularity, but relatively thin material for learning commonsense causality like "drop a cup and it breaks," or worldly, everyday knowledge. Composition builds certain capabilities and pays a penalty on others. The lead on grammar and the shortfall on commonsense reasoning are two sides of the same cause. This causal link is the most intellectually interesting part of the evidence, and it meshes with the boundary the authors draw in the next section.

This evidence earns trust because the authors spell out the limits of their own result. The paper does not declare "there is no performance tax." Instead it acknowledges a paradox it names the open data paradox: the total volume of legal, open data collectable today is enough to pretrain small models, but not yet enough to build anything larger. In the paper's own words, the data collected (2 trillion tokens) is "well-suited to pretraining size-limited models, but larger models would require substantially more data."

The limits fall along three lines. First, model size: small models from 350M to 1.2B are within reach, but past a few billion parameters the data runs short. Second, training stage: the authors write that this data "lacks instruction-tuning or task-specialized data." It supports pretraining, but the raw material for the stage that teaches a model to follow instructions is not yet there. Third, the kind of capability. As we saw, it leads on grammar but trails on commonsense reasoning. The "no performance tax" rebuttal holds only inside these three fences.

So what survives from this evidence? This report does not make the claim that "clean data lets you catch up to closed data all the way to a GPT-scale large model." That is territory the data has not yet reached. What survives is a narrower, firmer proposition. Within the range of small, multilingual, grammar-oriented models, rights-cleared data paid no performance tax. Knowing that range precisely is as much the purpose of this piece as rebutting the assumption. And that very range — small, specialized, multilingual models — being exactly what regulation and industry most need right now is the starting point of the next section.

There is weight in the timing of this evidence arriving now. A few days ago, on 2 August 2026, the supervisory and enforcement powers of EU AI Act Chapter V took effect. Obligations for providers of general-purpose AI (GPAI) models already began on 2 August 2025; now the Commission has moved to the stage where it can actually enforce compliance. Models placed on the market before August 2025 have until 2 August 2027 as a grace period — but the clock has already started ticking.

One of the things Chapter V asks of GPAI providers is publishing a sufficiently detailed summary of training data and complying with copyright policy. The penalty ceiling is 3% of annual global turnover or €15 million, whichever is greater. Here one thing must be made clear. The EU AI Act text neither mentions nor recommends Common Corpus. Regulation did not single out this dataset. The argument of this piece is that the process regulation demands (documenting sources and disclosing licenses) and the structure Common Corpus already had at the design stage (complete provenance at the collection and license level) fit together structurally.

Flip that alignment around to the lens of cost and the implication sharpens. Auditing licenses after the fact is never free. According to an earlier Pebblous analysis, a data-license audit runs 40–160 hours per program and 3.5–15 hours per dataset. Interpreting ambiguous clauses and judging commercial use consumes lawyers' time. The approach of building a model first and then checking backward whether "we were allowed to use this data" grows harder to sustain as the number of datasets multiplies.

This report stands in the line of an earlier discussion we covered on the asymmetry of source disclosure. There, Common Corpus sat in a footnote as the flagship of the "enumerate every source" fully-open camp. This piece lifts that footnote into the body, digging into what the fully-open extreme actually delivered in performance. Between those who conceal their sources and those who enumerate them, if the latter does not lose performance too, the balance of the choice shifts.

The reason Pebblous watches this evidence closely is simple. The process by which Common Corpus classified 2 trillion tokens by license and source, filtered them, and normalized them across languages is an empirical, dataset-scale version of the data-quality diagnostics we run with DataClinic. It is a case that backs, with external benchmark numbers, the proposition our AI-Ready Data series has repeated: that the state of training data determines model performance, and that there are signals that tell you whether data is ready.

Looked at more precisely, this evidence goes a step beyond "data makes performance." It shows which property of the data builds which capability. The orderly, book-law-science-heavy composition lifted grammatical competence; the absence of casual narrative cut into commonsense reasoning. This causal link, running from training-data composition (source, license, language diversity) to specific model capabilities, maps directly onto the axes we diagnose: source, license, diversity. "Data comes before the model" stops being a slogan here and becomes a measurable proposition.

The practical implications are just as clear. For an organization facing EU AI Act GPAI obligations, "legal data can perform too (at least within the small, multilingual, grammar range)" is a real-world basis for compliance decisions. Rather than paying data-license audit costs after the fact, organizing provenance from the design stage is more sustainable — and, as we argued, the bottleneck for sovereign AI is ultimately legal, organized source data. This evidence adds a counterexample to both diagnoses: that organizing sources is not the same as losing performance.

So here is where this piece lands. If the assumption that data quality and legality are at odds with performance is wrong — at least within a specific range — then preparing data moves closer to a condition that creates competitive advantage than to a cost regulation imposes. This is not a Pebblous claim alone; it is an observation a French startup put on the table in the form of a benchmark. We intend to keep watching how far that observation extends, from small-and-multilingual toward large-and-general-purpose.

Academic · Datasets

• 1.Langlais, P. et al. "Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training." arXiv:2506.01732, 2025 (ICLR 2026 poster; OpenReview id=0wSlFpMsGb). — Source for token counts, collection breakdown, benchmark Table 2, and the open data paradox.
• 2.PleIAs. "Common Corpus dataset card." HuggingFace, 2024– . — Primary source for the current 2.27T-token release, six collections, and language distribution.
• 3.Langlais, P. "Releasing the largest multilingual open pretraining dataset." HuggingFace Blog, 2024-11-13. — v1 announcement (2.003T tokens) and language distribution.
• 4.Kandpal, N. et al. "The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text." arXiv:2506.05209, 2025 (EleutherAI). — Comparison against a sibling fully-legal dataset.
• 5.Penedo, G. et al. "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale." arXiv:2406.17557, 2024. — Web-crawl scale (15T tokens) reference.
• 6.Soldaini, L. et al. "Dolma: An Open Corpus for Language Model Pretraining." Allen Institute for AI, 2024. — Semi-open scale (3T tokens); OLMo's training data.
• 7.Bommarito, M. et al. "The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models." arXiv:2504.07854, 2025 (ALEA Institute). — Fully-legal sibling (1.35T tokens) comparison.

Policy · Statistics

• 8.Future of Life Institute. "Enforcement of Chapter V under the EU AI Act." artificialintelligenceact.eu, 2026. — GPAI obligations, enforcement timeline (2025 / 2026 / 2027-08-02), and penalty ceiling.

Pebblous-adjacent

• 9.Pebblous. "The Tool Is Named, the Sources Are Hidden — The Asymmetry of AI Training-Data Disclosure." 2026-07-11. — Prior report citing Common Corpus as the flagship of the fully-open camp.
• 10.Pebblous. "The Cost of Auditing Open-Dataset Licenses." — Reference point for the 40–160 hours of retroactive auditing.
• 11.Pebblous. "The Conditions for AI-Ready Data" · "Five Signals of AI-Ready Data." — On the link between data state and performance.
• 12.Pebblous. "Distillation and Sovereign Data." — On the bottleneck of legal, curated source data.