Executive Summary

Four numbers run through this piece.

The title of Will Lockett's piece is provocative: "AI Distillation Attacks Are Profoundly Stupid." The gist runs like this. Anthropic went so far as to send letters to regulators claiming that a competitor had distilled its models — yet Anthropic itself trained its models on books scraped without the copyright holders' consent, and paid $1.5 billion for it. If someone else's unauthorized training is an "attack" while your own unauthorized training is "innovation," that standard doesn't hold together.

First, the facts need sorting out, because two events are routinely conflated here. One is the distillation activity Anthropic disclosed in February 2026, at a scale of 16M conversations / 24,000 accounts — but that figure is the combined total across three companies: DeepSeek, Moonshot, and MiniMax. The other, revealed in a June 2026 letter to the U.S. Senate Banking Committee, is a scale of 28.8M conversations / roughly 25,000 accounts, which Anthropic described as "the largest known distillation attack ever" and attributed to Alibaba (Qwen). The window was specified as April 22 to June 5, 2026. The two events differ in both scale and target.

Lockett's verdict — "stupid" — has a basis too. Experts point out that distillation is, to begin with, a legitimate technique used widely across academia and industry. The lineage of knowledge distillation has been a standard tool ever since Hinton and colleagues formalized it in 2015. So the moment you label distillation itself an "attack," the boundary between a legitimate technique and improper unauthorized use blurs. It may be a terms-of-service violation, but whether it can be regulated as copyright infringement or trade-secret theft is still undecided.

So the hypocrisy debate ultimately converges on one point. The heart of the matter is not "what was trained on" but "where you got the data and how." The same structure recurred in the Meta 267TB lawsuit we covered previously: it was the acquisition path, not the act of training, that determined liability. The debate over hypocrisy is the surface; beneath it lies the real question — the absence of norms governing data provenance. To answer that question, we first have to look at what distillation actually transfers.

The mechanism of distillation is surprisingly humble. You pose questions to a large teacher model and train a student model to imitate its responses and probability distributions (the logits). When the teacher answers "the next word after this sentence is A with 68% probability, B with 21%," the student copies even those subtle shades of probability. That is a far richer signal than being handed a single correct answer. Here is the crux: the teacher's output is a compressed version of the source data distribution the teacher learned from. The student never sees a single line of the source data, and cheaply inherits only the statistical shadow that data etched inside the teacher.

That is why even a small amount of teacher output can produce a powerful student. DeepSeek's publicly released R1-Distill-Qwen-32B scored 94.3% on MATH-500, replicating much of the reasoning ability of far larger frontier models. Judging by the benchmark numbers alone, the distance between original and student is startlingly small. This is exactly where the free ride happens: the student does not inherit the hundreds of millions of dollars the teacher's makers poured into data collection and curation.

The gap is stark in numbers. DeepSeek-V3's final training compute is reported at roughly $5.6 million, which is 10 to 100 times cheaper than the $50–100 million range spent on GPT-4-class frontier models, depending on the baseline. But SemiAnalysis did not read this low cost as purely an algorithmic triumph. The real reason R1's reinforcement-learning cost stayed around $100,000, it argued, was that the supply of high-quality question–answer pairs was limited from the start. Here too the bottleneck was not compute but data. Beneath the surface of "low cost" sits the scarcity of the source data that props that low cost up.

But there is something the benchmarks fail to show: what distillation cannot transfer. The tails of the distribution — the rare but decisive long-tail cases — the robustness of alignment, and rare-domain knowledge are seldom captured in samples of teacher output. And when you iterate across generations on distilled and synthetic data, this loss accumulates. This is model collapse.

The study Shumailov and colleagues published in Nature in 2024 measured this collapse precisely. Repeatedly retraining on pure synthetic data pushed perplexity from 34 up into the 50s, and after roughly nine generations the output degenerated into meaningless repetitive lists. What disappeared first? Because common cases are reproduced in abundance while rare cases number only a few, recursive training forgets these scarce tails first. The model converges more and more toward the average alone. We explored the mechanism and market implications of this phenomenon in greater depth in the economics of synthetic-data contamination.

What distillation steals is the outward appearance of capability, and what it cannot steal is the diversity and robustness that source data holds. Distillation, which looks like low-cost replication, in fact parasitizes the cost the original paid. So how much does it cost to secure that original legally?

The price tag on source data is no longer an abstraction. In Bartz v. Anthropic, Anthropic agreed to pay more than $1.5 billion, including interest, over 482,460 works acquired from pirate libraries such as LibGen. That works out to about $3,000 per book. One analysis noted that, in theory, the potential exposure could have exceeded $70 billion.

The logic of the ruling matters especially. Judge Alsup drew a line between two acts. Training a model on copyrighted works may itself be fair use, but acquiring and storing pirated copies is infringement. Here the framing from the previous section is fixed in the language of the court: the problem was not the training but the provenance.

Why does source data cost more than distillation for the same performance? Because the source comes with this legal bill attached. Properly licensed data pays its cost at the acquisition stage. Distillation, by contrast, takes only the teacher's output and sidesteps that bill. But sidestepping is not the same as immunity. Bartz left open the possibility that the teacher model's provenance risk transfers to the distilled student.

The market is already pricing this scarcity in. According to the Stanford AI Index 2025, the share of restricted tokens in the representative crawl dataset C4 jumped from 5–7% in 2023 to 20–33% in 2024, and tokens restricted by terms of service reach 45–55% of the total. The doors to open data are closing fast. Meanwhile, real licensing deals have taken shape: News Corp–OpenAI at $250 million over five years, Reddit–Google at roughly $60 million a year. The market specialized in data licensing is projected to grow from $4.8 billion in 2025 to $22.6 billion in 2034, an 18.8% annual rate. For more on the movement around licensing standards, you can continue with our report on the RSL content-licensing standard.

Widen the view to the level of nations and the bottleneck comes into sharper focus. Countries pursuing sovereign AI mostly pour money into the same places: GPUs and foundation models. CNAS's Sovereign AI Index shows where sovereign AI project budgets flow worldwide. Compute infrastructure dominates, and national data initiatives sit dead last.

Korea is no exception to this pattern. For 2026, KRW 2.08 trillion (about $1.4B) was earmarked for strengthening AI compute resources, and the total AI budget grew to roughly KRW 9.9 trillion (about $6.8B) — but the center of gravity is on GPUs, and the scale of securing domestic data is comparatively opaque. A structural constraint compounds this. According to a Tony Blair Institute analysis, English makes up about half of Common Crawl while Arabic accounts for less than 1%. The data imbalance across languages and cultures is a burden non-English sovereign AI efforts carry from the very start.

Hence a paradox. To secure sovereignty you need high-quality domestic data — but because that is in short supply, you end up chasing the global frontier by distilling it. When the media reports goals like Korea's "securing 95% of national-champion AI performance," it effectively means benchmarking against the frontier, which is an indirect form of chasing and a distillation pressure. Epoch AI has analyzed that, on current trends, high-quality language data could be exhausted around 2026, and that as high-quality data concentrates among a few actors, the payoff from adding compute diminishes. Data, it argues, is a bottleneck that sits above compute.

A single chain runs from start to finish. The hypocrisy debate reduces to a question of data provenance; provenance leads to the value of the source data that distillation cannot transfer; that value hardens into a price on the licensing market. And whether you have the domestic data infrastructure to afford that price decides the fate of sovereign AI. The conclusion is clear: real sovereignty comes not from the number of GPUs but from the lineage of the data.

So where does practice begin? An organization looking to adopt a low-cost distilled model should first reckon with three long-term risks. First, the IP and copyright exposure by which the teacher model's provenance risk can transfer to the student (the precedent set by Bartz). Second, the degradation in long-tail and edge-case performance that benchmarks fail to catch. Third, the collapse that arrives at the end of generational iteration unless you retrain on your own data and manage its lineage.

The national-level remedy rests on the same logic. An allocation of 59% to compute and 7% to data is not sustainable. Collecting and licensing source data in your own language and domains, and building a pipeline that verifies its diversity and lineage, must go not after GPUs but alongside them. If you are curious about the texture of the closed-versus-open debate around DeepSeek and Qwen, our piece on Meta, MUSE, Spark, and closed-source models offers adjacent context.

Let us return to Lockett's question. If a distillation attack really is a "stupid" move, the reason is not only that it is hard to block with regulation. Distillation, in the end, merely borrows the foundational asset that is source data; it does not let you own it. No matter how well you copy someone else's distribution, without the data that first created that distribution you cannot build the next generation on your own. That is why you can't buy sovereignty by distilling it.

Editor's Note

Academic Papers

• 1.Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). "AI models collapse when trained on recursively generated data." Nature, 631, 755–759.
• 2.DeepSeek-AI. (2025). "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv:2501.12948.
• 3.Villalobos, P., Sevilla, J., Heim, L., et al. (2023). "Will we run out of data? Limits of LLM scaling based on human-generated data." Epoch AI, arXiv:2211.04325.
• 4.Hinton, G., Vinyals, O., & Dean, J. (2015). "Distilling the Knowledge in a Neural Network." arXiv:1503.02531.

Policy, Statistics & Industry

• 5.CNAS. (2026). "Sovereign AI Index." Center for a New American Security.
• 6.Stanford HAI. (2025). "AI Index Report 2025." (Surge in restricted tokens in C4).
• 7.Anthropic. (2026). "Detecting and preventing distillation." Official blog and reporting on the Senate Banking Committee letter (CNBC, Bloomberg).
• 8.NPR / Copyright Alliance. (2025). "Bartz v. Anthropic $1.5B settlement."
• 9.DataIntelo. (2025). "Dataset Licensing for AI Training Market." ($4.8B→$22.6B, CAGR 18.8%).
• 10.SemiAnalysis. (2025). "DeepSeek Debates." (Distillation cost structure).
• 11.Lockett, W. (2026). "AI Distillation Attacks Are Profoundly Stupid." Medium. (Opening hook).

Related Pebblous Pieces

• 12.Pebblous. (2026). "It Was Provenance, Not Training — The Meta 267TB Lawsuit."
• 13.Pebblous. (2026). "The Economics of Synthetic-Data Contamination."
• 14.Pebblous. (2026). "The RSL Content-Licensing Standard."
• 15.Pebblous. (2026). "Meta, MUSE, Spark, and Closed-Source Models."