Executive Summary

The picture most people carry of model collapse is a gentle downhill slope. Train across generations on synthetic data, diversity narrows and bias accumulates, and performance slips a little across the board. That image of the whole eroding uniformly is intuitive, but it did not match what this study found when it looked at the level of individual skills.

The researchers split GSM8K problems by the skills they require, then compared Llama-3-8B-Instruct's mastery before and after fine-tuning on synthetic data. Collapse, it turned out, did not run evenly in one direction. Skills that were already strong grew stronger, and skills that started weak grew weaker. The paper pins this down not as uniform decay but as a polarization of competence.

The same training splits the fate of each skill. This distinction matters because it changes the diagnosis and the prescription entirely. If collapse were uniform, cleaning out contamination would be enough. But if collapse is rich-get-richer, no amount of filtering keeps the weak skills from staying starved and setting hard. The problem is not cleanliness but distribution.

Why does it tilt in only one direction? The answer lies in how the data gets made. When a model generates the problems and answers it will feed itself, generation flows toward what is more probable. And high probability is exactly where the model is confident, and the areas it is confident in are the ones it already handles well. So generating without constraints pulls problems from the strong skills more easily and more often.

What follows is a self-reinforcing loop. Strong skills receive more training signal and grow stronger, while weak skills generate little data in the first place and grow weaker for want of signal. This is a different kind of problem from contamination, where the statistics of real and synthetic data drift apart. It is not that the data is dirty; it is that the hand drawing the data leans toward the strong side, a sampling bias.

To pin this diagnosis down in numbers rather than by eye, the researchers borrow an unexpected tool: DINA, a cognitive diagnosis model from educational measurement that infers which concepts a student has mastered from their item responses. Tag each problem with the skills it requires, and you can track how the model's per-skill mastery rises and falls generation by generation. It amounts to grading an AI's weaknesses the way a human exam is scored.

The paper does not stop at diagnosis; it offers a fix. The name is KITE (Knowledge-boundary Instruction Tuning via Exploration). The idea is simple. If collapse comes from starving the weak skills, then aim the data generation deliberately at those weak points. It breaks into two steps.

3.1Generate more where the model is weak

First, DINA diagnosis picks out the skills with low mastery. Then generation is pushed to venture outside the model's comfort zone. Simply raising the sampling temperature is deemed not enough, so the method uses token probability rankings to gently lift the lower-ranked choices that would rarely have been drawn. As a result the model produces more problems from its weak areas instead of the ones it already handles well.

3.2Keep only the borderline cases

Generating a lot in the weak areas does not mean using all of it. Candidates are sorted into three bands by uncertainty. If the answer easily converges to one solution, it is already known and there is little to learn. If the answer scatters, it lies beyond the model's knowledge boundary and is close to noise. Only the middle band, where several plausible answers diverge and the case sits right on the edge of what the model knows, is used for training. Competence grows exactly at the boundary between the known and the unknown.

Just how ruthless this selection is shows up in the numbers. In the actual experiments KITE generated 2,000 candidates each generation and kept only 500 for training. Discarding three of every four makes plain that the force preventing collapse lay not in pouring in more but in choosing what to keep.

Which step drives the performance? An ablation that removes each component one at a time gives the answer. Dropping the weakness profiling cost the most, at -1.9pp, while dropping the ranking-based noise injection cost only -0.3pp. Where to aim mattered far more than how diverse the generation was.

The remaining question is whether the fix actually prevented collapse. Across five models and four benchmarks (GSM8K, MMLU-Pro, MATH, GPQA), models trained iteratively with KITE improved rather than declined as generations accumulated. After five generations Qwen-3-4B reached 95.04% on GSM8K, surpassing even the human-data-only run (94.28%), and Llama-3-8B rose from 78.32% to 81.12%, a gain of 2.8pp.

The more striking point is long-run stability. Even in an experiment that pushed iterative training out to nine generations, performance rose monotonically and then saturated gently; it never collapsed or reversed at any generation. Collapse, in other words, was not an inevitability of accumulating generations but a matter of choice, hinging on how the data is selected.

The gains carried over to unfamiliar benchmarks never used in training (OlymMATH, AIME) as well. That is evidence the model did not merely memorize a benchmark distribution to lift its score but grew in reasoning ability itself. Still, the paper does not hide its limits. Every method leaned on a stronger external model (gpt-5-mini) to verify answers, so this is not yet fully closed, purely self-taught learning.

Translate this study into the language of data practice and the question we ask at the preparation stage changes. Until now, the thing we most often asked about synthetic data was how clean it is. Is it uncontaminated? Do its statistics not drift from the real data? But if collapse is polarization, cleanliness alone cannot keep the weak skills from starving. What we have to ask is which weakness this data fills.

If aim is the answer, the next bottleneck appears at once: reliably telling what the weakness actually is. Even this paper leans on skill tags assigned by an LLM rather than a human expert, and the authors acknowledge the limit themselves. If the line of aim is drawn in the wrong place, no amount of sophisticated generation and curation will point at the right target. The very definition of weakness becomes the next question of trust.

The AI-Ready Data that Pebblous has long argued for turns, here, into a demand one layer more specific. It has to cover not only whether the data is clean, but which gap in the model it aims at, and how trustworthy the label that identified that gap is. In the age of aim, the representativeness of the data and the reliability of the labels become axes as important as cleanliness.

• 1.Luo, X., Huang, Y., Guo, K., He, P., Zou, C., Hua, T., & Zhang, X. (2026). "Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning." arXiv:2607.17043.
• 2.de la Torre, J. (2009). "DINA Model and Parameter Estimation: A Didactic." Journal of Educational and Behavioral Statistics, 34(1), 115–130.
• 3.Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022). "Self-Instruct: Aligning Language Models with Self-Generated Instructions." arXiv:2212.10560.