Executive Summary

Four numbers give the size and the character of the gap first. The drop in the worst combination, that same drop at the level of individual words, the change when the parameter count went up, and the number of settings that kept the direction intact.

The sense that AI writing all sounds alike is old news. Turning that impression into a test requires a reference point, and most diversity metrics so far have compared generations only against each other: take ten answers a model gave to one prompt and measure how different they are. That design cannot rule on whether a model is narrow, because it never says narrow compared with what. A preprint posted to arXiv on September 2, 2026 by Youqi Wu and Farzan Farnia of the Department of Computer Science and Engineering at the Chinese University of Hong Kong puts the training corpus in that empty slot.

The design is plain. Pull a paragraph from the corpus the model trained on and cut its opening off as a prefix. The corpus already holds the tokens that actually followed that prefix, and feeding the same prefix to the model produces a continuation of its own. Vectorize both sets of continuations the same way, compute conditional entropy on each, and set the two side by side. The unit of measurement in the main experiment is a 3-token prefix and a 2-token continuation, over 20,000 prefixes. This is not an experiment on paragraph-length or document-length responses, a constraint that holds for everything below.

Measuring conditionally is the heart of the method. No collection of multiple human reference answers per prefix is required. Across the paired input and output samples, subtracting the input-side entropy from the joint entropy leaves only the continuation-side variation that the prefix does not explain. Written as von Neumann entropy applied to kernel matrices built from embeddings, it looks like this.

What the paper puts in its tables is not that entropy but its exponential, $\exp(H_{\mathrm{vN}})$. The authors say they report this form because it can be read as the effective number of conditionally distinguishable outputs. A value of 338.58 is better read as a count than as a score: under the same conditions, roughly 339 mutually distinguishable continuations. Every table number in this article is a count of that kind.

1.1The Baseline Is Not the Whole Corpus

So what exactly sits on the training-data side of the comparison? Appendix D.1 gives the procedure. Documents are split on blank lines into paragraph candidates, only paragraphs with at least 500 characters and at least three sentence-ending marks are kept, and candidates are sampled uniformly within each document until the first valid paragraph is taken. So the baseline is not the corpus in full but a subset filtered down to prose paragraphs above a length threshold. Code snippets, tables and short lists never enter it. Whether that filter widened or narrowed the gap is not something the paper examines.

The corpora themselves are Ai2's Dolma and EleutherAI's The Pile. Their composition and licensing are laid out in our A Guide to Open Datasets for LLM Training, so we will not repeat them here.

1.2The Same Group Built the Metric and Used It

The metric is not new to this paper, and §2 says so directly. The $\exp(H_{\mathrm{vN}})$ reported here is the same quantity as the Conditional Vendi Score introduced by Jalali, Ospanov, Gohari and Farnia at AISTATS 2026. The earlier study that first measured this kind of gap on unconditional generative models also came out of Farnia's group. Both pieces of prior work share a corresponding author with this paper. This is not an independent third party validating someone else's metric. It is the first application of the authors' own conditional metric to a training-corpus comparison.

Appendix A lists earlier reports of the same phenomenon on its own account. Instruction formats have been shown to pull outputs toward semantic similarity, image generators have been shown to cover only part of the diversity in their training distribution, and captioning models have been shown to produce flatter, narrower sentences than people do. What this paper adds is the procedure: it puts the corpus the model actually ate in the position of the comparison group.

The paper is still a v1 preprint. It carries no venue information, the experiments ran on two RTX-4090 cards, and we could not find a link to reproduction code in it. Every table cited below was read directly from the body and appendices of that preprint.

The models under test are OLMo-1B, Pythia-1B and GPT-Neo-1.3B, each run under three decoding strategies: greedy, which takes the single most likely token at every step; nucleus sampling, which draws from the smallest token set whose cumulative probability exceeds p = 0.9; and ancestral sampling, which draws from the full next-token distribution. The values below are the effective counts described above, and the figure after each ± is the standard deviation over five independent evaluations. The relative gap column on the right does not appear in the paper; we computed it against the training-data value in the same block.

▲ 3-token prefix / 2-token continuation, 20,000 samples, Qwen3-Embedding with a Gaussian kernel. Values from Table 1 of the paper; relative gaps calculated by Pebblous.

Reading the table across the rows shows what a practitioner should take from this paper. The number most likely to be quoted is OLMo's 35.4% under greedy decoding, and that figure is the worst single combination on all three axes at once: model, decoding and length. Change nothing but the decoding, to ancestral sampling, and Pythia falls to 7.9% inside the same table. That does not mean the gap went away. It does mean that a diversity number published without its generation settings cannot be compared with any other. For reference, the default in Hugging Face transformers generate() is greedy, and the default temperature on commercial APIs is 1.0. A pipeline left at defaults and a pipeline with tuned sampling sit in different places on this table even when the model is the same.

2.1Changing the Lengths Takes the Gap Down to the 3% Range

The paper anticipated the objection that three tokens in and two tokens out is a short unit. §6.3 extends prefixes to 8, 16 and 32 tokens and continuations to 16, 32 and 64, for twenty-seven additional combinations. Checking all of them, the direction with training data on top holds in all twenty-seven. No cell reverses. The size, however, changes a great deal. Pulling out the relative gaps under greedy decoding gives the following.

▲ Four of the twenty-seven rows in Table 2 of the paper. Relative gaps calculated by Pebblous.

Two tendencies are layered here. Hold the prefix fixed and lengthen the continuation, and the gap grows, because a longer continuation leaves more room for valid variation and the model covers less of it. Hold the continuation fixed and lengthen the prefix, and the gap shrinks, because stronger conditioning narrows the range of plausible continuations in the first place. Even the ablation that lengthens the prefix by exactly one token, from three to four, takes OLMo's greedy gap from 35.4% down to 20.9%. Carrying the 35% figure over to long-form text is therefore an overstatement. The paper's own closing line on this reads:

"Nevertheless, no configuration eliminates the gap."

Appendix D.1

2.2The Conclusion Held Under a Different Embedding and Kernel

The appendix also answers the question of whether this is an artifact of a well-chosen kernel. Swapping the embedding to T5 and the kernel to a third-degree polynomial and to cosine similarity yields three tables in which not a single setting reverses the ordering, and all three cover all three models. The absolute scale shifts substantially, but training data on top and greedy at the bottom survives. In the authors' terms, the qualitative conclusion does not change, so the gap is not a product of one embedding or one kernel family.

Beyond embeddings and kernels, the unit used to chop up a sequence was varied too. The twenty-seven combinations above represent a long sequence by concatenating the embeddings of 2-token chunks; changing that chunk to 1 token and to 4 tokens preserves both the training-data-on-top structure and the ordering of all four groups. The scope of this check is narrower than the three tables above, though. It ran on GPT-Neo alone, in the long-sequence setting.

The appendix reports the contrast with surface-level diversity as well. Under stochastic decoding, Distinct-n scores, which count the proportion of unique n-grams, come fairly close to the training data. On the same samples, the conditional entropy gap stays positive. Producing more varied word combinations at the surface and producing more mutually distinguishable continuations in semantic space are not the same event, which is exactly why a kernel-based metric earns its place alongside the surface one.

Finally, §6.4 supplies the material for asking whether this is a language-model phenomenon at all. Four class-conditioned ImageNet generators and four text-conditioned generators given MS-COCO captions show a gap in the same direction. At 20,000 MS-COCO samples, the reference data scores 40.52 while SDXL scores 30.25 and PixArt-Σ 25.25. The comparison group there is a reference dataset rather than the model's training corpus, so the claim of setting output beside training data applies only to the three language-model families above. Even with that qualifier attached, the paper's judgment stands: this is not the problem of one architecture or one training recipe.

2.3How Many You Measure Also Changes the Answer

Every value in the tables above was measured on 20,000 samples. That condition moves the size of the gap too. The paper repeats the same measurement at eight points from 2,500 up to 20,000, and the gap widens in one direction as the sample grows. The three main models behave the same way. OLMo's greedy gap is 30.0% at 10,000 samples and 35.4% at 20,000; Pythia goes from 27.8% to 32.8% and GPT-Neo from 26.1% to 31.0%. The full curve across all eight points appears in the appendix tables for the larger models.

▲ Three of the eight sample sizes reported in Tables 11 and 12 of the paper. Relative gaps under greedy decoding; percentages calculated by Pebblous.

The paper locates the cause on the training-data side of the ledger. As the sample grows, the training value climbs faster than the generated value. Add more examples and the corpus keeps producing continuations it has not produced yet, while the model circles inside the range it has already used. So the size of the gap depends on how much you measure as well. Take the same model, the same decoding and the same lengths, and 2,500 samples put it under 20% while 20,000 put it at 35%. To the earlier requirement of recording the decoding setting, add the sample size.

A number that comes out of a kernel matrix is hard to check on its own. It happened in embedding space, where there is nothing for a person to look at. Appendix E.9 takes that weakness head on and asks the same question again without the metric, using words anyone can count.

The audit starts from three fixed word lists: 192 countries, 80 sports and 118 occupations. Count how often those terms appear in each group's continuations, by exact case-insensitive match. Normalize the counts into a categorical distribution and two figures fall out. One is the number of distinct terms that actually appeared. The other is the exponential of that distribution's Shannon entropy, which says how many terms it amounts to if all of them had been used equally often. This audit was run on OLMo alone, at 20,000 samples.

▲ Table 10 of the paper. OLMo only, 20,000 samples. The effective count is 2 raised to the Shannon entropy, in bits, of the normalized term-frequency distribution.

All three effective-count columns line up in the same order: training data, ancestral, nucleus, greedy, which is exactly the ordering the kernel metric produced. The count of distinct terms is not nearly as tidy. In sports, nucleus uses 25 terms against ancestral's 24, and in occupations it leads 69 to 68. Variety and evenness are different questions. On the country list, greedy did use 82 terms, but it did not use those 82 evenly, and the effective count drops to 26.9.

The paper shows what that concentration looks like. Among country mentions under greedy decoding, 20.0% went to "UK" and 19.8% to "United States". In the training paragraphs built from the same prefixes, the two terms took 8.4% and 4.3%. The denominator needs pinning down here. It is not all generated text but the total mentions falling on the 192-term country lexicon. Reading this as 40% of the output being two words is wrong. It says that when the occasion to name a country came up, four out of ten of those mentions went to two countries.

Distributions piling onto particular values is not unique to text generation. The way rare combinations disappear first when demographic axes intersect in synthetic persona data is something we covered in Synthetic Respondents Given Two Identities Used Only One. The subject and the metric differ, but the shape is from the same family: the total holds up while the edges thin out.

Every experiment so far runs on models of roughly a billion parameters. If the shortfall were a matter of capacity, of the model lacking room to hold the corpus's range, then a bigger model should close the gap. Appendix E.10 takes that question, adding OLMo-3-7B and extending Pythia to 2.8B and 6.9B under the same settings.

Starting with OLMo-3-7B, the relative reductions the paper states for itself at 20,000 samples are about 33.0% for greedy, about 16.2% for nucleus and about 12.0% for ancestral. Set against the 1B model's 35.4%, 15.1% and 12.2%, the two nearly coincide. OLMo-1B and OLMo-3-7B differ by more than parameter count, though. The training corpus moved from Dolma to Dolma 3, and the training tokens went from 3 trillion to 5.93 trillion. This is not a comparison that isolates the effect of size.

Pythia is where that condition is met. The 1B, 2.8B and 6.9B models all trained on The Pile for one epoch, roughly 300 billion tokens. It is a rare design in which the data variables stay fixed and only the parameter count moves. One further condition makes the comparison fully clean. Pythia-1B's training baseline is 262.55 while the baseline for 2.8B and 6.9B is 265.87. The same corpus filtered by the same procedure still yields a different baseline value in a different table, and the paper does not say where that difference comes from. But 2.8B and 6.9B carry an identical training baseline, to the decimal, inside one table. That is why only those two are set side by side here.

▲ Table 12 of the paper, 20,000 samples. Relative gaps in parentheses and the change row calculated by Pebblous. The measured parameter counts are 2,909,430,000 and 6,991,520,000, a factor of about 2.4.

While the parameter count went up by a factor of 2.4, the greedy relative gap moved from 35.1% to 34.5%. That is 0.6 percentage points. The stochastic decodings improved by a little over a point each, so the direction is toward improvement, but this is far too slow to call it a closing gap. This is not a result in which growth makes things worse; nucleus and ancestral clearly improved. The accurate summary is almost unchanged, and the paper's own sentence stops at the point that increasing model size alone does not remove the gap.

Every model in the experiment is at least one generation behind by today's standards. GPT-Neo is a 2021 model whose repository states that maintenance stopped in August of that year. Pythia arrived in 2023 and OLMo-1B in February 2024. Not one instruction-tuned or RLHF-trained commercial model appears in this experiment. The criterion for picking them was, from the start, whether the training corpus was public, and that criterion leads to the last argument in this article.

Once the gap is measured, the next question is what can be done about it. Retraining is not an answer for most organizations, and the paper does not recommend that route. It proposes working on output that has already been generated instead: produce ten candidates per prefix, then redistribute the weights on those candidates under a conditional entropy constraint. Model parameters are untouched and no new output is generated.

Why that redistribution is a solvable problem is set out in §4. What the paper actually proves is that the kernel-based conditional von Neumann entropy is concave as a functional of the joint distribution. Concavity means the distributions satisfying an entropy-above-some-value constraint form a convex set, and pushing candidate weights into that set becomes a convex optimization. The computation itself is solved by mirror descent over a product of simplices, one per prefix. The proof stops there. The diversity gaps in the earlier sections are measured values. The mathematics guarantees the procedure for reducing the gap; the existence of the gap rests on measurement.

5.1Raising the Temperature Is Not Enough

The easiest alternative is to raise the sampling temperature, and the paper tested it seriously. Searching for the temperature at which OLMo's conditional entropy matches the training data gave roughly T = 1.9. The entropy number, then, can be matched. At that temperature two quality measures both got worse. One is precision, the share of generated probability mass falling inside the estimated support of the training continuations. The other is external conditional negative log likelihood, scored by an independent language model on the same prefix.

▲ Table 8 of the paper, on OLMo. The temperature of 1.9 is tuned to match the conditional entropy of the training data.

The point of the table is that matching one number, the entropy, and moving to the closest distribution that satisfies the constraint are different operations. One line needs care, though: the projection's NLL of 3.713 sits below the training data's 4.679. That does not mean it beats human writing. It means an external language model finds it more predictable.

5.2Where the Weight Moves

What the reweighting actually does shows up most clearly in a controlled probe. The paper takes two prompts, samples 1,000 continuations for each, and applies the projection. On "A vibrant city in Northern America is ___", the mass on Toronto fell from 0.206 to 0.177, Vancouver from 0.198 to 0.137, and New York from 0.096 to 0.068. Miami picked up that room, going from 0.018 to 0.046, Denver from 0.020 to 0.035, and Philadelphia from 0.010 to 0.022. The effective number of distinct continuations went from 13.29 to 17.34. On the second prompt, which asks for a renowned celebrity, it rose from 53.46 to 81.25.

"it reduces excessive concentration on dominant generated answers and increases the representation of plausible alternatives already present in the candidate pool, rather than promoting low-quality or implausible outputs."

Appendix E.8

The same probe reports the shift in another unit. The discrete entropy of the city prompt rises from 2.587 to 2.853 and that of the celebrity prompt from 3.979 to 4.398, which are the effective counts above before exponentiation. Nothing enters the pool from outside it, which is why reweighting adds no new fact-checking burden: the projection never leaves the candidate set it was handed.

5.3How Much Is Recovered Depends on the Metric

The size of the effect comes out quite differently on the two metrics. At 20,000 samples, conditional von Neumann entropy rises 2.8%, 3.0% and 3.7% across the three models. Measuring the same experiment with conditional RKE, the order-2 form, gives 18.2%, 20.7% and 20.3%. The two must not be averaged or merged into a single effect size, because they are different quantities. Neither reading supports a claim that the gap was filled, and the paper too stops at being able to raise conditional diversity after the fact. One more caveat: the pre-reweighting values in this experiment come from a separate sample built by drawing ten candidates per prefix, so they are not numbers that can be set beside the training baselines in the earlier sections.

Usefulness on a real task was tested too. In MBR decoding, where one output is selected from several candidates, the variant that swaps in these projected candidate weights had the highest conditional VNE and Distinct-2 and the lowest Self-BLEU-2 in all three settings. The quality side needs a careful reading. The comparator is a length-normalized model-based MBR, and over 50 test inputs with 2,000 paired bootstrap resamples, the 95% confidence intervals for the ROUGE-L and BERTScore differences against that method contained zero in all three settings. That means no difference was established, not that there is none. On XSum, in fact, the ROUGE-L value itself falls from that method's 0.350 to 0.331. Fifty inputs is a thin basis for saying quality was preserved.

One lever that acts during generation rather than after it has also been tried. Feeding the same conditional entropy in as a guidance signal for diffusion sampling takes SDXL from 30.25 to 32.27. That still falls short of the reference data's 40.52, so it narrows the gap and stops there. And this experiment was run only on the image side. The paper has no result for applying the same signal at a language model's decoding step.

What is missing is less the method than the habit. The authors' own Conditional Vendi Score implementation sits in a personal repository with four stars and no PyPI release. The unconditional Vendi Score implementation is at roughly 164 stars. HELM, the widely used model evaluation framework, names seven core metrics: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. Diversity is not among them. The instrument has been published; the practice of using it has not arrived.

Data quality checks mostly look at individual records: is the value missing, does it match the format, is the label correct. What this paper takes on is the question that survives a dataset passing all of those. Every record can be correct while the distribution they add up to has narrowed against the original, and how far it narrowed is invisible to record-level checks. That is why metrics for distribution and coverage are needed, and it is the same family of measurement as the axes Pebblous has worked on in DataClinic.

A pipeline that feeds generated data back into training needs one more check on the list: measure the conditional diversity gap against the source corpus before the data goes in. The way synthetic data holds average capability while shaving the edges first is something we covered in Training AI on Synthetic Data Polarizes Its Skills, and the lexical audit in this paper ends up showing the same shape. The total is comparable and the spread is not.

The minimum procedure an organization can adopt today comes down to four steps. Choose the corpus that will serve as the reference. Build paired samples from both sides under identical conditions, using prefixes drawn from it. Record which decoding settings produced how many samples. Then measure the two side by side. The third step is the one that gets waved through. As we saw above, changing only the decoding moves the same model's gap between the 35% range and the 8% range, and holding decoding fixed while changing only the sample size moves it between the 20% range and the 35% range. A diversity number recorded without its settings cannot be compared with the same measurement next quarter. And after the measurement, that some of the loss is recoverable, and that recovering it takes candidate reweighting rather than retraining, is the rest of what this paper hands to practice. What accuracy actually measures is laid out in The Mathematics of Data Quality.

The last item is the heaviest. This diagnosis only begins if the corpus the model ate is available. Open weights are not enough, because the procedure requires drawing the same prefix back out of the corpus and reading what followed it there. The paper's limitations paragraph states the situation plainly.

"for many widely used LLMs and generative AI models, the training data are not publicly accessible, making the analysis more difficult but potentially valuable."

§7

OLMo, Pythia and GPT-Neo were chosen on that condition, not on performance. And the families that clearly satisfy it can still be counted on one hand. Ai2's OLMo line and EleutherAI's The Pile line are effectively the whole list. The tendency for the strongest-performing models to be the least open is something we already documented in The Most Capable AI Discloses the Least. Openness, then, is not a transparency slogan but a precondition for measuring quality. A model that has not opened its corpus cannot sit for this test, and a model that cannot sit for the test is one where nobody can say whether it narrowed. How far a contraction in output range spreads beyond the quality of a single service is covered in AI Made Individual Scientists Stronger, Made Science Itself Narrower.

The sentence from this paper that lasts longest is not in a table. It is in the limitations paragraph explaining why only three families were tested. Measuring a model's diversity requires a baseline, and the baseline is the corpus that model actually ate. Without it the diagnosis never starts. Data that has only passed an accuracy check goes on to cause quiet trouble in the next training round. Correct and evenly spread are separate tests.

Every figure in this article was taken from the full text and appendices of arXiv:2609.02275 v1, checked table by table against the source. The relative-gap percentages do not appear in the paper; unless noted otherwise, Pebblous calculated them against the training baseline in each table. The prior-work entries are cited indirectly through the paper's own bibliography and were not opened and verified individually.

Primary source

• 1.Youqi Wu & Farzan Farnia, "Do Large Language Models Capture the Diversity in their Training Data?" arXiv:2609.02275v1 [cs.CL] (Sept. 2, 2026). Department of Computer Science and Engineering, The Chinese University of Hong Kong. CC BY 4.0. Link

Lineage of the metric

• 2.Mohammad Jalali, Azim Ospanov, Amin Gohari & Farzan Farnia, "Conditional Vendi Score: An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models," AISTATS 2026 — the source of the metric this paper uses. Code: mjalali/conditional-vendi
• 3.Farzan Farnia, Mohammad Jalali & Azim Ospanov, measurement of the von Neumann entropy gap in unconditional generative models (ICLR 2026 workshop) — the earlier measurement this paper extends to the conditional case.
• 4.Dan Friedman & Adji Bousso Dieng, "The Vendi Score: A Diversity Evaluation Metric for Machine Learning." Code: vertaix/Vendi-Score

Prior reports (via Appendix A of the paper)

• 5.Yun et al., diversity collapse in LLMs — a report that formatting and instruction structure induce semantically similar outputs.
• 6.Dombrowski et al., a report that image generators cover only part of the diversity in the training distribution.
• 7.Wang & Chan, a report that captioning models are less diverse than human writers.

Models and datasets

• 8.Dirk Groeneveld et al., "OLMo: Accelerating the Science of Language Models," arXiv:2402.00838. Checkpoint allenai/OLMo-1B-hf
• 9.Stella Biderman et al., "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling." Checkpoints EleutherAI/pythia-1b, pythia-2.8b, pythia-6.9b — all three sizes trained on The Pile for one epoch, roughly 300 billion tokens.
• 10.Luca Soldaini et al., "Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research." ODC-BY. The Dolma 3 mix is allenai/dolma3_mix-6T
• 11.Leo Gao et al., "The Pile: An 800GB Dataset of Diverse Text for Language Modeling," arXiv:2101.00027. Link
• 12.EleutherAI, the GPT-Neo repository — the README states that maintenance stopped in August 2021. Checkpoint EleutherAI/gpt-neo-1.3B

Related Pebblous publications

• 13.Pebblous, "A Guide to Open Datasets for LLM Training." Link
• 14.Pebblous, "When AI Eats AI, Human Data Gets More Expensive." Link
• 15.Pebblous, "Synthetic Respondents Given Two Identities Used Only One." Link
• 16.Pebblous, "Training AI on Synthetic Data Polarizes Its Skills." Link
• 17.Pebblous, "The Mathematics of Data Quality." Link
• 18.Pebblous, "The Most Capable AI Discloses the Least." Link
• 19.Pebblous, "AI Made Individual Scientists Stronger, Made Science Itself Narrower." Link