Executive Summary

Key Numbers

The first card is how far coverage carried over into accuracy. The other three are what clustering-based selection gained, where it fell short, and what it cost.

Source: arXiv 2611.07323 (November 5, 2026) · multi-domain figures are OLMo 2 7B fine-tuned on the Tülu 3 mixture

Data selection is the idea that when you cannot afford to train on everything, you take the most valuable examples first, and the stand-in for value that has been used longest is difficulty. The intuition runs that an easy example holds little the model does not already know, so the harder ones carry more information, and in smaller classification models that intuition has held up. To see whether it survives in the fine-tuning of large language models, the researchers gathered five widely used difficulty scores and ran them under identical conditions.

The five measure slightly different things, and they do not cost the same to compute.

The test bed was two models, Llama 3 8B and OLMo 2 7B, together with three generative tasks where fine-tuning lifts the score clearly: CommonsenseQA for everyday reasoning, CosmosQA for reading narrative context, and Social IQa for judging social situations. The five scores and random were compared side by side at selection ratios running from 1 percent of the training data up to 75 percent.

The paper states the result in one sentence. The random baseline stayed competitive across every selected percentage and every dataset, and was often the best performing method of all. Among the difficulty scores, variance of gradients had the highest average accuracy, and even that best showing did not clear random. Cost makes the picture more awkward. Every score except perplexity needs the model trained once on the whole dataset before it yields a number, and PVI needs that done twice. The compute a team hoped to save by cutting the data gets spent earlier, on the cutting.

The hypothesis is simple. Selecting exclusively for hard cases leaves a subset that covers only a limited region of the input data, which is to say its coverage drops. A difficulty score is assigned example by example, so slicing off the top tends to gather examples that resemble one another. Random measures nothing at all, and in exchange the subset it draws comes out shaped like the data it came from.

Testing the hypothesis means turning coverage into a number. The researchers did it this way. First the whole training set is clustered using model embeddings, which gives every example a cluster label. That leaves two distributions over cluster labels, one for the full data and one for the selected subset. Measuring the gap between those two with Jensen-Shannon divergence produces a value where smaller means the subset covered the whole more evenly.

Setting that distance against actual fine-tuning accuracy makes the relationship plain. In all six pairings of two models with three tasks the Spearman correlation came out negative, landing between −0.71 and −0.90. The less a subset departed from the overall distribution, the better it scored. The strongest pairing was Social IQa with OLMo 2 7B, at −0.90.

If coverage is what matters, a method aimed straight at it ought to do well. The one the researchers built adds no new model and no extra training. The training data is passed once through the very model about to be fine-tuned, the last-token vector of the final hidden layer is pulled out, those vectors are clustered with k-means, and one example is drawn at random from each cluster. The number of clusters is the target ratio times the total number of examples, divided by 100. Picking 5 percent out of 100,000 examples means building 5,000 clusters and taking one from each.

Randomness does not disappear from this design. Inside a cluster the draw is still random. All that changes is that the input space is divided into cells beforehand and no cell is allowed to stay empty. The diagram below puts the three approaches side by side on one input space.

Whether the method actually beat random on the single-domain tasks is a separate question, and the paper's answer is cautious. Across the three tasks clustering-based selection stayed on par with random sampling and pulled ahead only occasionally. What did change is the shape of the results. Accuracy almost always rose as the selection ratio rose, and at 25 percent of the data the model landed near the fully fine-tuned one. The researchers repeat their earlier judgment at this point, writing that "random sampling is rarely systematically outperformed by any more principled method tested so far."

On cost the method is light as well. A single forward pass over the training data is all it needs, which puts it an order of magnitude away from difficulty scores that require training the model on everything. The paper reports that performance stayed stable when the cluster count was varied by powers of two. It is still not free. Measured against random, which computes nothing, the overhead is real, as the authors note in their limitations.

Random was strong on the single tasks because a random subset resembles the whole on its own. Once the data comes from more than one source, that changes. The second setting the researchers chose, Tülu 3, is the mixture used to instruction-tune OLMo 2 7B, and it holds many subtasks of different character. In a mixture like that, a random draw can pull one domain in heavily while another barely appears.

The table below is OLMo 2 7B fine-tuned on Tülu 3 and measured on three benchmarks. The figure in parentheses is the share of the full data actually used.

Table 2 in the paper carries five ratios: 1, 5, 10, 15 and 25 percent. Only the 5 and 25 percent columns are reproduced here, and the 1 percent and 10 percent cells appear in the text below. Source: arXiv 2611.07323

On MMLU and BBH, the paper writes, clustering-based selection was the only method that consistently matched or beat the fully fine-tuned score as the ratio moved. The 10 percent cells, left out of the table above, are where that shows best. Clustering-based selection reached 62.7 on MMLU and 49.6 on BBH, above full fine-tuning on both, while random at that ratio came in at 61.6 and 48.5. There are cells like BBH at 5 percent in the table where random cleared the baseline too, but that did not hold once the ratio moved.

The difficulty scores broke down worst at the 1 percent extreme. On BBH, variability dropped to 7.5 and perplexity to 11.5, and on HumanEval perplexity fell to 6.5. In the same cells random scored 47.4 and 30.8, and clustering-based selection 48.6 and 30.5. The less data there is to spend, the more it matters which regions are left empty.

It did not win every cell. At 5 percent on HumanEval, clustering-based selection trailed random, 31.8 against 36.5, and only at 25 percent did it turn the gap around at 40.2. HumanEval is the one benchmark of the three where no selection method reached the 45.9 of full fine-tuning. On an evaluation that hangs on a specific capability, such as code generation, even coverage leaves part of the gap unfilled.

This paper is not alone in doubting difficulty. Another study, presented at ICML this year, reported that no difficulty level is universally optimal and that the optimum moves with the data budget, shifting toward harder data as the budget grows. By the same token, a small budget is better served by easier data, which runs in the same direction as the collapse this paper found in its tightest cells. A report pointing the other way exists too. PPL-Factory, released in July, says that perplexity used with the task and the budget in mind exceeds full-data fine-tuning accuracy by 0.9 on GSM8K and 4.8 on MATH, using 10 percent of the data. The position closer to the evidence is not that difficulty is a useless ingredient, but that what the score measures and how much data will be spent have to be settled together.

What to take from this paper is not the clustering recipe but the way it judged selection rules. The researchers did not argue about what good data is. They built a subset under each rule, trained on it, and compared the scores against a random draw of the same size. Without that control group, all five scores would have gone on being used, each with a plausible story attached.

A team cutting its data to save cost usually settles on one selection rule and bolts it into the pipeline. If that rule belongs to the difficulty family, the question this paper asks comes straight back at them. Three things are worth checking.

• • What is our rule actually measuring — High-loss examples, examples the model found confusing, examples a human labeled as hard: every one of these belongs to the difficulty family. None of those scores tells you which region of the original data the subset came from.
• • Have we ever run a random control — If the subset has never been scored against one drawn at the same ratio with no criterion at all, what the rule contributed has not been measured yet. That comparison is cheaper than building the rule was.
• • Is the data of one kind or many — Within a single domain, random is already even enough. Coverage earns its keep on a mixture of sources.

This blog has watched a sophisticated method fail to beat random once before. Hunting outliers in large production tables, a random sample beat the clever sampling strategies. The setting differs and so does the mechanism, but the lesson overlaps. Building a rule carefully and confirming the rule helps are two separate jobs, and skipping the second leaves the cost higher with nobody the wiser.

The control group is what Pebblous keeps in view while shaping data into a state AI can use directly. Saying you will measure quality is the same as saying you have decided what to measure it against. With nothing to compare against, there is no telling whether a rule got better or worse.

Thanks for reading this far. The figures here were checked directly against the body of the paper posted to arXiv, and where the paper held back we have held back too. We would be glad to hear which rule you use when you cut a training set, and whether you have ever set it beside random.

Pebblous Data Communication Team
November 9, 2026

• 1.Pandia, L., Misra, K., & Ettinger, A. (2026). "Rethinking Data Selection: The Importance of Coverage over Difficulty in LLM Fine-Tuning." arXiv:2611.07323, November 5, 2026.
• 2.Lambert, N. et al. (2025). "Tülu 3: Pushing Frontiers in Open Language Model Post-Training." arXiv:2411.15124.
• 3.Liu, S., Chen, T., Li, X., Wang, Y., & Zhang, J. (2026). "Data Difficulty and the Generalization–Extrapolation Tradeoff in LLM Fine-Tuning." arXiv:2605.12906, ICML 2026.
• 4.Zhang, H., & Gross, W. J. (2026). "PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning." arXiv:2607.18199, July 20, 2026.