Executive Summary

Key Numbers

Source: Uzunoglu et al., Tracing the Scientific Discovery Capacity of LLMs to Their Training Data, arXiv:2611.03617 (2026)

A monofact is a fact observed exactly once in the training set. The monofact rate divides the number of such facts by the number of training documents, and the paper writes it μ̂. Statistics has used the quantity for a long time. If a sample holds many elements that turned up a single time, the population probably still holds that much more the sample never reached, which is the Good–Turing estimator, a calculation Alan Turing and Irving Good are known to have used on codebreaking work during the Second World War.

What counts as one fact had to be fixed separately in each domain. In materials science the unit is the chemical system, the set of elements a material is built from. Water (H₂O) and hydrogen peroxide (H₂O₂) are different substances, and both belong to the single chemical system H–O. Regulatory genomics has no such natural grouping, so the line was drawn by sequence overlap: a DNA sequence is a monofact when no other example in the training set overlaps it on more than 150 nucleobases.

In 2024 Kalai and Vempala showed that this number gives a floor under the hallucination rate. A well-calibrated model will generate false statements at least as often as once-only facts appear in its training data. Inside that frame an unseen statement is treated as false by default, which leaves discovery nowhere to stand. This paper makes a place for it by splitting the statement space in three: the observed band the model trained on, the candidate band of statements not yet seen but plausibly true, and the arbitrary band beyond it. Discovery lives in the candidate band in the middle.

One qualification comes attached. Kalai and Vempala sorted facts into two kinds. Arbitrary facts, learnable only by observation, such as "The Shining" was published in 1977; and systematic facts, whose truth follows from rules, such as 2.345754 < 8.754. A theory that counts once-only facts bites on the first kind. Discovery, the authors argue, fits neatly into neither, because it leans on abduction, forming hypotheses from plausible guesses, rather than on deduction or induction. The paper reaches for Newton at this point: "No great discovery was ever made without a bold guess." The candidate band is the strip drawn to give that guess somewhere to stand.

Theorem 6 and Theorem 7 stand on that structure. One writes the ceiling on the probability a model can place on discovery in terms of the monofact rate, the other writes the floor. The ceiling is proportional to the rate; the floor takes the same quantity and shaves it by how much of the model's unseen mass leaks into the arbitrary band instead of the candidate band. The gist shrinks to a line. With no once-only facts in the training data, the most a model could discover is lower before it starts.

Testing the theory means the model has to hit answers it genuinely does not know. The team split training from evaluation along time. In materials science they trained on the November 14, 2024 snapshot of the Materials Project and evaluated only on formulas that first appear in the September 25, 2025 snapshot, the materials people registered over the ten months in between. The genomics data is a set of MPRA results released in October 2024, later than either model's knowledge cutoff, so there the evaluation examples were held out at random instead.

The monofact rate r runs from 0.0 to 1.0 in steps of 0.1, eleven levels, in both domains. Source: Section 4.1 of the paper.

Claude Opus 5 produced the stage-two supervision. The teacher model received the ground-truth answer and was asked to write the path to it out of the constraints and structural information present in the input. On the materials side every field revealing element identities or composition was stripped from the input, leaving geometry such as lattice parameters; on the genomics side the true sequence and the edit coordinate were supplied for silent verification only, with instructions not to cite them in the reasoning. That yielded 6,865 traces in materials and 6,000 in genomics, and the appendix records the generation cost at $1,661.63.

The models are gpt-oss-20b and gemma-3-12b-it, with knowledge cutoffs in June and August 2024 respectively, both ahead of the two evaluation sets. Discovery was not counted generously. The ground-truth formula or DNA sequence has to come back character for character to score. Evaluation ran under deterministic decoding, pass@1, alongside pass@16, which samples sixteen times at temperature 1.0 and scores a success if any one of them matches.

The result held across all sixteen combinations of two domains, two model families, two training stages and two sampling settings. Discovery rose monotonically with the monofact rate. Drawing sixteen candidates lifts the absolute discovery rate a great deal and the stage-two reasoning supervision lifts it again, yet the slope between rate and discovery survives both. Re-running the evaluation at low, medium and high reasoning effort left it in place as well. Thinking for longer does not refill facts the data saw only once and the training set never carried.

An objection surfaces around here. Raising the count of chemical systems that appear once also raises the number of distinct chemical systems in the training set. So perhaps what lifted the score was variety rather than rarity. The authors put that objection first and then concede the charge with their own data in the appendix. Training sets with a higher monofact rate did carry a larger fraction of distinct chemical systems. The two quantities moved together inside the main experimental design.

That the two are different quantities is shown with four-example sets. Diversity is the fraction of distinct chemical systems; the monofact rate is the fraction appearing exactly once.

The middle and bottom rows share a diversity value and differ in monofact rate; the top two share a monofact rate and differ in diversity. Source: Table 1 of the paper.

The way to pull them apart turned out to leave the data alone. The training sets stayed exactly as they were, and stage two ran again with every example shown three times and then five times. Formulas, chemical systems and elements are untouched, so the differences in variety between sets survive intact. A chemical system that used to appear once is now seen three or five times, which drives the effective monofact rate toward zero in every set. If variety is the culprit, the trend should remain; if rarity is, the trend should vanish.

It vanished. Discovery that climbed with the monofact rate at one epoch lost its monotonic shape at three epochs and at five. At the high rates, the condition that saw each example once beat the three-epoch and five-epoch conditions on pass@1 and pass@16 alike. Training for longer produced fewer discoveries, which also rules out blaming the one-epoch trend on undertraining. Variety left the list of suspects at that point.

A second suspect remained. If the model's probability estimates grow more accurate at the higher rates, better calibration could account for the extra discoveries. The authors collected formula-level frequencies from the pass@16 generations and measured expected calibration error. Calibration mostly stayed where it was as the rate changed, and the slight drop in error on gpt-oss-20b fell well short of explaining the rise in discovery.

Discovery and hallucination come out of the same pocket. Both are drawn from the share the model never saw in training, and raising the monofact rate means enlarging that pocket. Enlarge it and there is more room for true answers and more room for false ones. The authors pin this down as an exchange rate. When a statement in the candidate band is true half the time, each discovery arrives with one hallucination. Push that probability toward 1 and the ratio improves while the candidates crowd in close to what is already known, so the discoveries turn trivial; push it toward 0 and the candidates grow ambitious while the noise grows with them.

The actual measurement splits the evaluation into three places, and the answer differs by place. Examples the model trained on; examples held out of training but drawn from that same November 2024 snapshot; and the September 2025 examples that appeared in the snapshot ten months later. The first two sit in the distribution the model trained on, and only the last one looks at a target that has moved.

Measured in the materials domain with gpt-oss-20b. Source: Section 4.3 and Appendix C of the paper.

The third row is what makes this a paper about discovery. The authors put the difference down to distribution shift. Compare the element-frequency distributions of the training sets against each evaluation set with Jensen–Shannon divergence and the held-out split from the same snapshot reads 0.0078 while the test set ten months later reads 0.1006. That is close to a thirteenfold gap, and against a target this far away nearly everything the model answers is unobserved anyway. With no total left to grow, what the higher rate buys goes into changing the composition.

Turn that around and a condition appears. Monofacts convert into discovery only when the target really has new facts in it. Apply the same prescription to stamping out variations inside yesterday's distribution and what grows is the wrong answers. Raising the monofact rate is no free improvement, which is also why this paper should not be read as a hymn to rare data.

If monofacts build discovery capacity, will rare data from any field do? The team put the question to the test with materials science as the target and genomics as the outside source. Two conditions went head to head. In the mixed condition, materials data at monofact rate 1.0 forms the floor, 4K genomics examples go on top, and only the genomics rate moves. In the pure cross-domain condition, a model trained on genomics alone is brought over to the materials task as it stands.

In the mixed condition the effect carried over. Materials discovery generally improved as the genomics monofact rate rose, and the model mixed at rate 1.0 edged slightly past the baseline trained on materials data alone on both metrics. Unlike the main experiment, though, the relationship was not monotonic. In the pure cross-domain condition the absolute discovery rate dropped substantially and the relationship with the monofact rate disappeared.

The conclusion is conditional. Once-only facts from another field help discovery when data from the home field sits beside them. They do not manufacture a discovery capacity that works regardless of domain. The lines the authors drew around themselves belong here too. The ceiling and the floor in the theory rest on two values for how often the candidate band is true, and those values are hard to estimate in practice. What the paper claims, then, is the relationship between monofact rate and discovery rather than any specific number on the boundary. The homework left over is equally clear. Nobody has yet measured how the monofact rate changes along the long pipeline from pretraining through posttraining, or whether once-only facts introduced at one stage survive the next.

Past this point the paper is behind us. The received wisdom on the data-preparation side has pointed one way for a long time. Strip the duplicates, and show quality-checked examples often enough to stick. The grounds were sound: duplication costs money, and data that turns up rarely either fails to lodge or causes an incident. The paper's own conclusion gathers prior work along those lines in three strands. Factual recall is unreliable for rare knowledge, data left in a single copy creates privacy and safety exposure, and training on model-generated data erases the tail of the distribution first. All three read the long tail as a risk. Pebblous has worked the trade between deduplication speed and recall before, in the early-stop rule. The question there was how much deduplication to do.

This measurement raises the question that comes before it. What must not be deleted? The first things a cleaning pipeline usually clears out are the examples that appear once and are awkward to verify, and on this paper's map those are precisely the monofacts. If deduplication is not a free gain but a trade against discovery capacity, the first thing to ask is whether anyone has ever priced that trade. The broader turn toward expecting scientific discovery from AI is something Pebblous has covered in AI Changes the Questions, Not the Answers; this paper sits one floor below that, tracing where the capacity comes from back to the statistics of the training data.

Overreach has to be guarded against. The measurement manufactured its rates inside fine-tuning sets of 10K and 20K examples, and the paper does not say the finding carries straight over to a deduplication policy for a whole pretraining corpus. Three questions still look worth asking from a seat where data pipelines get designed.

• On what grounds does our cleaning pipeline throw away the examples that appear only once? Does anyone know where the similarity threshold puts that line?
• Has anyone measured the monofact rate before and after cleaning? The share of items appearing exactly once is a number countable today.
• Do we want the model to repeat what it knows accurately, or to produce answers that do not exist yet? The two goals ask different things of the same data.

Thanks for reading this far. The paper is at arXiv:2611.03617. If your team has actually counted the monofact rate in its own data before and after cleaning, we would like to hear how far the number moved.