Executive Summary

Key numbers

The first two mark the places where the gap opens widest. The last two show what is left after applying the two remedies people usually reach for.

Source: Zeng et al., arXiv:2610.30631 (2026-10-22)

Clinical guidelines get rewritten every few years. A recommended dose changes, the age at which screening starts moves down, one procedure is swapped for another. ChronoEvo is an exam assembled only out of those rewritten places. The researchers began with 3,411 guideline trajectories, kept the points where the full text on both sides of a revision survived, and settled on the 1,709 items that passed an automated verification step. They come from 308 revision points across 275 longitudinal guideline lineages.

The part of the design worth pausing on is that scoring was split into three columns. Did the answer recognize that a revision had happened. Is the answer clinically correct. Did it fully incorporate the revised recommendation. Most evaluations fold the three into a single score. Pulling them apart made it visible that the middle is empty.

On the surface the answers look as though they absorbed the update. When the researchers matched the wording of each answer against both the old and the new recommendation, 90.2% leaned toward the new one, with an average shift of 0.276. Words had indeed been pulled from the new text. Score the same answers for clinical meaning, though, and close to half fail to carry the revision through. The wording followed; the conclusion did not.

To check whether the automated scoring was being generous, the researchers asked two specialists, one in orthopedic surgery and one in cardiac surgery, to review 300 cases again with the automated labels hidden. The scores they gave were 92.3% for recognition, 79.3% for correctness and 52.0% for full incorporation, with the gap widening to 40.3pp. Of the 144 cases judged to have missed full incorporation, 61 were clinically incorrect as well. The reassurance that an answer can fall short of full incorporation and still land on the right result breaks in roughly four out of ten.

Setting the two scorers side by side shows where they part. Scored automatically, those same 300 cases give 91.3% recognition and 57.3% full incorporation, a gap of 34.0pp, narrower than the 40.3pp the clinicians produced. Agreement makes the location of the difference plainer still. On whether a revision had been recognized, the automated scorer and the clinicians overlapped almost entirely at 0.933. On whether it had been fully incorporated, agreement dropped to 0.705. Machines and people see noticing the same way, and part company on whether it was followed.

One point has to be made plainly. This exam did not measure retrieval. The evidence from the revised guideline was handed to the models in the prompt beforehand. The setup is closer to assuming that retrieval had succeeded perfectly and delivered the correct current document, and the paper itself writes in its limitations that the task of retrieving the right guideline, and the task of choosing the version that fits a particular patient at a particular time, fall outside what it covers. So these numbers do not point toward room left for better retrieval. They point to a problem that begins after retrieval has gone well.

There were three ways of supplying evidence to begin with. One gave the clinical topic alone with no guideline information at all. One gave the prior recommendation only. One gave the evidence from the revised guideline as well. Every number above comes from the last of the three, the most generous seat a model can be offered.

The 1,709 items are sorted into twelve types of change. The largest is a wholly new recommendation added, at 541 items, followed by a changed diagnostic criterion at 177, a changed procedure at 164 and a changed dose or schedule at 152. A changed strength or certainty of a recommendation accounts for 149, a moved threshold for 116, and a widened target population for 104.

The remaining five types are small groups. A changed monitoring plan accounts for 76 items, a recommendation dropped outright for 75, one recommendation swapped for another for 73, a narrowed target population for 65, and items that fit nowhere else for 17. The twelve columns add up to 1,709.

At the two ends of full incorporation by type stand replacement and removal. The swapped type was fully incorporated 41.1% of the time, the lowest of the twelve; the removed type reached 72.0%, the highest. The difference between these two held up after correction for multiple comparisons. The difference across all twelve types, however, was not statistically significant. There is no basis for saying each type behaves distinctly, and there is a basis for saying replacement is particularly low.

This contrast comes with a caveat. The two types hold 73 and 75 items out of 1,709, small groups in which a single proportion swings on a handful of cases. What holds the two numbers up, then, is not the impression that 41.1% and 72.0% sit far apart, but the fact that the distance survived the correction.

What makes replacement hurt is not the incorporation rate itself but the column beside it. In the same type, 94.5% of answers recognized the revision. Nine in ten see precisely that this recommendation has been swapped for another, and more than half of those never move across to the new one. At 53.4pp it is the widest gap among the twelve types.

Counting how often the old recommendation simply stays in the answer makes the pattern sharper. The overall average is 5.9%, and here the differences between types are statistically significant. They remain so after adjusting for conditions such as the interval in years and the length of the recommendation.

The adjusted odds ratio gives how many times more likely an old recommendation is to remain, with the addition of a new recommendation set at 1. For all four types in the table, the difference from that reference category stayed significant after adjustment.

Two remedies come to mind the moment a result like this arrives. Use a bigger model, or instruct it more forcefully that the revision matters. The researchers tested both, and neither closed the gap.

Fourteen models were run, including pairs from the same family that differ only in size. The seven below show the two ends and the middle.

In all fourteen the gap stayed clear of zero, which is to say this is not one model's habit. The effect of scaling is also smaller than expected. Moving Qwen3 from 4B to 32B narrowed the gap by 1.5pp and moving Llama from 8B to 70B narrowed it by 2.2pp, but both confidence intervals include zero, so neither narrowing can be called real. MedGemma, built for medicine, stands in the same place. GPT-5.6 Sol went furthest, recognizing almost every revision and leading the field on full incorporation, and it still left 20.1pp between what it noticed and what it followed.

Across the fourteen models the gap is scattered between 20.1pp at the narrow end and 37.4pp at the wide end, and the scatter does not line up in one direction. Gemma-3-12B, which has the widest gap, is clinically correct 86.0% of the time, the highest outside the three GPT-5.6 models, while its full incorporation sits at 55.1%, third from the bottom of the fourteen. The ability to produce a correct answer and the ability to carry a revision all the way through do not move in step. In an evaluation that scored all three as one, this model sits near the top.

The second remedy is the prompt. The researchers added an explicit instruction that a revision had occurred and that the most recent recommendation should take priority, then ran the same exam again.

Read only the improved rows and the instruction worked. Old recommendations are spoken aloud less often and correctness rises a little. Yet the share of answers that carry the revised recommendation in full is all but unchanged, and the gap actually grew by 0.1pp. What the instruction corrected is a visible symptom, not an ability. It can make a model utter the old sentence less often. It cannot make the model judge how far the new recommendation ought to change the conclusion.

So what the authors ask for in their conclusion comes in four steps. A model has to judge how new evidence changes the prior guideline, preserve the parts that still hold, strip out what has gone stale, and carry the revision through to the final recommendation. It is the claim that exposure to current information is not enough, broken out into things a model has to do. A forecast runs alongside it: the closer model scores come to the ceiling on questions of factual knowledge, the more it matters to measure noticing and following apart from each other.

The researchers narrowed the scope themselves as well. ChronoEvo covers one route by which clinical knowledge gets updated, the revision of a clinical guideline. Drug safety communications, regulatory actions, newly published trial evidence and society consensus statements are not in it yet. They also record as a limitation that much of the item writing and scoring was handed to GPT-5. Because one model's bias can ride on both the writing and the grading, they added cross-evaluation with a different model and the clinician review on top.

Why any of this happens is not something this paper answers. The authors write plainly that they measured model behavior only and did not establish what goes on inside. It is tempting to say the training data has hardened around its cutoff, but the experiment does not support that explanation. What has been established is that even after the evidence arrives, the conclusion does not follow it the whole way.

Clinical guidelines are not the field Pebblous works in. The two columns this exam pulled apart, though, are things that have always traveled together wherever data quality is assessed: swapping a source for its current version, and that swap showing up in the output. We usually manage the first and assume the second follows along. What ChronoEvo shows is that the assumption breaks somewhere between 90.8% and 58.5%.

The way we check carries the same story. Look again at the wording figure from Section 1: 90.2% of answers leaned toward the new recommendation. A checklist that confirms currency through lexical similarity would have recorded this exam as a pass. Close to half of the answers it passed fell short of full incorporation once clinical meaning was scored. When a currency check stops at the level of words, where it went wrong never comes into view.

Carried over into practice, the paper leaves three questions behind.

• Have you ever looked at the two in a pair, to see whether the output genuinely differs once a source has been updated? Unless the old version and the new one are put to the same question side by side, the only thing that changed may be the input
• Do you count only what was added, and never what should have been deleted? The widest gaps opened exactly at replacement and at narrowing
• Does your currency check look at words or at conclusions? The two need not move in the same direction, and in this study they came apart

Of the three, the first costs the least. If both sides of a revision survive in your own material, that material is already an exam paper. It is, in the end, what ChronoEvo did: gather the places that were rewritten, demand the answer that follows the rewriting, and count whether the old answer stayed.

Thank you for reading this far. The figures in this article were checked directly against the paper posted on arXiv, and the account of causes stays inside the range the authors marked as not established. If you have a way of confirming, inside your own organization, that the output really changed after a source was updated, we would like to hear how you do it.

Pebblous Data Communication Team
October 25, 2026

• 1.Zeng, M., Zhan, Z., Yu, K., & Zhang, R. (2026). "Large Language Models Show Systematic Failures in Incorporating Evolving Medical Knowledge." arXiv:2610.30631.