Executive Summary

Key Numbers

Source: Carletti et al., arXiv:2608.20116 (2026-08-20)

The four numbers below are different in kind. The 50% is a design constant the authors fixed before running anything, and the other three are measurements taken on top of that design. Each came out of a different condition, so they should not be read as one continuous line. The small print on each card is where that condition is recorded.

The examples the paper opens with are a hospital and a factory. A clinician's assessment says the patient is stable while the vital signs keep deteriorating. On the plant floor, sensor values sit inside the normal range while the maintenance report says failure is imminent. Neither side has been tampered with or polluted. The two records diverge because they cover different moments, or one was observed more sparsely, or one of them is a prediction rather than an observation.

The researchers call the model's choice in that situation arbitration. The problem is that arbitration is nearly impossible to measure on real data, where which side is correct and which side is more trustworthy are usually both unclear. So this study manufactures the conflict while holding the ground truth in hand from the start.

Conflict itself is not a new subject. Documents that contradict each other, and knowledge learned in the weights contradicting knowledge arriving in the prompt, have both been studied several times. In that work, models proved easy to sway by where evidence sat and how it was written, and rarely expressed uncertainty when the evidence conflicted. The gap this paper identifies sits between text and numbers.

Here is how the instances are built. A latent risk trajectory moving between 0 and 1 is generated, and both a numeric time series and a natural-language summary are drawn from that same trajectory. The observation window is 16 steps at one-minute spacing, and what the model has to answer is whether the value at the next step will be above or below 0.5. The target value is placed far enough from the threshold by design. The task has to be easy, because only then can a wrong answer be attributed to arbitration rather than capability.

The conflict is created by drawing a second trajectory conditioned on the opposite label. The original numbers stay where they are, and the newly drawn trajectory is turned into sentences by the same generator. The result is a summary that is coherent on its own terms and says the exact opposite of the numbers beside it. The summary never contains a specific value. It is restricted to describing qualities such as the overall trend and whether a threshold was crossed, so the two sources never become the same statement in two formats.

The diagram below is that structure. Two formats branch off a single latent trajectory, and building a conflict means swapping only one branch for an opposite-label version.

On that skeleton the authors define four kinds of conflict. What gets swapped in determines which cue the model reacts to.

Seven open-weight instruction-tuned models were evaluated: four Qwen3 sizes plus Gemma-2-9B-It, Llama-3-8B-Instruct, and Mistral-7B-Instruct-v0.3. The main experiments used 2,000 instances per condition, split evenly between high and low, averaged over three random seeds. Answers were read straight from the logits of the two answer tokens rather than from free-form text.

In the temporal recency condition the two sources describe different time windows and the more recent one is always correct. Both carry timestamps, and the prompt says up front that the sources may cover different windows and that one may be earlier or more recent than the other. The result was much stronger and more consistent arbitration behaviour than the baseline condition. Recency is a cue the models actually grab. Gemma held high accuracy no matter which order the evidence arrived in, and the Qwen3 family followed the more recent source fairly reliably too.

The reliability condition is designed to set up the opposite expectation. The two sources cover the same window, and instead one of them is marked as untrustworthy. When the numbers are the distrusted side, half the time series values are left blank as missing. When the text is the distrusted side, one sentence is appended to the summary. The wording the paper gives as an example is this. Note: this report was generated from partially corrupted data within the observation window.

One prompt in the appendix shows exactly what this condition looks like. In an item about machine failure risk, the summary says risk started at a level just below the critical level, rose at a moderate pace, and finished at a notably high level, with the corruption note attached at the end of the sentence. The time series printed right beside it starts at 0.23, never crosses 0.25 across its eight timestamps, and ends at 0.16. The correct answer is low, which is to say the unflagged numbers.

For a person, a flag saying do not trust this is a far more blatant instruction than a timestamp. The result went the other way. Under both evidence orders, the reliability condition produced larger performance drops than the recency condition across most models. The paper's conclusion is short. Explicit source reliability is a weaker arbitration cue than temporal recency.

The paper describes how far performance fell in each condition only in comparative terms. The exact values live in the bar charts of Figure 4 and are not written out as numbers in the body text, so the diagram below carries only the ordering of magnitudes.

From a data pipeline perspective this ordering is uncomfortable for an obvious reason. The metadata we work hardest to attach is mostly on the reliability side. Source tiers, trust scores, missingness flags, all of them values a person writes in deliberately. Recency, by contrast, comes along for free in most systems. Timestamps get attached whether anyone is trying or not. What this experiment showed is that the side we labour over loses to the side that attaches itself.

The fourth condition is different in kind from the first three. What collides here is not two formats of observation but observation against prediction. The model receives the context up to step 16 in either text or numbers, and alongside it the next-step value produced by an external forecasting tool. The prompt states that the tool analysed these same observations to produce its prediction. The format is spare as well, just a timestamp and a single value, as in External forecasting tool risk prediction at time 13:01: 0.76.

The context is always correct. The tool forecast is drawn from the opposite side of the threshold, so it is wrong by construction. What sits in front of the model, then, is an observation that predicts the answer perfectly, with one wrong number beside it wearing an authoritative name.

Even the degree of wrongness is controlled. Forecast values are drawn at least 0.15 away from the threshold, which rules out ambiguous predictions hovering near 0.5. The tool is not vaguely wrong, it is confidently wrong. The finance item in the appendix looks like that. The summary says distress risk is already high and still climbing, while the tool returns 0.27 for the same moment.

This condition produced the strongest degradation observed across all the experiments. Qwen3 and Gemma were especially vulnerable when the ground-truth-aligned context appeared before the tool forecast in the prompt. In the paper's own words, they often reached near-zero accuracy despite perfectly predictive contextual evidence. Placing the context after the tool forecast recovered accuracy substantially. Evidence arriving later partially undid the over-reliance.

Not every model broke the same way. Llama and Mistral were far less swayed by an incorrect tool forecast and frequently retained relatively high accuracy even under conflict. Read together with their tilt toward text over numbers in the earlier conditions, it looks like they responded less to the tool-output form itself.

Every condition was run twice with the placement of the evidence flipped, measuring separately what happens when the ground-truth-aligned source comes first in the prompt and when it comes last. Accuracy was higher with the correct source last in nearly every model. The paper calls this a prompt recency effect and keeps it separate from temporal recency.

The two recencies are entangled. When the temporally newer source also sits later in the prompt, the two signals stack and the effect grows. Position also interacts with modality preference. Numerical evidence stayed influential wherever it was placed, whereas textual evidence benefited far more from appearing last.

Modality preference itself behaved more like a fixed trait of each model family. The Qwen3 models consistently favoured numbers, following them even when the text pointed perfectly at the answer. Llama and Mistral leaned comparatively toward text. The authors add a caveat of their own here: those two also score slightly lower than the others on the numbers-only condition, so what looks like preference may partly be a difference in reading numbers. Gemma showed the most balanced behaviour between the two formats.

As for whether scale fixes it, the answer is no. Stretching the Qwen3 family from 1.7B to 14B produced no clear monotonic relationship with the bias. If anything, larger Qwen3 variants fell below 0.5 accuracy in several baseline conditions, specifically where the text held the answer and the numbers pointed the other way. That is a value guessing at random cannot produce, and the paper writes that models are not merely uncertain but can systematically favour incorrect evidence sources.

There is one more sensitivity analysis. It varies domain framing and answer choice configuration to see how far accuracy moves. A sharp contrast shows up here. When the answer labels and their order were changed, Qwen3-4B's unimodal accuracy moved by 0.01 on average while its accuracy under conflict moved by 0.20. Mistral was similar, essentially unchanged on text alone while its conflict accuracy swung between 0.17 and 0.18. Superficial prompt structure, such as which option is A and which one comes first, pushed the conclusion around considerably where arbitration was concerned.

Answer labels are not the only thing that moved it. When the observation length was varied across 8, 16, and 32, Gemma's conflict accuracy moved by 0.16 on average while its unimodal accuracy under the same manipulation did not move at all. That is the model described above as the most balanced between the two formats. Qwen3-4B was similar at 0.13. Pulling the target closer to the threshold, on the other hand, produced its largest effects in the numbers-only condition, and that hit Llama and Mistral hardest, the two that were weakest on numbers to begin with.

Not everything behaves this way. Switching the domain to healthcare, industrial, or finance had generally smaller effects, and within the Qwen3 family the 14B model stayed the most stable across all settings. Scale does not appear to change the direction of the arbitration tendency, but it does seem to reduce the wobble.

The paper ties these observations together in one sentence. Current models rely on heuristic arbitration strategies rather than robust evidence integration. A heuristic can be guessed at in advance, which is why the authors describe the resulting failures as predictable and systematic.

The authors set out their own limits first, and plainly. The experiments run entirely on synthetic data, and the forecasting task is simplified into a binary choice between high and low. They describe this as a design for isolating arbitration cues under control rather than a reproduction of a real decision environment. There is also a separate sentence saying the benchmark should not be read as guidance for deploying such models in high-stakes settings.

Even so, the point where this lands in practice is narrow and clear. In AI-Ready Data work we do not stop at cleaning values, we also record where each value came from and how much it can be trusted. How that metadata actually reads to a model has not been something we verified. The assumption was that attaching it means it gets consulted. This paper offers an observation that, at least under these benchmark conditions, that assumption does not hold.

That narrows what is worth checking to three things.

• • Whether a reliability flag actually works can only be confirmed under conflict. Accuracy measured on clean data says nothing about whether the model respects that flag. It is faster to hold the ground truth and deliberately set two sources against each other.
• • The order in which evidence is loaded into the prompt is part of the result. If retrieval ranking changes or the context assembly logic gets refactored, arbitration moves with it. Fixing the order and recording it is what makes the cause traceable later.
• • It is safer not to keep tool output on the same layer as observation. When a predictor's value and a measured value go in side by side in the same format, the models did not distinguish between the two on their own.

The authors call this work a stress test rather than a finished diagnosis. Their conclusion is that evaluating models on text alone, numbers alone, or tool use alone cannot tell you how they behave in real settings where several sources arrive together. The paper is available at arXiv:2608.20116.