Executive Summary

Key numbers

The four figures below rest on different denominators. The first two measure the same phenomenon with a narrow and a wide yardstick; the last two point to where the effect comes from and what happens on the opposite side.

Source: arXiv:2608.14375 §5 and Table 9, Argonne National Laboratory (2026-08-14)

What a selection rule really needs to know is not whether the answer a message proposed was correct. It is whether feeding that message into the integration step moved the reasoning that followed closer to the right answer. The paper pulls those two apart and calls the second one trajectory value. Filters in production look only at the first and use it as a proxy for the second, and this paper measures how often that proxy misses.

The observation that you can learn from what is wrong is not itself new. There is prior work showing that failed trajectories carry training signal, and reports that in-context learning with deliberately incorrect demonstrations still improved performance. What the paper cites from that line of work, though, is mostly aggregate. It tells you how the average moved once wrong material was mixed in, not which message helped in which context. Trajectory value asks that same question one message at a time.

To measure the gap, the messages have to hold still. Collaboration structures built on debate, revision or winner selection can raise final accuracy, but they rewrite or delete the very messages being measured. The protocol the team designed, DHD (Diverse Hypothesis Deliberation), is not a device for maximizing accuracy. It is an instrument. A recruiter assigns five complementary roles per problem, and five hypothesizers each write one message containing their reasoning and a proposed answer, without seeing a peer's message, evaluator feedback or the ground truth. That pool of five is generated once and never regenerated.

The measurement itself is two runs of the integrator. One run sees the whole pool; the other hides a single target message. The problem, the model, the prompt template, the ordering of the remaining messages and the grading procedure all stay fixed. If only the run containing the message is correct, the message is helpful; if only the run without it is correct, it is harmful; if both agree, it is neutral. The integrator is never told to vote or to copy the most common answer, and the evaluator sees only the submitted answer, never the reasoning behind it.

The case in the diagram comes from a real trace in the paper. One Omni-MATH-2 problem asks which values of L make a perfect square out of the sum of the floor of n divided by 2, taken from n equals 1 through L. The answer is L equals 1, or L even. The fifth message split L into even and odd cases and correctly showed that the even branch always yields a perfect square while the odd branch produces the product of two consecutive integers, which cannot be one. It then dropped the boundary case of L equals 1 and submitted all even L as its answer. Integrate with that message and the correct answer appears; integrate without it and only the even case survives. The proposed answer was wrong, yet its parity split gave the rest of the pool the opening to recover L equals 1.

Four in ten is not a quirk of one benchmark. The experiment spans five math and science benchmarks: Omni-MATH-2 with 4,181 competition math problems, JEEBench with 515 Indian engineering entrance problems, SciBench with 580 college science problems, a text-only slice of LAB-Bench with 741 items, and MaScQA with 649 materials science problems. The models are two open-weight families, gpt-oss-120b and gemma-4-31B-it. The headline analysis covers 91,740 valid messages for the first model and 83,020 for the second.

The result first. Among the events where a message with a wrong proposed answer actually flipped the final verdict, 41.9% (gpt-oss-120b) and 45.3% (gemma-4-31B-it) fell on the helpful side. Resampling at the problem level puts the 95% intervals at 39.5–44.3% and 42.6–48.1%. The spread across benchmarks is wide, with Omni-MATH-2 highest at 44.1% and MaScQA lowest at 26.9%. Being wrong does raise the odds that a message does damage, but it does not settle whether that message helps or hurts.

The same input producing a different result could have manufactured these numbers on its own. The team measured that noise first, repeating the comparison on 1,000 problems per model. Two calls with identical inputs disagreed on final correctness 7.3% of the time for gpt-oss-120b and 2.1% for gemma-4-31B-it. They then swapped the full-pool and leave-one-out labels within matched replay blocks to build 5,000 null datasets, and not one of them reached the observed number of effects (p=0.0002).

The grading side got its own audit. Regrading 16,724 saved submissions with three different evaluator models produced 94.2–96.6% agreement on individual answers. Narrow that to the strict criterion where the proposed answer, the full-pool outcome and the leave-one-out outcome must all agree, and the rate falls to 71.6–78.6% for gpt-oss-120b and 82.4–88.7% for gemma-4-31B-it, which means the compound label does move somewhat with the evaluator. Even so, every evaluator recovered both the wrong-but-helpful cell and the correct-but-harmful cell.

Keeping the number honest means showing the other side of it too. Set the denominator to all wrong messages rather than flipped events, and helpful flips come to 6.3% and 3.2%. Applying the Benjamini-Hochberg procedure for multiple comparisons at the problem level left 11 reproducible cases, all in gemma-4-31B-it and scattered across all five benchmarks, with none surviving for gpt-oss-120b. The claim is not that wrong logs are always treasure. It is that a wrong label alone cannot tell you what to throw away.

The opposite direction shows up in the same experiment. A MaScQA problem asks for the maximum fraction of sites that can be cross-linked when 10 grams of sulfur, atomic mass 32, is added at a one-to-one ratio to 50 grams of natural rubber with a repeat unit mass of 68. The answer is 42–43%. The fourth message converted both masses to moles and divided correctly, proposing roughly 43%. Alongside that, it added a caveat suggesting a second look at whether one sulfur atom bridges two sites, and that caveat propped up another message in the pool proposing 85%. With this message present the integrator's final answer was 85%; without it, 42.5%.

None of this surfaces in the averages. Moving from solving alone with no messages to seeing all five shifted benchmark-average accuracy by 1.6 points for gpt-oss-120b and 0.3 points for gemma-4-31B-it, with the direction varying by benchmark, because helpful and harmful cancel out in the mix. Meanwhile, in the verified gpt-oss-120b matrix, the share of problems where at least one of the five messages proposed the correct answer ran 76.0–95.4% depending on the benchmark, which sits 4.2–36.2 points above actual final accuracy. The mere presence of a correct candidate in the pool does not prove usable reasoning was there with it. It does establish that a final score alone cannot distinguish failing to generate a good candidate from failing to use one.

More information was not reliably better either. Problems the same integrator solved with a single message and missed with all five came to 9.8% for gpt-oss-120b and 6.9% for gemma-4-31B-it. The reverse direction, missed with one and solved with five, was 0.5% and 0.2%. That gap is the volume of cases where a path to the correct answer sat in the pool and was buried during integration.

This is not a majority-vote problem, and the paper checks that too. Among complete gemma-4-31B-it pools, the majority opinion was correct in only 36.9% of cases while overall accuracy was 78.7%. The integrator is not counting ballots; it is verifying and synthesizing messages, and that is exactly where one message gets revived and another gets crowded out. A message's value therefore does not live inside the message. It is defined only within the pool and the integrator it sits in.

If a wrong message helped, where did the help come from? It could be an answer that happened to point the right way, or the reasoning that preceded it. The researchers separated the two by masking either the reasoning or the proposed answer while leaving the message's position and remaining fields untouched. The targets were 22 gemma-4-31B-it messages selected by criteria fixed before the outcomes were observed, and the integrator success rate across the 10 confirmed helpful anchors split as follows.

Erasing the wrong answer cost 18 percentage points. Erasing the reasoning cost 38. The 44% left after masking the reasoning is effectively the same as the 46% from swapping the message for filler, which is to say a message reduced to its answer did no more work than text occupying the slot. The paper's own wording is careful here: it says the diagnostic is more consistent with the reasoning account and leaves the reason the complete message was strongest as an open question. It also attaches the caveat that this is a ten-case diagnostic and not a prevalence estimate.

The team also checked whether the label is good for anything. Using four repeated blocks of the same problem, they let a policy choose between keeping everything and dropping one message, then scored that choice on a fifth block held out of the selection. Accuracy rose 1.68 points for gpt-oss-120b and 2.61 points for gemma-4-31B-it. A comparison policy that looked only at whether the proposed answer was correct managed 0.94 and 1.00 points. This was measured under repetition of the same problem, so it is not a policy that transfers as-is to unseen problems.

The selection rules teams actually use were benchmarked on the same records. Picking one message at random, following answer consensus and taking the message with the highest self-reported confidence produced accuracies that did not differ much from each other, and on gemma-4-31B-it neither consensus nor confidence beat random. Selecting with an oracle that knows correctness in advance did clearly better, gaining 4.63 points over random on gpt-oss-120b. Even so, that oracle finished 7.1 points below the best single-message route in the record, and 5.5 points below it on gemma-4-31B-it. The best route is a hindsight ceiling picked after the fact, not a policy anyone can deploy. What the gap does show is that correctness as a yardstick fails to identify every successful route.

There is a point here where the paper crosses into practice. A did-it-help label never falls out of rereading a log, however carefully. It exists only once you rerun the round with the message and the round without it on the same model, the same prompt and the same ordering, then compare. That is why the paper accepted the overhead of five fixed roles and a cached pool. To measure the value, you have to be able to reproduce the log under exactly those conditions.

So for a team trying to turn internal agent logs into a training asset, the first question is not what to discard but whether it can be rerun. Start by checking whether the log records which model, which prompt and which ordering the message sat in, and whether the integration step alone can be re-executed under those conditions. Without reproducibility the only option left is filtering by correctness, and the share this paper measured disappears along with it.

This experiment does not stand in for every environment. The authors state plainly that the results may not carry over to interactive debate, teams mixing different models, frontier models, or tasks without a stable ground truth. There is also the condition that a single model family filled all five roles within one run. What travels, then, is not the figure of 41.9% but the question: which messages in our own logs changed the outcome, and do we have any means of finding out?

Earlier pieces along the same axis include Original Authors Graded the AI's Paper 2 out of 6 on how the identity of the evaluator changed the score, Eighteen of thirty agents opened a branch with the same name on agents colliding over one shared resource, and Doing Science with a Tool You Cannot Reproduce on the gap an irreproducible tool leaves in science. The paper and its reproducibility archive are at arXiv:2608.14375.

• 1.Yang, C.-H. et al. (2026). "Wrong but Useful: Trajectory Value Beyond Answer Correctness in Multi-Agent Messages." arXiv:2608.14375.