Executive Summary

Key Numbers

Four numbers stand out. The first two show how unstable the "reason" label is, and the last two point at where that instability came from.

Source: Beesetty, M. S. R., "Agreeing on the Answer but Not the Reason: Measurement Limits of LLM Reasoning Failure Taxonomies", arXiv:2611.13140 (IEEE CogMI 2026).

Model reports carry a familiar sentence. Something like: 40% of this model's reasoning failures were Compositional Collapse. It looks friendlier than an accuracy number. It tells you where the model stumbles, and what to fix next seems to follow from it. The question Beesetty put to that sentence is whether it is a measurement that reproduces.

The experiment ran on mGSM-Symbolic, a variant of GSM8K, the grade-school math word problem collection, rewritten into other languages with the numbers and names swapped as well. Twelve models of widely separated capability went onto it, from Claude Haiku 4.5 and Claude Sonnet 5 and DeepSeek-V4-Pro through to Mixtral-8x7b and Neural-Chat-7b, and the problems were solved in eleven languages: Chinese, Thai, Japanese, German, Spanish, French, Italian, Russian, Swahili, Bengali and Telugu. That produced 2,316 graded records. The count of records is not the count of problems. The items themselves number only 22, drawn from eight distinct structures, and 2,316 is what you get after substituting numbers into them and carrying them into eleven languages. The author states plainly that when describing the breadth of the design, the smaller figure is the one to write down.

The grading went to an "inspector" rather than a person. That is a judge model under a fixed prompt, handed the translated problem, the reference answer and the solution the model actually produced, and required to return a correct-or-incorrect verdict plus quoted evidence for each failure category in a set format. Claude Sonnet 5 filled the seat. That means the model doing the grading also sits in the list of twelve models being graded, and below, "inspector" always refers to the grading role.

Verification went to people. Native speakers of the eleven languages read 179 transcripts themselves and issued the same verdicts. On answer correctness, the inspector and the native speakers agreed at kappa 0.932. Two graders writing down nearly the same thing. Put the same transcripts in front of them and ask only "does this failure get the Compositional Collapse label," and kappa falls to 0.093. Kappa is agreement after the overlap you would expect by chance has been taken out, so a value near zero means the two graders might as well have been labeling independently of each other.

The taxonomy holds four categories. Compositional Collapse is when the model notices a required condition in the problem on its own and then, further down, drops it or applies it to the wrong thing. Reasoning Drift is when the early steps go correctly, an error appears at one middle step, and that error propagates through everything after it. Format Instability is when the required output format breaks, and everything left over is gathered into Mismodeling.

One spot invites confusion with a similarly named concept. Other research uses "compositional collapse" for a failure to connect two stable facts, and the author states in the body of the paper that his definition differs from that one. Collapse here does not come from failing to know a fact. It comes from the model discarding a condition it wrote down itself.

The trouble starts not with the categories but with the rule that narrows them to one. The classifier runs the three testable categories in order, Compositional Collapse then Reasoning Drift then Format Instability, and fixes the label of a case at whichever test fires first. The three tests are independent of one another, so a single case can carry evidence for several categories at once, and the classifier takes only the one that came earlier in the sequence. Cases like that are 50.1% of all failures. The ones that fit a single category cleanly come to fewer than half. And the overlap sits almost entirely on one pair: of the 775 cases carrying evidence for two categories or more, 761 hold Compositional Collapse and Reasoning Drift together.

The author ran all six orderings over the same data. Three tests means six ways to queue them. Swapping the position of just two tests changed the most common failure type in eleven of the twelve models. Lay all six orderings side by side and Compositional Collapse's share runs from 26.4% to 76.4%, while Reasoning Drift moves from 14.9% to 64.7%. A span close to fifty percentage points for each.

The one model of the twelve whose most common type did not budge was the inspector itself. Its share of failures with duplicate evidence is 10.9%, the lowest of the twelve, and 195 of the graded records are solutions this model wrote. The author records this as a limitation. Whether the evidence separated unusually cleanly, or the model tidied up the ambiguity in its own solutions in its own favor, is something this design cannot tell apart.

Where the overlap comes from is simple. One solution can hold two traces at once: a condition the model wrote down for itself and then let go of, and a middle step that went wrong and dragged the rest of the solution with it. Both traces are really there, so whichever test looks first is the one that finds something.

What this structure does to model comparison is direct. Suppose a comparison table says 40% of model A's failures are Compositional Collapse against 20% for model B. Both models went through the same classifier, so it looks fair. Flip that classifier's test order once, though, and the two numbers move together, by an amount that can exceed the difference between the models. A share table that does not disclose its ordering mixes a property of the model with the shadow of the grading procedure.

So the author takes back the ranking claim his own paper was built on. Three of the six orderings put Compositional Collapse first and three put Reasoning Drift first, so whichever you write down has half the evidence behind it. What he reports instead is a quantity that does not ride on the order: how many failures carry any Compositional Collapse evidence at all, which runs from 50.0% to 90.2% by model and exceeds half of failures in most of them. Even this figure, he notes up front, is still a number the inspector assigned. Why he attached that caveat becomes clear in the next section.

Read kappa 0.093 on its own and the conclusion rushes to one place: sorting out why a model failed is a thing that cannot be done. The author did not take that road. He attached a second native speaker per language to rate the same transcripts again, then kept only the pairs where both had given a definite yes or no and counted once more. Human against human came out at 0.659 there. A substantial level of agreement. On the same pairs, each human against the inspector stayed at 0.094 or below.

Four values side by side make the point of divergence plain.

Source: arXiv:2611.13140. The population measured differs from row to row. The first row is the 178 items carrying verdicts from both sides, the second the 142 of those the native speaker judged incorrect. The third is the 121 items where both native speakers answered definitely, the fourth the 106 of those where the inspector running the human instructions also answered definitely. On that same set of 106, the original prompt scored 0.147.

The last row is the turn in this paper. The judge model stayed where it was, and the grading instructions the human raters had actually been holding were pasted in verbatim for a second pass, which lifted agreement with the second native speaker from 0.147 to 0.636. No model swap, no move to a bigger model. What put the inspector at odds with the humans reads less like an inability to sort out reasons than like a prompt that had been pointing somewhere other than the humans' unspoken standard.

Where it pointed instead is written down too. The definition handed to the inspector took in cases where a condition was "noticed or invented," while the definition given to the humans covered noticing alone. The inspector was told to hunt for evidence of every category on every transcript, and the humans answered a single yes or no, and only on transcripts they had judged incorrect. The illustrative example went to the humans only. The remaining one was an instruction to judge against the reference answer, and that item runs straight into section 5. Even for one and the same model, the labels assigned under the human instructions matched its own labels under the original prompt at only 0.197.

The direction leaned one way as well. The inspector called 66.2% of all failures Compositional Collapse where the native speakers called 36.6%, a factor of 1.81. A lean like that lands on the aggregate share intact. That 1.81 belongs to this sample, though, not to the benchmark. The number of items reviewed differs by language and the languages with the widest gaps are heavily represented, so counting again with every language weighted equally gives 1.38. The author prints both.

Would a second judge model have caught the lean? The author put the same 179 items to one frontier model and one open-weight model as well. On the pairs with definite verdicts, DeepSeek-V4-Pro sided with the inspector rather than the humans, at 0.333 against the inspector and 0.090 or below against the two native speakers. Qwen2.5-32b went off the other way, assigning Compositional Collapse to only 13.2% of cases, and its kappa against the humans fell below zero. What all three have in common is that none reproduced the human labels, and the only thing particular to each is the direction it went off in. Adding model graders shows you that the labels scatter; it does not show you which of them holds the human standard.

The author traced that 1.81 lean back down to the level of individual cases. One item from one round came out of it. It is a word problem that counts 702 puppies, and the English original attaches a clause that puts those puppies on the spotted side, as in "13 puppies with spots." That is the basis on which Ibrahim's 702 puppies enter the numerator. Only the Bengali and Chinese translations render the passage as "13 puppies," deleting the attribute. Italian, French, Russian, Spanish, Swahili, Thai and Telugu kept it.

Lose that clause and the problem becomes a different problem. Solve faithfully what the translation says and the answer comes to 11%, while the reference answer used for grading is the 89% the original supports. Written as fractions, 99/900 against 801/900, which makes it an item no one can get right by any route. Accuracy on this round was 0.0% across all twelve models in both languages. The three frontier systems that clear 67% overall were no exception, and Chinese, which reaches 30.6% in its other rounds, fell into the same trap here.

The second native speaker, brought in later, confirms this reading separately. On the Chinese instance of the same item the inspector recorded Compositional Collapse and both native speakers said it was not, and the second reviewer wrote a reason on the answer sheet without being asked for one. The model, the note says, counted the 702 puppies in the total but treated them as zero spotted because spots were never mentioned. The passage the inspector quoted as its own evidence, meanwhile, follows the sense of the English original rather than the translation. Which is exactly what the instruction to judge against the reference answer told it to do.

Take out the whole round those two items belong to, count again, and the gap between the inspector and the native speakers narrows from 66.2% against 36.6% to 56.5% against 48.2%. The 1.81 becomes 1.17. In the author's phrasing, the translation defect explains much of the lean but not all of it. Without that trace back, though, a defect in one line of translation was on course to be reported as the reasoning failure tendency of twelve models.

What this paper points at is a place the Pebblous blog has been twice before. It has covered a case where harm labels split along the leanings of the annotators, and it has looked at where an LLM judge wobbles when it is put in charge of verifying citations. All three arrive at the same conclusion. The quality of a label is not settled by the ability of whoever applied it alone. The procedure and the instructions that told that person how to apply it settle it just as much.

The four checks the author appends at the end of the paper cost almost nothing.

• • Reverse the order and run it once more. If the most common type moves, drop that share table and report a quantity the ordering does not touch instead. One re-run is all it takes.
• • Publish the co-occurrence rate as it stands. This means writing down how many categories a single case carried evidence for. The value is already in hand, so it costs nothing.
• • Collect a separate human sample, and cover the hard cases with it. What it covers matters more than how large it is. The author's own interim tally on the six languages that came back first gave a kappa of 0.280 with almost no gap in the shares, and the languages that came back first happened to be the ones models solve well.
• • Trace a leaning gap down to its cases before promoting it to a rate. That is how the Bengali item was caught.

For a team building evaluation data, these four read less like a work instruction than like a questionnaire. In the dataset we built, is the column that records why the model failed a column two people could fill in the same way? Does an unspoken priority hide inside the process that narrows the label in that column to one? Do the instructions given to the annotators and the prompt given to the judge model state the same standard? The last question is also where this paper got its largest effect for the smallest price.

The answer column and the reason column sit at different levels of quality-control difficulty. Put two people on the answer column and it converges; put two people on the reason column and it will not converge if their standards differ. Yet most of the value in evaluation data comes out of the reason column, because that is the one telling you where to fix things. Without first confirming that two passes over the same data return the same answer, that column ends up describing the grading procedure instead of the model.

Thank you for reading this far. The full paper is at arXiv:2611.13140. In your own evaluation data, does the column that records why return the same answer when the order is changed?

References

• 1.Beesetty, M. S. R. (2026). "Agreeing on the Answer but Not the Reason: Measurement Limits of LLM Reasoning Failure Taxonomies." arXiv:2611.13140 (accepted, IEEE CogMI 2026).