Executive Summary

Key Numbers

Source: Xiao & Cai, Follow the Policy, Fail the Benchmark, arXiv:2611.07042 (2026) · BAR repository

The four numbers below rest on four different denominators. The first two are out of the 96 LegalBench tax cases, the third out of the 300 RuleArena tax cases, the fourth out of ConditionalQA's 285. Even within one paper each task is measured on its own ruler, so these figures cannot be added together or compared directly.

LegalBench is a public benchmark for legal reasoning, and its SARA numeric task presents a tax scenario and asks how much tax is owed. The answer is a single amount, and the task description asks only that the amount be stated first. Nothing anywhere tells the model to leave out the cents. Scoring is described as accepting any answer within 10% of the reference.

The trouble lies one step before that 10% comparison, in the stage that turns the answer into a number. As the authors record it after reading LegalBench's scoring function directly, the original parser deletes commas and periods and then reads the first run of digits as a non-negative integer. Deleting the comma is the normal business of stripping a thousands separator. Deleting the period on the same line is where the accident lives. The two cents digits stay glued to the dollars, and the amount the model wrote enters the comparison a hundred times too large.

What makes the bug nasty is that it runs in both directions. To see what the original scorer would accept, the authors fed hand-built answer strings to the same case. Against a reference of $259,487, writing 259487 is accepted. Writing the same amount as $259,487.00 becomes 25948700 and is rejected. Yet $2,594.87, a hundred times smaller, reads as 259487 and is accepted. The exact answer carried to the cent falls, and an answer off by two orders of magnitude passes. Far from measuring ability, the scorer was measuring a habit of notation.

There is one more defect. The original acceptance rule takes the model's value p and the reference r and credits the answer when |p/(r+0.1) − 1| < 0.1 holds. Put a reference of zero into that formula and p has to land between 0.09 and 0.11 to pass. An answer of exactly zero, correct because no tax is owed, cannot be credited by construction. That makes a second defect, counted separately from the parsing one. Four of the 96 test cases have a reference of zero.

With both defects in play, the scorer's own ceiling comes down too. Echo all 96 references back as the answers and the original scorer returns 92 points. Copying the answer key verbatim does not produce a perfect score. The four that fall are the zero-reference cases. The authors are explicit that 92 is a scorer diagnostic and ceiling rather than a model result.

Start with how the rescoring was set up. For each case the authors collected one response from Haiku 5.5, GPT6 Luna, and Gemma 4 31B, kept those responses frozen, and swapped out only the scoring code. For SARA they used the input template the benchmark's authors supply, untouched, across all 96 TEST rows after excluding the four TRAIN rows. The archive holds 2,664 stored responses across four tasks: 900 for the tax task, 288 for SARA, 855 for ConditionalQA, and 621 for OpenExempt.

The repair was measured in two stages. Stage one inserts the decimal-preserving parser and leaves the original formula alone; stage two adds a rule that credits a zero reference only when the answer parses as exactly zero. Here is where the three models land.

Accepted answers out of the 96 LegalBench SARA numeric cases. Right-hand block of Table 1 in the paper. No model loses a single previously accepted answer.

Nearly all of the climb comes from stage one, where the decimal alone carries Haiku from 8 to 87. Stage two adds exactly four to each model, the share tied to those four zero-reference cases, and none of the other 92 judgments move at all. The abstract records this as an acceptance gain of more than 80 percentage points for each of the three models.

A count of the responses explains why almost everything was caught. The first amount the corrected parser finds is a decimal in 82 of Haiku's responses, 76 of Luna's, and 78 of Gemma's. All three wrote cents in roughly eighty of the 96 cases, and that habit, which the task description never forbade, turned into a wrong answer under the original scorer. Every response yielded a readable amount. Of the 241 judgments that changed, 229 come from parsing and the remaining 12 from zero handling.

Sharper than the scores is the ordering. Under the original scoring GPT6 Luna stood alone at the front with 9 while the other two tied at 8. Corrected, the board reads Haiku 91, Gemma 88, Luna 87, and the model that had been first is now last. The gap between 8 and 9 was noise, and the noise had been passing for a ranking.

The defect the same paper found in another benchmark sits on the reference side rather than in the scorer. RuleArena hands over real rule documents — tax code, airline policy, basketball league regulations — and asks that they be applied to a case. For its tax task the answers are not written by a person; a reference program computes them.

US tax forms attach an instruction to certain lines: if the result drops below zero, do not carry the negative number, enter zero instead. The authors call this a zero floor, and it is part of the arithmetic the form requires. It is also written into the material the model is given. Line 22 of the case the paper presents reads, "Subtract line 21 from line 18. If zero or less, enter -0-." That zero floor was exactly what the RuleArena tax reference program omitted. The negative number survived, dragged the lines beneath it down, and the skewed final amount was registered as the correct answer.

On that case Haiku 5.5 answered $703, the value you get by applying the floor as the form instructs. The original answer key called it wrong; the key with the missing rule restored called it right. Not one character of the model's answer changed. Two components inside the same benchmark read the rules differently. The one that analyzes rule by rule does apply the floor. Only the program that produces the final amount skips it.

With the floor restored and the amounts beneath it recomputed, 27 of the 300 references changed. The scoring side was left alone. Acceptance on the tax task runs on a relative tolerance of 10⁻⁵, which in practice demands an exact match, and that threshold stayed put too. All 300 cases remain in every denominator, and every changed judgment falls inside those 27.

Which 27 was settled before any model was asked for a response, by rerunning the reference program on its own. How any model might score played no part in the selection. Because the cases carry equal weight, the most this correction can move a fixed set of predictions is fixed in advance: nine percentage points in either direction. How the distances between models shift inside that narrow band is in the next table.

Answers matching the reference out of the 300 RuleArena tax cases. Left-hand block of Table 1 in the paper.

What surfaces here is a distortion of the distance between models. The spread between Haiku and Gemma was 14 cases under the original scoring and 4 after the correction. The difference in their ability had been inflated more than threefold. The order of the three models does not change, but much of the distance separating them was manufactured by a defect in the answer key.

Those 61 new acceptances were not taken on trust: the authors read the response text and split it in two. In 48 of them the model states in its response that it is applying the zero floor; the other 13 show a subtotal of zero without saying so. Haiku's single new rejection belongs to the latter group. It did set the subtotal to zero, but errors elsewhere in its arithmetic cancelled out and its final amount happened to match the old, wrong reference. Separately, they note that this one does not count as a failure to apply the floor.

Nine entries were audited: RuleArena, OpenExempt, LegalBench SARA, DeonticBench, ConditionalQA, ShARC, PolicyGuardBench, WorkArena++, and TheAgentCompany. One criterion governed the selection. The task had to apply a supplied rule document to a case, and its inputs or code or scorer had to be open to outside inspection. The method mixed three things: source tracing, program execution, and controlled checks.

The paper sorts what it found into four kinds. The preceding three sections cover only two of them. The other two show how wide this audit swept.

• Reference answers violate what the task requires. RuleArena Tax from section 3 belongs here. OpenExempt, the bankruptcy exemption task, falls in the same kind with one addition: its mismatched reference values are passed forward as inputs to a later stage. Section 5 takes that up on its own.
• A faulty reference program passes evaluation. In one DeonticBench tax case the reference program counts a dependent twice and omits the taxpayer's own exemption. The two errors cancel, the exemption total of $4,000 comes out right, and the authors' scorer accepts the answer. Judged on its output alone, the program looks correct.
• Scoring goes wrong or a check is missing. LegalBench from section 1 lives here. TheAgentCompany's budget task is blunter still. The agent has to tell each manager both the total equipment cost and whether the budget suffices, yet running constructed replies through the official scoring chain earned full credit even when the budget judgment was dropped or outright reversed, so long as the expected total strings were there. One WorkArena++ scheduling task requires at most an hour between consecutive requests while its validator checks against a 24-hour limit.
• A target is unsupported by the inputs the model can see. PolicyGuardBench holds 279 groups, covering 780 records, in which the visible inputs are completely identical yet the binary labels conflict. The strict release that removes the overlap still leaves 60 groups and 143 records inside the test split alone. Any method deciding from the visible input is guaranteed to be wrong on at least 61 of them. That is a consistency bound, not a verdict that 61 labels are wrong.

The nine entries did not all receive the same disposition. Corrected code ships for seven components across five tasks. RuleArena Tax, Airline, and NBA, LegalBench SARA numeric, and ConditionalQA are the five, with Airline accounting for three components on its own because its policy transcription error, its amount scoring, and its missing minimum-cost objective each need separate attention. The remaining entries were published as audit evidence with no revision. On the NBA task, two fields out of 216 records were corrected and not a single binary verdict moved — a controlled case left on the record of a defect that does not shift the score.

ConditionalQA runs the other way. Its original aggregation dropped any question the model did not answer out of the denominator and scored only what was submitted. The stored responses contain 77 such gaps: 24 from Luna, 53 from Gemma. Counting a missing submission as zero and keeping all 285 questions in the denominator pushed both models down. Luna's exact-match rate fell from 52.34% to 47.93% and Gemma's from 58.02% to 47.23%. Haiku, which had no gaps at all, did not budge from 58.63%.

Those gaps, though, arose because the models failed to honor the JSON response format the authors used, not because the benchmark's labels are wrong. So the drop in score measures a choice of aggregation rather than the size of a defect. The authors checked exactly that by feeding eight external submission files, every question answered, through both scorers; all four metrics and the per-category denominators came out identical. Where nothing is missing, the two scorers are indistinguishable.

Everything so far has been about scores. Correct the scoring and the numbers change, and sometimes the ranking changes with them. But there is a place where a benchmark's defect walks off the scoreboard entirely. Some tasks break a long calculation into stages, solve the earlier stage in advance, place the answer in the model's hands, and then ask about the stage that follows. What happens when the answer placed there is wrong?

OpenExempt is such a task. It asks how much property a bankruptcy filer can shield from creditors under each exemption provision. An intermediate stage computes the maximum protectable amount for each asset under each provision and passes those values into the later stage as inputs. That intermediate calculation, though, applies a provision only once. For the same $8,200 car under the same provision, the intermediate answer records $4,450 as the maximum while the final allocation applies the provision a second time and shields the entire $8,200. The final answer exceeds the figure labeled as the maximum. The conflict touches 38 entries across 33 of the 100 records at that stage. The authors are careful here: this establishes that two reference artifacts disagree with each other, not which of the two amounts is legally right.

To find out whether those intermediate answers actually move model decisions, the authors ran an experiment. They rewrote the task with its premises nailed down: the spouses file jointly, both use federal exemptions, each owns half of every listed asset, and no debts are secured against those assets. On that footing the task asks for the smallest total asset value that must remain unprotected from creditors after federal exemptions are applied. Three conditions were run over 69 records — hand over the released intermediate answers as they stand, hand over intermediate answers recomputed to fit the premises, or hand over none at all. Policy, question, and case facts are identical across all three.

A fresh response was generated and scored for every condition. The OpenExempt minimum non-exempt value task with premises added, over 69 cases. Table 3 in the paper. Acceptance requires the specified JSON format and an amount within $0.005 of the target.

The column that stands out is the one on the right. Handed no intermediate answers at all, all three models got every one of the 69 cases right. Handed the released ones, Haiku sinks to 50. Values placed in its hands to help tripped it nineteen times. Before collecting any responses, the authors had set aside the 19 cases where treating the supplied amount as a ceiling changes the final answer. Sixteen of Haiku's 19 rejected answers are in that set. The model trusted the number it was handed and used it as the ceiling on its own calculation.

The models did not all move the same way. The corrected intermediate answers lifted Haiku from 50 to 68 while Gemma slipped from 65 to 64. Luna's one new rejection gives the right amount but breaks the JSON format; Gemma's three keep the format and get the amount wrong. The scope needs narrowing too. This is a task the authors rewrote with premises added rather than the original OpenExempt task, each case and condition drew a single response, and no tools were used. They say as much: not every response change can be laid at the door of an adjudicated defect in the original benchmark.

Everyone doubts the quality of training data. Is the label right, is the provenance clean, is the distribution skewed — some version of that procedure exists in nearly every organization. Far fewer teams doubt, with the same intensity, the quality of the code that evaluates the model built from that data. A scorer is usually written by one person in a few days, and once it starts running nobody opens it again. And the decisions get made by looking at the numbers that code emits.

What this audit exposes is less the size of the defect than the way a defect hides. The LegalBench bug is one line of code, and it had already been filed as a public issue two months earlier. The numbers that scorer produced keep getting cited anyway. A wrong score does not look wrong. An 8 just looks like a low score, and low scores attract easy explanations: the model does not know tax law, the model is weak at long calculations. In fact most of the failing answers failed for writing out the cents.

A few posts on this blog have covered the neighboring ground. A buggy answer key made AI agents look worse than they are dealt with an answer key that was itself wrong; this LegalBench case has a sound answer key read by broken code. Eight AI agent benchmarks that hand out perfect scores without solving anything was about somebody deliberately working an opening, whereas here nobody attacked anything and the scores were still wrong. One bug with no malice behind it travels further than an attack.

For an organization that runs its own evaluation system, there are four procedures to take from this paper.

• Do you test the stage that converts an answer into a number or a category on its own? Deliberately feed it an answer identical to the reference, an answer with a decimal point, an answer of zero, and an empty answer, and establish first what the scorer accepts. That is what the authors did.
• Do you keep model responses in their original text? Only if the responses survive can you correct the scoring and set old scores beside new ones without calling the models again. Discard the responses and keep only the scores, and when a scoring defect surfaces there is no way back through the history.
• Do you use a stored score as an input to the calculation, or as an assertion to verify against? BAR's reproduction tool rescores every stored response from scratch each run and compares the result to the old score, terminating in failure if a single one disagrees. Instead of trusting the score and moving on, it proves it again every time.
• Is there a place where you solve an intermediate calculation and hand it to the model? If that value is wrong, it is not only the score that goes wrong but the model's conclusion. In the experiment in section 5, the condition that handed over nothing was the most accurate. Verify any intermediate value you plan to supply with the same rigor as the final answer key, or, if you cannot, do not supply it.

Primary sources

• 1.Xiao, Y. & Cai, Y. (2026). "Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks." arXiv:2611.07042 [cs.CL], submitted November 4, 2026. Every figure, quote, and the defect taxonomy in this article trace back to this paper.
• 2.Xiao, Y. & Cai, Y. BAR — Benchmark Audit and Revision repository (MIT License, v0.2.0). github.com/SugarRuy/BAR-Policy-Decision. Publishes the audit evidence, corrected code, and a reproduction archive of 2,664 stored responses. Table figures in sections 2, 3, and 5 were cross-checked against the README and docs/REPRODUCING.md.

Related Pebblous posts

• 3.A buggy answer key made AI agents look worse than they are (section 6) · Eight AI agent benchmarks that hand out perfect scores without solving anything (section 6)