Executive Summary

Ask a deep-research agent for a long answer and every sentence comes back with a footnote attached. It looks reassuring. Click a link and the page opens; even the title alone seems on-topic. But a link resolving and the document actually supporting the sentence in front of it are two entirely different things.

One prior study put a number on that gap. It parsed the citations produced by deep-research agents and measured them three ways: link validity came in above 94% and topical relevance above 80%, yet the share of citations that factually supported the claim landed at just 39% to 77%. Scaling the search tool from 2 calls to 150 dropped accuracy by an average of 42%. Looking harder did not make the footnotes any sturdier.

A natural question follows. If the footnotes are this unreliable, who verifies them, and how cheaply can that verifier be built? Where the earlier study questioned the trustworthiness of the citations themselves, this new paper takes aim at the verifier side.

The paper lined up eight off-the-shelf LLM judges in one place. It spread three families evenly: Claude (Haiku, Sonnet, Opus 4.6), Gemini (3.1 Flash Lite, Pro), and GPT (GPT-5-mini, GPT-5.4-mini, GPT-OSS-120B). The material under review was an adversarial benchmark seeded across several domains with deliberate factual errors and misleading citations. A human-reviewed gold set of 1,248 items, plus 378 hard cases where judges disagreed and a person arbitrated, became the baseline.

There are two axes of evaluation. One is source relevance: does the citation relate to the topic? The other is factual support: does the cited content actually back the claim? Relevance is the relatively easy call; factual support is far harder.

On source relevance, the winner was the low-cost model. GPT-5-mini topped all eight judges at F1 0.908 on the pass class, with agreement κ of 0.636. Frontier-tier models like Claude Opus 4.6 and Gemini 3.1 Pro sat below it. For deciding relevance, the assumption that the most expensive model is the best judge simply did not hold.

On the harder factual-support axis there was no separate winner. All eight models' confidence intervals overlapped, so they were statistically indistinguishable. By the raw numbers Claude Opus 4.6 edged ahead at F1 0.750, but not by a meaningful margin. On neither axis did the data support the hypothesis that a pricier model is a better judge.

The problem appears when you pick a judge on F1 alone. F1 is a scalar that lumps false positives and false negatives together, so which way a judge errs stays buried inside it. This is exactly where the paper digs in.

All eight judges rated citations more strictly than the human gold pass rate of 79.3%. The direction was the same for all of them: they lean conservative, crediting supporting citations less than they should. But that strictness spread from 42.9% to 72.0% across models. One direction, yet a magnitude scattered by nearly 30 percentage points.

The subtler point is that two judges with the same F1 can still have wildly different false-positive and false-negative rates. One judge tends to err by letting unsupported citations through; the other errs by failing perfectly good ones. When the two errors cancel out, the F1 scores land close together. So F1 alone makes the two judges look identical. Attach them to a downstream system, though, and they produce completely different outcomes.

Why the direction of the bias matters becomes clear once you see where these judges are used. Reinforcement learning now stands up an LLM judge for each scoring rubric, and while training runs, that judge is the reward model. If the judge says pass, reward goes up; if it says fail, reward goes down.

Now suppose you seat a judge that tends to err toward false negatives as the reward model. This judge keeps failing claims that are in fact supported. The reward signal grows that much sparser, and the downstream model learns as if it were punished every time it adds a citation. The result shows up two ways: it plays it too safe and avoids firm assertions, or it simply cites less. A one-sided bias in the judge hardens into a habit in the model as it passes through the training loop.

Hence the paper's blunt conclusion: if you want to use a citation rubric as a reward signal, calibration comes first. Unless you measure and correct which way the judge leans and by how much, then no matter how high the F1, that bias flows straight downstream. And, the experiments show alongside, this calibration does not require the most expensive model.

The shape of the problem — a metric distorting behavior once it becomes a reward — is not unfamiliar. Pebblous has covered it before, in the distortion of using a proxy metric as a reward and in the traceability problem of the data that verifiable-reward RL learns from. This paper moves the root of that distortion one step earlier. If the judge that produces the reward is already skewed, everything optimized on top of it skews with it.

Read as a data-trust problem, this paper converges on a single point. In an era where AI verifies data, the verifier itself becomes the new pressure point for trust. More and more LLM judges are taking the seat that decides whether a footnote is right, whether a label is accurate, whether an output obeyed the rules. Fail to measure which way that judge leans first, and the whole pipeline tilts, silently, to one side.

Bring it down to practice and two things remain. One, automating verification cheaply is now a realistic option; measurement shows a low-cost model holds even with frontier tier on relevance calls. Two, you still must not pick a judge on a single aggregate score like F1. Calibrate first — does it lean toward false positives or false negatives, and how much stricter or more lenient is it than gold — and only then attach it to a reward or a gate.

The AI-Ready Data that Pebblous talks about goes one step past data that is cleaned and labeled. Only when the bias of the tools that verify that data is also measured and corrected does the trust you build on top of it stay steady. Measuring the verifier before you trust it — that is the data-quality version of the old question of who judges the judges. Which way is the judge inside our pipeline leaning right now?

Academic

• 1.Leung, E., Lumer, E., Feld, C., Huber, A., Subbiah, V. K., & Paul, K. (2026). "Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution." arXiv:2607.08700.
• 2."Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents." (2026). arXiv:2605.06635.

Pebblous

• 3.Pebblous. "Why Does AI Ace the Test but Fail the Job?" blog.pebblous.ai.
• 4.Pebblous. "Nobody Can Trace, Atom by Atom, the Data That Verifiable-Reward RL Learned From." blog.pebblous.ai.