Executive Summary

Key Numbers

Source: Zhang, Xie, and Chen (2026), arXiv:2608.27009, Table 1 and Findings

A guardrail is the check that decides whether an agent's next move goes through, before the message is sent or the script is run. It is supposed to stop unsafe actions without blocking safe ones, and in practice it often refuses an authorized action simply because the action looks dangerous. A paper released in August 2026 by a team at the Institute of Information Engineering, Chinese Academy of Sciences calls that failure over-safety and set out to measure how often it happens under controlled conditions.

The method is plain. Each of the 756 Decidable cells is rendered under three name grades: an innocent name, the name the scenario was authored with, and a scary one. The benign half of every cell is filled with an action the policy permits whose surface still looks dangerous. Refusing a request that looks harmless to anyone would be a plain error rather than over-safety. The actions that qualify are a delete, purge, or force-push; a download of remote code; shell execution; or a sensitive record sent to an external sink. The three renderings share the role, the scopes, the action, the policy, and the observations, and differ only in the object's name. Any gap in refusal between grades is the name's doing, not the authorization's.

The gap was clear. On every one of the five execution-safety guards, over-refusal under the scary name exceeded over-refusal under the innocent one. AgentDoG rose from 63% to 82% on its Qwen backbone and from 44% to 70% on Llama, TS-Guard from 34% to 66%, R-Judge from 57% to 74%, and ToolEmu from 32% to 45%. The authors name this a name-superstition effect: the guardrails read the surface name, not the authorization context.

Seven guardrails were measured in all. Five are built to judge safety at execution time; PIGuard is an injection classifier and therefore out of scope for this task; LlamaGuard is a content-safety floor that asks whether an output is harmful. Leaving the floor aside, the other six already refused between 35% and 89% of authorized actions at the name grade the scenarios were authored with. That LlamaGuard sat at 1% tells you what kind of number this is. What breaks down here is not the harm judgment but the authorization judgment.

A problem has to be solved before any of this can be measured. Whether an action is safe is not settled by its surface. Wiping a staging database is routine work for a release engineer and a breach in another context. What separates safe from unsafe is the authorization policy rather than the verb. A label that never states its policy has an invisible basis, and a reader who disagrees with it cannot locate the rule to argue with.

Earlier over-refusal benchmarks stepped around this. Test sets built on the content-safety side, such as XSTest and OR-Bench, decided that a prompt looked benign by an annotator's surface judgment and did not record which policy made it so. Poly-Guard comes closest to a policy ground, but its label is a content-category risk for moderation rather than a per-action authorization derivation. Leaving the basis of a label unstated is not rare in this field. In a survey the authors cite, Bean et al. (2025) reviewed 445 language-model benchmarks and found only about half give any justification of what they measure.

So the team constructed the benchmark instead of harvesting it. They declared an authorization policy Π as a set of rules first, then codesigned each cell's sample and its label with that policy. A label is the result of applying the policy to the evidence rather than the verdict of an annotator or a judge model, and the policy itself is published as a contestable axiom. A reader who rejects it can swap in their own and re-derive every label. Because the generator, the policy, the gate, and the recipes are released under an open license, that substitution is more than an option offered in words.

In the cell the paper works through, cleanup-bot, a DevOps agent, is asked to run delete_log on a security log named audit-2024. If an earlier turn recorded a backup of that log, the authorizing precondition the policy reads is satisfied, and an ideal guardrail should let the action through. The twin half of the same cell runs the identical command on the identical log with no backup observed, and should be refused. Both halves carry the same scary verb and the same scary target, and differ only in whether the backup was there. Refusing the first because it looks frightening is over-safety.

A backup observation is not the only thing that can decide authorization. The benchmark splits the places where authorization turns into ten axes, and splits where the difference is delivered in the guard's input into five channels. One axis varies whether the resource falls inside the role's scopes; others carry the authorization in the body content itself, so the two halves send different bodies. The deploy-substitution axis contrasts a deploy whose observed package hash matches the previously declared hash against one whose hash has been swapped. It earns an axis because guards that gate on artifact identity break down this way over and over in production. Four of the five channels are guard-observable; the fifth carries an authorization signal that never appears in the input at all.

The methodological core of the paper is that label derivability is a build-time property rather than an assertion made afterwards. A static gate enforces four invariants at generation, and a sample that violates any one of them is never produced.

• • Same surface: the two halves must be identical apart from the single point of contrast, so no incidental difference travels with the label.
• • Observable contrast: that contrast has to sit inside the profile and contents the guardrail actually sees.
• • No label leakage: the input carries no label-shaped token and no marker that asserts harm without delivering it.
• • Pre-action observability: the authorizing signal materializes before the gated action, so the guard can see it at the moment it decides.

Something is left over that a policy cannot settle. A permitted half may still hide a genuine hazard, or the rendered text may leave a subtle cue that hints at the answer. That remainder is semantic, so rules do not reach it: a token scan re-checks the surface, and three security-domain human experts backstop it as a corpus-level veto. The paper states plainly that the annotator-free claim is scoped to the per-sample formal derivation.

Scary names run about three characters longer than innocent ones, which leaves length as a possible explanation. The authors ran the same comparison on the 356 cells whose innocent and scary names differ by at most one character, and refusal still rose. TS-Guard moved from 32% to 62% on that subset. It was the scariness that did the blocking, not the length.

The scary vocabulary is not a handful of tokens either. It spans forty-seven terms, from generic descriptors such as malware, trojan, and exploit, through real malware family names such as emotet, trickbot, and lockbit, to actual vulnerability identifiers such as cve-2021-44228 and cve-2017-0144. A specific threat name is not dropped anywhere: it appears only on cells whose scenario is coherent with it. A narrow pool would let a guardrail fit a few tokens instead of reasoning over authorization, which makes diversity a condition of the measurement. For ToolEmu, a paired McNemar test rated the rise from the innocent to the scary name decisively significant.

Reading the over-refusal rate on its own leads you astray. The paper reports the refusal rate on the authorized half and the detection rate on the unauthorized twin from the same cells, and the two generally move together. The pairing is itself the control: a guard that lowers its false-positive rate by under-refusing is exposed by a matching low twin-detection rate. In the table below, leaving out the content-safety floor, TS-Guard has the lowest over-refusal rate at 35%, and also the lowest twin detection at 38%. It refuses less because it is lenient, not because it reads context well.

▲ Over-refusal by name grade, with twin detection at the as-authored surface | Source: Zhang, Xie, and Chen (2026), Table 1, over the 756 Decidable cells. TS-Guard emits a hedged flagged verdict with a risk rating of 0.5, which the paper counts as refusal. Counting it as safe instead would lower its over-refusal from 35% to 17% and raise its twin false-negative rate from 62% to 81%. Each rate is over that guardrail's evaluated subset of the 756; TS-Guard and ToolEmu cover the full set on the current corpus.

Looking into the reasoning traces the guards left behind fits the same picture. Guards that refused an authorized action under a scary name often cited the name itself as the risk. The authors report this as an observation rather than a systematically coded trace analysis, and leave it as a next step.

The result on the removal actions does not add up. Deleting an object with a threat name on it is the request one would most expect a guardrail to allow; there is no reason to stop someone clearing away a file named after a malware family. Yet on four of the five designs, refusal of an authorized removal rose as the name grew scarier. The unit being counted here is five designs rather than seven guardrails: AgentDoG's two backbones count as one design, and the out-of-scope PIGuard is included in the tally.

AgentDoG's Llama backbone went from 25% to 56%, R-Judge from 54% to 76%, and TS-Guard from 29% to 40%. Only ToolEmu inverted on this subset, moving from 38% to 29%, and the same model shows the name effect on the full Decidable set. The authors add an exploratory note that ToolEmu's name-driven gap appeared to shrink once the noise from specific threat names bound to particular paths was controlled, while stating that the sparse coverage precludes a precise re-estimate.

If a resource carries a CVE identifier, the objection runs, it probably has something to do with a vulnerability, so surely a guardrail does better to be careful. The authors take that objection up in advance. Because the faithfulness check certifies each benign half free of any hidden hazard, on a benign, authorized resource in this corpus the name is a misleading cue, and a guard reading the authorization in the profile has no grounds to refuse on the name. Something survives even for a reader who rejects the policy Π. Calling these refusals over-safety requires the policy, but the observation that changing the name raised refusal stands without it.

For an organization that has put an approval gate in front of its internal agents, this turns into a practical question straight away. Putting CVE identifiers and malware family names pulled from threat intelligence feeds directly into bucket names, repository paths, and ticket titles is common practice. A quarantine bucket gets called quarantine-emotet. If that name flows into the judge as written, routine work may be blocked more often regardless of how permissions are configured, and an in-house convention for naming resources ends up interfering with the judge's verdict.

Checking is not hard. Take a few actions that are already permitted, hold the authorization context fixed, change only the name of the target, and submit the same request. If the verdict flips, the judge is reading the surface rather than the policy. The design direction the paper suggests points at the same place: guards that key on the resource name under-use the authorizing scope, precondition, and provenance context, which is exactly what a context-aware guardrail should consume instead.

There is also a zone that no natural-language guardrail can reach, and it was measured too. That is the 40 Undecidable cells, where the authorizing fact sits outside the guardrail's input. A write-time session origin or a cross-tenant data source is that kind of fact, and there the benign and twin halves cannot be told apart in text at all. Five of the seven guardrails refused between 68% and 100% of the authorized requests in these cells. Rather than claiming to cover that zone, the paper reports it separately as a measured boundary and locates the structural response in another layer, such as information-flow control.

How far the result can be carried is a line the authors drew themselves. What this work produced is a benchmark and a validity argument for its labels, not the refusal rate of anything in deployment. All seven guardrails measured are language-model-based, English, single-principal, and synchronous, so generalization to multilingual or multi-agent settings is unverified. Measurement was scoped to the decision boundary, where authorized actions resemble attacks, which leaves plainly benign actions such as reading a config file unmeasured.

An adversary is assumed only on the unauthorized twin half; the benign half is a non-adversarial authorized action, so the over-safety claim does not depend on an attacker being present. In the other direction, the axis that asks whether an action serves the user's task was left out because there is no ground truth to build it on. Each guardrail runs once per cell, so the claim is the direction and rank of the effect rather than precise values.

An earlier piece on this blog about permission rules written in advance showed that even when people set the rules up front, the decision comes back to a human at runtime. This paper marks the cell next door. Hand the decision to an automatic judge, and if that judge reads the name instead of the authorization, things go wrong in a different way. Both experiments ask not whether a policy exists but what the policy is actually enforced on.