Naive caution system-prompt drives claude-sonnet-4-5 and gpt-5 to 100% over-refusal on real emergencies
Auto-published from arXiv:2610.34535 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [no-prompt-excerpt])
Category
over-refusal
Model
claude-sonnet-4-5; Azure gpt-5
Surface
agent harness (tool-using agent with swappable defense wrapper)
Setup
A prompt-only preamble — "when in doubt, do not act; escalate" — is prepended to the system prompt of a tool-using agent handling public-safety scenarios (emergency dispatch, alert broadcast, records queries). The agent is then evaluated on 40 urgent benign control scenarios (family A) drawn from real 911-call transcripts and disaster-response messages. No verbatim full system prompt is reproduced in the body; the quoted directive is the caution preamble as described in Section VI.
Reproduction anchor
benchmark: SentryBench, 190 scenarios (families A–F), real data from HuggingFace: deepset/prompt-injections (Apache-2.0), jailbreak-classification (Apache-2.0), disaster_response (research), 911-call-transcripts (MIT); family A n=40 urgent benign controls; no code repository URL found in paper body.
Observed behavior
Over-refusal rate (ORR) reached 100% on both Sonnet and GPT-5 under the caution defense — the agent refused to dispatch help or take any warranted action for every urgent emergency. Mean harm score rose to 1.80 (Sonnet) and 1.60 (GPT-5) versus 1.00 baseline, making caution the worst-performing defense on all three models. Attack-success rate (ASR) changed by at most 3.7 points, so the refusal tax bought essentially nothing defensively.
Expected behavior
The agent should dispatch responders, send alerts, or take other warranted tool actions for genuine emergencies; refusing all urgent scenarios is a first-class deployment failure in public-safety contexts.
Reproducibility
medium
Threat model
A public-safety operations center that adds a standard 'refuse-when-uncertain' safety guardrail to its LLM dispatch agent inadvertently causes 100% failure to respond to real emergencies, directly increasing risk of death or serious harm. The attacker need not do anything — the defender's own caution policy is the failure vector.
Novelty
First formal proof and empirical demonstration that a refuse-more caution policy is harm-dominated in a bidirectional, cost-calibrated public-safety benchmark; shows caution strictly increases deployment harm relative to an undefended baseline when attack leverage is low.
Source
- arXiv: 2610.34535
- PDF: https://arxiv.org/pdf/2610.34535
- Categories: cs.CR, cs.AI
- Authors: Md Abu Yusuf, Oyshi Tabassum Aditi, Md Faysal Hosain Mazed
Triage notes (auto)
- paperType:
benchmark - estimatedCaseCount: 3
- triage reason: Benchmark with 840 episodes grounded in real emergency transcripts and real injection/jailbreak payloads, demonstrating concrete agent-level failures (over-refusal, over-reaction, adversarial tool misuse) across three frontier models with quantified failure rates.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet