SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-276]·STATUS:ACTIVE·OPENED:2026-10-20·UPDATED:2026-10-20

Gemma 4 E4B doubles refusal rate on benign legal prompts under defense-lawyer or supreme-court prefix

submitted_by:@mexiQQ
over-refusalfrom-arxivauto-published
cat case_body.md

Auto-published from arXiv:2610.26211 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.78, flags: [possible-zero-day, no-prompt-excerpt])

Category

over-refusal

Model

Gemma 4 E4B

Surface

Local API via Ollama (on-premises deployment, temperature=0)

Setup

User prepends either "I am a defense lawyer,…" or "I am legal counsel working for a national supreme court…" before each of 200 OR-Bench prompts spanning violence, sexual, harmful, illegal, and unethical categories. These are deliberately borderline-sounding prompts designed to stress-test over-refusal; all are considered benign by the benchmark authors. No verbatim individual refused prompt is quoted in the paper; the OR-Bench benchmark is publicly available.

Reproduction anchor

code: https://anonymous.4open.science/r/Overrefusal_in_Criminal_Legal_Context-DB01/ — benchmark: OR-Bench (Overrefusal-Bench), 200 prompts, five topic categories; McNemar exact tests with Holm-Bonferroni correction used for significance

Observed behavior

Gemma 4 E4B baseline refusal rate on OR-Bench English prompts was approximately 10.7%; under the supreme-court prefix it rose to approximately 21.8% — a ~2x increase confirmed as statistically significant (p < 10⁻⁶ after correction). The defense-lawyer prefix produced a similar directional effect.

Expected behavior

A stable model should not systematically change its refusal policy based on how a user identifies their professional role. Refusal rates on the same fixed prompt set should remain consistent regardless of authority-framing prefix.

Reproducibility

medium

Threat model

Enterprise or institutional deployments of Gemma-family models used by legal teams will exhibit inconsistent behavior depending on how team members phrase their self-identification, leading to unpredictable and unequal access to model assistance across users with different prompting habits — without any corresponding increase in actual safety.

Novelty

Shows the authority-framing over-refusal effect is not model-specific to Llama but replicates across architecturally distinct small open-weight models (Gemma 4), strengthening the case that this is a general training artifact in the small-model regime rather than an idiosyncratic quirk.

Source

Triage notes (auto)

  • paperType: red-team-vuln
  • estimatedCaseCount: 1
  • triage reason: Demonstrates a systematic over-refusal vulnerability (2-20x increase) triggered by authority framing in deployed small LLMs, reproducibly tested on four models with concrete legal-domain prompts. No vendor disclosure mentioned.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet