SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-383]·STATUS:ACTIVE·OPENED:2026-11-09·UPDATED:2026-11-09

deepset and DistilBERT flag 91–100% of clean legal contracts as prompt injection

submitted_by:@mexiQQ
over-refusalfrom-arxivauto-published
cat case_body.md

Auto-published from arXiv:2611.08834 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.88, flags: [possible-zero-day, no-prompt-excerpt])

Category

over-refusal

Model

deepset DeBERTa-v3-base prompt injection classifier (deepset, 2023); DistilBERT distilbert-base-multi-cased-finetuned-injection (Rodriguez, 2023)

Surface

API (batch document chunk classification pipeline, 512-token non-overlapping windows, vendor default threshold 0.5)

Setup

Both classifiers were run on two corpora of unmodified legal contracts — 683 proprietary commercial contracts and 510 CUAD contracts — with no injected payload whatsoever. Documents are split into 512-token chunks; a document is flagged if any chunk exceeds threshold 0.5. The paper provides no single trigger prompt because the false positives arise from normal legal directive language (deontic modals, operative clauses). Figure 5 shows deepset and DistilBERT exhibit 'saturating degeneracy': scores ramp to ceiling by 16–24 tokens, before a single full sentence is read.

Reproduction anchor

code/data: https://osf.io/67hyb/overview?view_only=267d968d563f47d3a83707b0f3434069 (insertion manifests, payload registry, LLM prompts); CUAD corpus is public CC BY 4.0; guard models are on HuggingFace under their respective repos.

Observed behavior

deepset flags 91.36% of the proprietary corpus and 100% of CUAD as containing prompt injection; DistilBERT flags 92.39% and 100% respectively — on entirely clean, unmodified contracts. Paired AUC for attack vs. own-host baseline is only 0.588/0.828 (deepset) and 0.563/0.681 (DistilBERT), confirming that flagging tracks host identity, not payload presence.

Expected behavior

Guards should produce low false positive rates on benign legal contracts and reserve high scores for documents containing adversarial injection payloads.

Reproducibility

high

Threat model

Any organization deploying these guards as a hard filter in a legal-contract AI review pipeline would have 91–100% of all inbound contracts blocked or routed for manual review, rendering automated review non-functional. As a soft filter, the near-universal alert rate produces alert fatigue that erodes human oversight of genuine attacks.

Novelty

First published FPR measurement of prompt injection guards on legal contracts; shows that legitimate legal directive language systematically saturates guards trained on chat-style injections, creating a fundamental domain-mismatch failure.

Source

Triage notes (auto)

  • paperType: benchmark
  • estimatedCaseCount: 3
  • triage reason: Systematically measures 11 prompt injection guards against legal contract injection attacks, documenting false positive rates up to 92.39% and a novel threat category (unauthorized scope claims) where all guards score zero. Tests reproducible on defined corpora with controlled probes.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet