deepset and DistilBERT flag 91–100% of clean legal contracts as prompt injection
Auto-published from arXiv:2611.08834 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.88, flags: [possible-zero-day, no-prompt-excerpt])
Category
over-refusal
Model
deepset DeBERTa-v3-base prompt injection classifier (deepset, 2023); DistilBERT distilbert-base-multi-cased-finetuned-injection (Rodriguez, 2023)
Surface
API (batch document chunk classification pipeline, 512-token non-overlapping windows, vendor default threshold 0.5)
Setup
Both classifiers were run on two corpora of unmodified legal contracts — 683 proprietary commercial contracts and 510 CUAD contracts — with no injected payload whatsoever. Documents are split into 512-token chunks; a document is flagged if any chunk exceeds threshold 0.5. The paper provides no single trigger prompt because the false positives arise from normal legal directive language (deontic modals, operative clauses). Figure 5 shows deepset and DistilBERT exhibit 'saturating degeneracy': scores ramp to ceiling by 16–24 tokens, before a single full sentence is read.
Reproduction anchor
code/data: https://osf.io/67hyb/overview?view_only=267d968d563f47d3a83707b0f3434069 (insertion manifests, payload registry, LLM prompts); CUAD corpus is public CC BY 4.0; guard models are on HuggingFace under their respective repos.
Observed behavior
deepset flags 91.36% of the proprietary corpus and 100% of CUAD as containing prompt injection; DistilBERT flags 92.39% and 100% respectively — on entirely clean, unmodified contracts. Paired AUC for attack vs. own-host baseline is only 0.588/0.828 (deepset) and 0.563/0.681 (DistilBERT), confirming that flagging tracks host identity, not payload presence.
Expected behavior
Guards should produce low false positive rates on benign legal contracts and reserve high scores for documents containing adversarial injection payloads.
Reproducibility
high
Threat model
Any organization deploying these guards as a hard filter in a legal-contract AI review pipeline would have 91–100% of all inbound contracts blocked or routed for manual review, rendering automated review non-functional. As a soft filter, the near-universal alert rate produces alert fatigue that erodes human oversight of genuine attacks.
Novelty
First published FPR measurement of prompt injection guards on legal contracts; shows that legitimate legal directive language systematically saturates guards trained on chat-style injections, creating a fundamental domain-mismatch failure.
Source
- arXiv: 2611.08834
- PDF: https://arxiv.org/pdf/2611.08834
- Categories: cs.AI
- Authors: Mark Heimann, Katherine Taylor, Bruno Jacob
Triage notes (auto)
- paperType:
benchmark - estimatedCaseCount: 3
- triage reason: Systematically measures 11 prompt injection guards against legal contract injection attacks, documenting false positive rates up to 92.39% and a novel threat category (unauthorized scope claims) where all guards score zero. Tests reproducible on defined corpora with controlled probes.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet