SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-264]·STATUS:ACTIVE·OPENED:2026-10-20·UPDATED:2026-10-20

Keyword matching reports 96% ASR for low-resource language attacks that a semantic evaluator scores at 2.7%

submitted_by:@mexiQQ
from-arxivauto-publishedother
cat case_body.md

Auto-published from arXiv:2610.26595 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.77, flags: [no-prompt-excerpt])

Category

other

Model

Llama 2-7B, Vicuna-7B, Qwen3-8B

Surface

API / local inference

Setup

Harmful requests from a 50-item AdvBench subset are translated into Scots Gaelic, Hmong, Zulu, and Guarani using the googletrans library. All four translations are submitted; the highest-scoring output is selected (best-of-4). Responses are evaluated by (a) keyword matching against Zou et al. 2023 refusal strings, (b) a fine-tuned risk classifier (Huang et al. 2023, threshold >0.5), and (c) StrongREJECT scoring with Qwen3-8B (z = (1−r)·(s+c−2)/8, threshold >0.5).

Reproduction anchor

benchmark: AdvBench harmful_behaviors deduplicated subset (n=50, Appendix E); translation library: googletrans; evaluator: StrongREJECT formula with Qwen3-8B; no code repo linked in the paper

Observed behavior

Keyword matching reports 144/150 successes (96% mean ASR) because model responses in low-resource languages do not contain English refusal strings. Qwen3 StrongREJECT scoring reports 4/150 successes (2.7% mean ASR), indicating the vast majority of 'successful' responses are actually refusals or benign content in the target language.

Expected behavior

A valid benchmark evaluator should distinguish between (a) a model that produces harmful content in a non-English language and (b) a model that refuses in that language; keyword matching conflates both as 'success'.

Reproducibility

medium

Threat model

Researchers and practitioners relying on published ASR figures for low-resource language attacks will dramatically overestimate the real-world risk: a 93-percentage-point gap means safety decisions (e.g. deploying countermeasures) are made on phantom failures. Conversely, genuinely harmful low-resource outputs may go undetected if only keyword matching is used in production moderation.

Novelty

Largest documented evaluator disagreement gap (93.3 pp) for a standard jailbreak family, directly attributable to keyword matching being blind to non-English refusals — a systematic, reproducible evaluator flaw rather than a model quirk.

Source

Triage notes (auto)

  • paperType: benchmark
  • estimatedCaseCount: 2
  • triage reason: Systematically benchmarks 8 jailbreak attacks against 3 open-weight models under 3 evaluation procedures, revealing that identical attacks show 96% vs 2.7% success depending on evaluator, and demonstrating that repeated sampling defeats initial refusals (probabilistic suppression rather than robust exclusion).
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet