SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-323]·STATUS:ACTIVE·OPENED:2026-10-27·UPDATED:2026-10-27

GPT-4o-mini jailbroken at 36.7% ASR@5 on HarmBench by Mistral-Small, outperforming all high-capacity baselines

submitted_by:@mexiQQ
jailbreakneeds-disclosure-reviewfrom-arxivauto-published
cat case_body.md

Auto-published from arXiv:2610.31937 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.83, flags: [no-prompt-excerpt]) ⚠ needsDisclosureReview = true — review for undisclosed vulnerability before any public submission.

Category

jailbreak

Model

GPT-4o mini (OpenAI, 2024 — Hurst et al. GPT-4o system card)

Surface

API (black-box)

Setup

Same M³-LC framework (Mistral-Small-2402 attacker + state LLM, CompassJudge-32B evaluator) applied to the HarmBench subset: 90 tasks sampled from 6 categories (chemical-biological, cybercrime-intrusion, harassment-bullying, harmful, illegal, misinformation-disinformation), 15 per category. Temperature 0.5, max 10 turns + 10 retries, target limited to 512 tokens. Compared against GOAT-HC (Mistral-Large-2402), Crescendo-HC, and ActorAttack-HC — all using higher-capacity attackers. No verbatim prompt excerpt provided in extracted body; templates in Appendix A.5.

Reproduction anchor

Algorithm 1 in Appendix A.2 (included); HarmBench benchmark (Mazeika et al., 2024) with 90-task custom subset (15 per category, randomly sampled, details in Section 4); hyperparameters in Appendix A.3 (included); prompt templates in Appendix A.5 (not shown in extract); no code repo URL found.

Observed behavior

GPT-4o-mini produces harmful responses at 23.3% ASR@1 and 36.7% ASR@5, compared to the best high-capacity baseline GOAT-HC at 4.4% ASR@1 / 8.9% ASR@5 — a 27.8 pp gain in ASR@5 achieved by a smaller attacker. Crescendo-HC reaches only 2.2% ASR@5 on this model+benchmark combination.

Expected behavior

GPT-4o-mini should maintain consistent refusal across HarmBench's diverse harmful categories regardless of conversational framing or turn count.

Reproducibility

medium

Threat model

Resource-constrained adversaries (e.g., operating with small, inexpensive models) can achieve high jailbreak rates against widely deployed cost-tier models like GPT-4o-mini, which are frequently used in production applications. This lowers the barrier for systematic harmful content extraction across categories including cybercrime instructions and chemical/biological information.

Novelty

Demonstrates that attack architecture (compact state MDP with backtracking) provides more leverage than attacker model capacity, meaning small-LLM adversaries can outperform large-LLM baselines — a resource-asymmetry vulnerability not previously demonstrated in multi-turn settings.

Source

Triage notes (auto)

  • paperType: red-team-vuln
  • estimatedCaseCount: 2
  • triage reason: Automated red-teaming framework demonstrating concrete multi-turn jailbreak successes against six target models with quantified ASR metrics; inherently case-rich for reproducible model-level failures.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet