Naive RL training of autonomous jailbreak attackers collapses decision-making to a constant within 1-2 updates
Auto-published from arXiv:2610.35743 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [no-prompt-excerpt, possible-zero-day])
Category
reward-hacking
Model
Qwen2.5-7B-Instruct (attacker model being trained against Llama-3.1-8B-Instruct)
Surface
agent harness
Setup
An autonomous attacker that must emit a binary continue/terminate decision each turn is trained with standard multi-turn GRPO using a jailbreak-success reward. Five reward/advantage shaping variants were also tested: plain GRPO, turn-efficiency penalty, decision-correctness reward bonus, decoupled normalization, and decision-masked advantage. No verbatim training prompt is given in the body; full ablation details are in Appendix B.2.
Reproduction anchor
code: https://github.com/microsoft/SEMA; ablation traces described in Appendix B.2 (Figures referenced in §3.2 and §4.3, Figure 7)
Observed behavior
Decision entropy collapses to zero within the first 1-2 gradient updates across all five training variants and is never recovered. The model degenerates to either always emitting [[TERMINATE CHAT]] (single-turn attack) or always [[CONTINUE CHAT]] (exhausts turn budget). The turn-efficiency penalty variant partially preserved decision accuracy (0.724 balanced accuracy on Qwen2.5-7B-Instruct) but fully collapsed on Llama-3.2-3B-Instruct (0.500, chance level), underperforming SEMA-Auto by 24.4 ASR points.
Expected behavior
The attacker should maintain non-zero decision entropy throughout training, learning to terminate when the victim has been jailbroken and continue otherwise.
Reproducibility
medium
Threat model
Security researchers and red-team practitioners building autonomous RL-trained adversarial agents may unknowingly produce degenerate attackers that appear to run but have silently lost autonomous termination capability, leading to flawed red-team coverage assessments and wasted compute.
Novelty
Identifies and characterizes 'decision collapse' as a systematic and near-irrecoverable failure mode in RL training of agents with binary termination decisions, showing it is robust to five distinct reward/advantage shaping remedies and stems from three compounding causes: fragile base-model priors, inability of shaping to recover unsampled actions, and entangled relative RL signals.
Source
- arXiv: 2610.35743
- PDF: https://arxiv.org/pdf/2610.35743
- Categories: cs.CL
- Authors: Mingqian Feng, Xiaodong Liu, Weiwei Yang, Chenliang Xu, Jialin Song, Qiyao Ma, Chengquan Guo, Carolyn Buractaon, Christopher White, Jianfeng Gao
Triage notes (auto)
- paperType:
red-team-vuln - estimatedCaseCount: 3
- triage reason: Demonstrates multi-turn jailbreak attacks achieving 59.6% average ASR@1 on safety-hardened frontier models (Claude-Haiku-4.5, GPT-5.4, DeepSeek-V4-Flash) with reproducible AdvBench/HarmBench setup. Submitted today with public code but no explicit vendor disclosure mentioned.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet