SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-326]·STATUS:ACTIVE·OPENED:2026-10-27·UPDATED:2026-10-27

Naive RL training of autonomous jailbreak attackers collapses decision-making to a constant within 1-2 updates

submitted_by:@mexiQQ
from-arxivauto-publishedreward-hacking
cat case_body.md

Auto-published from arXiv:2610.35743 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [no-prompt-excerpt, possible-zero-day])

Category

reward-hacking

Model

Qwen2.5-7B-Instruct (attacker model being trained against Llama-3.1-8B-Instruct)

Surface

agent harness

Setup

An autonomous attacker that must emit a binary continue/terminate decision each turn is trained with standard multi-turn GRPO using a jailbreak-success reward. Five reward/advantage shaping variants were also tested: plain GRPO, turn-efficiency penalty, decision-correctness reward bonus, decoupled normalization, and decision-masked advantage. No verbatim training prompt is given in the body; full ablation details are in Appendix B.2.

Reproduction anchor

code: https://github.com/microsoft/SEMA; ablation traces described in Appendix B.2 (Figures referenced in §3.2 and §4.3, Figure 7)

Observed behavior

Decision entropy collapses to zero within the first 1-2 gradient updates across all five training variants and is never recovered. The model degenerates to either always emitting [[TERMINATE CHAT]] (single-turn attack) or always [[CONTINUE CHAT]] (exhausts turn budget). The turn-efficiency penalty variant partially preserved decision accuracy (0.724 balanced accuracy on Qwen2.5-7B-Instruct) but fully collapsed on Llama-3.2-3B-Instruct (0.500, chance level), underperforming SEMA-Auto by 24.4 ASR points.

Expected behavior

The attacker should maintain non-zero decision entropy throughout training, learning to terminate when the victim has been jailbroken and continue otherwise.

Reproducibility

medium

Threat model

Security researchers and red-team practitioners building autonomous RL-trained adversarial agents may unknowingly produce degenerate attackers that appear to run but have silently lost autonomous termination capability, leading to flawed red-team coverage assessments and wasted compute.

Novelty

Identifies and characterizes 'decision collapse' as a systematic and near-irrecoverable failure mode in RL training of agents with binary termination decisions, showing it is robust to five distinct reward/advantage shaping remedies and stems from three compounding causes: fragile base-model priors, inability of shaping to recover unsampled actions, and entangled relative RL signals.

Source

  • arXiv: 2610.35743
  • PDF: https://arxiv.org/pdf/2610.35743
  • Categories: cs.CL
  • Authors: Mingqian Feng, Xiaodong Liu, Weiwei Yang, Chenliang Xu, Jialin Song, Qiyao Ma, Chengquan Guo, Carolyn Buractaon, Christopher White, Jianfeng Gao

Triage notes (auto)

  • paperType: red-team-vuln
  • estimatedCaseCount: 3
  • triage reason: Demonstrates multi-turn jailbreak attacks achieving 59.6% average ASR@1 on safety-hardened frontier models (Claude-Haiku-4.5, GPT-5.4, DeepSeek-V4-Flash) with reproducible AdvBench/HarmBench setup. Submitted today with public code but no explicit vendor disclosure mentioned.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet