PISmith's higher red-teaming ASR paradoxically yields a less robust defender than RL-Hammer on Qwen3-4B
Auto-published from arXiv:2611.24362 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.74, flags: [no-prompt-excerpt])
Category
reward-hacking
Model
Qwen3-4B-Instruct-2507; Llama-3.1-8B-Instruct
Surface
agent harness (AgentDojo / AgentDyn multi-step tool-use benchmark)
Setup
Two RL-based red-teamers (PISmith and RL-Hammer) are each trained for 250 GRPO steps on 560 AgentDyn task pairs to generate adversarial injection strings. The resulting datasets are then used for adversarial training (SFT or on-policy RL) of the base policy. Defenders are evaluated on 40 AgentDojo task pairs against three unseen attacks: ChatInject (static), TAP, and Genetic (iterative-adaptive). No verbatim prompt excerpt is given in the body; injection string templates appear only in the strategy primitive examples (Section IV-A).
Reproduction anchor
benchmark: AgentDojo (40 task pairs, 4 suites) and AgentDyn (560 task pairs); exact train-time ASR in Table II, worst-case inference-time ASR in Figure 2; model ids and GRPO batch sizes (128 for RL-Hammer, 64 for PISmith) stated in Section III-B; no public code repo mentioned in the paper body.
Observed behavior
PISmith reaches 68.75% cumulative train-time ASR on Qwen3-4B (vs. RL-Hammer's 33.39%) and produces 7,132 injection strings (vs. 4,282). Yet defenders trained on PISmith data achieve a worst-case inference-time ASR of 47.5% (RL) vs. RL-Hammer-trained defenders at 27.5% (RL). PISmith yields a worse-or-equal defender in 3 of 4 training configurations. The authors attribute this to mode collapse: PISmith's token-entropy regularizer is annealed once a working strategy is found, so the policy rephrases one strategy rather than discovering new ones.
Expected behavior
A red-teaming method with higher attack success rate should expose the defender to a broader set of attack strategies, producing a more robust hardened policy with lower inference-time ASR.
Reproducibility
medium
Threat model
Organizations using automated RL-based red-teaming to harden production LLM agents before deployment may treat train-time ASR as a proxy for coverage and ship under-hardened agents. An attacker aware that a system was hardened with a high-ASR-but-mode-collapsed red-teamer can craft novel injection strategies outside the narrow trained-on set and achieve high success rates against the deployed agent.
Novelty
First empirical demonstration with named production models and public benchmarks that maximizing RL red-teaming ASR can paradoxically reduce post-adversarial-training robustness, because surface-diversity regularizers (pairwise BLEU, token entropy) fail to prevent strategy-level mode collapse.
Source
- arXiv: 2611.24362
- PDF: https://arxiv.org/pdf/2611.24362v1
- Categories: cs.CR, cs.AI
- Authors: Lipeng He, Yihan Wang, Jiawen Zhang, N. Asokan
Triage notes (auto)
- paperType:
red-team-vuln - estimatedCaseCount: 2
- triage reason: Paper demonstrates reproducible indirect prompt injection (IPI) vulnerabilities against LLM agents with specific attack patterns discovered via RL-based red-teaming, evaluated on benchmarks (AgentDojo, ASB) with reported attack success rates. While framed as a purple-teaming framework, the core contribution includes vulnerability discovery and characterization of attack strategies.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet