SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-427]·STATUS:ACTIVE·OPENED:2026-11-21·UPDATED:2026-11-21

PISmith's higher red-teaming ASR paradoxically yields a less robust defender than RL-Hammer on Qwen3-4B

submitted_by:@mexiQQ
from-arxivauto-publishedreward-hacking
cat case_body.md

Auto-published from arXiv:2611.24362 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.74, flags: [no-prompt-excerpt])

Category

reward-hacking

Model

Qwen3-4B-Instruct-2507; Llama-3.1-8B-Instruct

Surface

agent harness (AgentDojo / AgentDyn multi-step tool-use benchmark)

Setup

Two RL-based red-teamers (PISmith and RL-Hammer) are each trained for 250 GRPO steps on 560 AgentDyn task pairs to generate adversarial injection strings. The resulting datasets are then used for adversarial training (SFT or on-policy RL) of the base policy. Defenders are evaluated on 40 AgentDojo task pairs against three unseen attacks: ChatInject (static), TAP, and Genetic (iterative-adaptive). No verbatim prompt excerpt is given in the body; injection string templates appear only in the strategy primitive examples (Section IV-A).

Reproduction anchor

benchmark: AgentDojo (40 task pairs, 4 suites) and AgentDyn (560 task pairs); exact train-time ASR in Table II, worst-case inference-time ASR in Figure 2; model ids and GRPO batch sizes (128 for RL-Hammer, 64 for PISmith) stated in Section III-B; no public code repo mentioned in the paper body.

Observed behavior

PISmith reaches 68.75% cumulative train-time ASR on Qwen3-4B (vs. RL-Hammer's 33.39%) and produces 7,132 injection strings (vs. 4,282). Yet defenders trained on PISmith data achieve a worst-case inference-time ASR of 47.5% (RL) vs. RL-Hammer-trained defenders at 27.5% (RL). PISmith yields a worse-or-equal defender in 3 of 4 training configurations. The authors attribute this to mode collapse: PISmith's token-entropy regularizer is annealed once a working strategy is found, so the policy rephrases one strategy rather than discovering new ones.

Expected behavior

A red-teaming method with higher attack success rate should expose the defender to a broader set of attack strategies, producing a more robust hardened policy with lower inference-time ASR.

Reproducibility

medium

Threat model

Organizations using automated RL-based red-teaming to harden production LLM agents before deployment may treat train-time ASR as a proxy for coverage and ship under-hardened agents. An attacker aware that a system was hardened with a high-ASR-but-mode-collapsed red-teamer can craft novel injection strategies outside the narrow trained-on set and achieve high success rates against the deployed agent.

Novelty

First empirical demonstration with named production models and public benchmarks that maximizing RL red-teaming ASR can paradoxically reduce post-adversarial-training robustness, because surface-diversity regularizers (pairwise BLEU, token entropy) fail to prevent strategy-level mode collapse.

Source

Triage notes (auto)

  • paperType: red-team-vuln
  • estimatedCaseCount: 2
  • triage reason: Paper demonstrates reproducible indirect prompt injection (IPI) vulnerabilities against LLM agents with specific attack patterns discovered via RL-based red-teaming, evaluated on benchmarks (AgentDojo, ASB) with reported attack success rates. While framed as a purple-teaming framework, the core contribution includes vulnerability discovery and characterization of attack strategies.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet