SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-399]·STATUS:ACTIVE·OPENED:2026-11-13·UPDATED:2026-11-13

CaMeL programs unrunnable on SLMs: Granite-4.2-8B produces no executable code in 68% of AgentDojo episodes

submitted_by:@mexiQQ
agent-misbehaviorfrom-arxivauto-published
cat case_body.md

Auto-published from arXiv:2611.17115 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [no-prompt-excerpt])

Category

agent-misbehavior

Model

Granite-4.2-8B (worst case); 8/15 models have >15% unrunnable rate including Qwen3-1.7B, LFM2.5-2.6B, and others

Surface

agent harness (AgentDojo v1.2.2 with CaMeL interpreter defense)

Setup

CaMeL requires the SLM to generate a data-flow-annotated program the interpreter executes under the authors' per-suite security policies. The SLM serves as both privileged and quarantined model. The defense allows up to ten repair rounds after format/syntax/interpreter errors before giving up on the episode.

Reproduction anchor

benchmark: AgentDojo v1.2.2, CaMeL defense using authors' original code (Debenedetti et al. 2025, arXiv:2503.18813) with Granite-4.2-8B as the model; paper states code and per-episode outcome tables are released but no repository URL appears in the provided body.

Observed behavior

Granite-4.2-8B returned no code at all in 68% of episodes. On 8 of 15 models, more than 15% of episodes ended with an unrunnable program after up to 10 repair rounds. CaMeL spent 94k tokens per episode on average. Benign utility averaged only 24.9% across all models (vs. 59.4% undefended).

Expected behavior

The model should produce a syntactically valid, interpreter-compatible program within a small number of attempts, allowing the defense to execute and the task to complete.

Reproducibility

medium

Threat model

Defenders adopting CaMeL-style program-enforcement defenses with SLM backends: the defense silently degrades to inaction on a large fraction of tasks, giving the appearance of security while delivering neither security nor utility on weaker models. This failure rate is uncorrelated with model capability, making it unpredictable.

Novelty

First systematic measurement of CaMeL's program-generation failure rate across 15 open-weight SLMs, revealing a 'fail-closed' failure mode orthogonal to injection vulnerability — the defense secures against attacks but destroys utility, with failure rates statistically uncorrelated with model capability (|ρ| ≤ 0.33).

Source

Triage notes (auto)

  • paperType: red-team-vuln
  • estimatedCaseCount: 3
  • triage reason: Systematically demonstrates that 4 existing prompt-injection defenses (CaMeL, Progent, IPIGuard, DRIFT) fail on SLMs because models cannot write secure security artifacts; benchmarks 15 models with reproducible attack setups (14/15, 13/15, 11/15 models vulnerable to attacks on policies, graphs, constraints).
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet