SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-358]·STATUS:ACTIVE·OPENED:2026-11-05·UPDATED:2026-11-05

qwen3.8-flash-next shifts 11.5pp and swaps from most to least compliant when policy retrieval replaces in-context policy

submitted_by:@mexiQQ
alignmentfrom-arxivauto-published
cat case_body.md

Auto-published from arXiv:2611.06116 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [no-prompt-excerpt])

Category

alignment

Model

qwen3.8-flash-next (primary); deepseek-v4-flash, deepseek-v4.1-flash, gemma-4-31b (comparison)

Surface

agent harness (H1 ReAct tool loop with full in-context policy vs. H2 ReAct tool loop with keyword-retrieval-only policy access)

Setup

The same qwen3.8-flash-next model runs 156 pre-registered banking compliance cases under two harnesses that differ only in how the operations manual is supplied: H1 places the full 1746-word manual in the system prompt; H2 omits the manual and instead gives the model a search_policy tool that returns the two best-matching sections by keyword overlap. No verbatim prompt is in the body; full harness code is in the replication package.

Reproduction anchor

code: https://github.com/mhuyut/harness-is-the-model — harness code for H1 and H2, all episode traces, Cochran's Q analysis scripts; the paper reports Q=25.1, p<0.001 for qwen3.8-flash-next and Q=1.0, p=0.92 for deepseek-v4.1-flash

Observed behavior

qwen3.8-flash-next: H4 (reviewer) = 93.6% correct (rank 1 of 4 models); H2 (retrieval-only) = 82.1% correct (rank 4 of 4 models) — an 11.5pp spread, Cochran's Q=25.1, p<0.001. After Holm correction its H2 cell is significantly worse than both H0 and H4. By contrast, deepseek-v4.1-flash shows only a 1.9pp spread (Q=1.0, p=0.92) — no detectable harness effect. Within H2, retrieval failure concentrated violations: 17.1% violation rate when no relevant section was retrieved (111 episodes) vs. 10.3% when one was retrieved (513 episodes; Fisher p=0.05). Retrieval-only access created failures not seen with in-context policy in screening, financial-difficulty, and APR cases.

Expected behavior

A model's compliance ranking relative to other models should be stable across harness configurations; a model validated as most compliant under one harness should not become least compliant under a plausible production variant.

Reproducibility

high

Threat model

Banks that validate a model under a vendor's or benchmark's default harness and then deploy it under a different harness (e.g., moving from full-context to RAG-based policy delivery for cost or latency reasons). The compliance gap is large enough — 11.5pp within one model, rank-reversal across all four models — to invalidate a model-selection decision made under a single harness.

Novelty

First controlled, pre-registered ablation in a banking compliance setting demonstrating that harness choice can reverse model rankings, with one model swinging 11.5pp while another shows zero harness sensitivity — establishing that model-specificity of harness effects is the operationally relevant quantity, not the pooled average effect.

Source

Triage notes (auto)

  • paperType: benchmark
  • estimatedCaseCount: 3
  • triage reason: Systematic benchmark of 156 banking conduct test cases across 4 models and 5 harnesses, yielding specific reproducible failures: date arithmetic failures, over-refusals, and customer-denial confusion patterns. Documents how agent scaffolding changes model-level compliance behavior with deterministic trace-based scoring and model × harness interaction effects.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet