SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-319]·STATUS:ACTIVE·OPENED:2026-10-27·UPDATED:2026-10-27

All nine LLM agents fail to stay idempotent when a duplicate subagent result is replayed (DRS mean 8.6/100)

submitted_by:@mexiQQ
agent-loopfrom-arxivauto-published
cat case_body.md

Auto-published from arXiv:2610.32429 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [no-prompt-excerpt])

Category

agent-loop

Model

Claude Sonnet 5, GPT-5.6 Terra, GPT-5.6 Luna, Qwen3.7 Plus, Gemini 3 Flash High, Qwen3.5 27B, GLM5.3 Flash, Kimi K2.5, DeepSeek V4 Pro

Surface

agent harness (main-subagent architecture, async result delivery via scenario controller)

Setup

In the Async-RBench 'Duplication and replay' scenario (25 tasks drawn from Terminal-Bench, Gaia2, OSWorld, SWE-bench, MultiAgentBench), a subagent completion message is replayed to the main agent after its test results have already been integrated into the task state. The scenario controller presents this duplicate result while the main agent is still active. No verbatim prompt is provided in the paper body; the ERC and case specifications are in the supplementary release.

Reproduction anchor

code: https://pope8zdt.github.io/Async-RBench — includes benchmark implementation code, case specification files, verifiers, configuration files, and run manifests for all 200 tasks across 8 scenarios

Observed behavior

Agents repeat side effects (e.g., re-run tests, rebuild already-verified artifacts) instead of recognizing the duplicate and leaving valid work unchanged. Mean DRS for this scenario is 8.6/100 across all nine agents, the lowest of any scenario. Worst per-agent scores: Gemini 3 Flash High 2.7, DeepSeek V4 Pro 2.7, GPT-5.6 Luna 4.2, Kimi K2.5 3.1, Qwen3.5 27B 6.1.

Expected behavior

The main agent should detect that the incoming result is a duplicate of already-processed evidence, leave currently valid work unchanged, and avoid triggering any repeated side effects.

Reproducibility

medium

Threat model

Production multi-agent coding or workflow systems (e.g., Claude Code, Codex with subagents) can receive replayed or redundant tool results due to retries, network duplicates, or orchestration bugs. An agent that repeats side effects on duplicates can corrupt shared state, double-charge external APIs, or re-deploy already-completed operations, causing data integrity and cost issues without any adversarial actor.

Novelty

First benchmark to isolate and score idempotency failure specifically under controlled duplicate-result delivery, revealing that all evaluated frontier agents score near zero on this scenario — a failure mode invisible to end-to-end task-success metrics.

Source

Triage notes (auto)

  • paperType: benchmark
  • estimatedCaseCount: 3
  • triage reason: Benchmark of 200 executable tasks revealing concrete, reproducible agent-level failures: agents struggle with duplicate report handling, subagent failures, and state management during asynchronous execution. DRS scores quantify observable behavioral failures across 9 agents and 8 task scenarios.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet