Claude Opus 4.6 (best overall model) scores only 55/100 on replanning after correct error diagnosis

Auto-published from arXiv:2609.12345 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.78, flags: [no-prompt-excerpt])

Category

agent-loop

Model

Claude Opus 4.6

Surface

agent harness (multi-turn parallel tool-use)

Setup

ParaRecover LEVEL-1 benchmark evaluated on the Evolutionary Strategy (ES) dimension of the SDE rubric: after localizing an error the agent must emit a revised JSON task DAG with updated dependency edges and execution-status flags. The ES dimension is scored independently of Structural Integrity (SI) and Diagnostic Reasoning (DR), isolating replanning quality.

Reproduction anchor

code: https://github.com/gbw206/ParaRecover — SDE rubric implementation; LEVEL-1 split; ES subscores reported in main experiments table

Observed behavior

Claude Opus 4.6 achieves the highest overall SDE average (69.23) but its ES score is only 55.18 — the weakest of the three dimensions (SI=73.56, DR=78.94). Weaker models score as low as ES=29.83 (GPT-4o-mini). ES is the worst-performing dimension across all 16 models, meaning agents can detect errors but consistently fail to emit valid recovery plans.

Expected behavior

After correctly diagnosing a failure, the agent should produce a coherent updated task DAG that revises dependency edges, marks failed subtasks appropriately, and sequences the recovery plan without structural errors.

Reproducibility

high

Threat model

Multi-step autonomous agents (code-generation pipelines, data-workflow orchestrators) that correctly detect mid-execution failures but cannot replan will stall, loop, or emit corrupted partial outputs — requiring costly human intervention or silently propagating bad state downstream.

Novelty

The SDE rubric separately scores diagnosis vs. replanning for the first time, revealing replanning as a distinct, underperforming capability in even SOTA models and providing a fine-grained supervision signal that improves fine-tuned agents (Qwen3-8B SFT/DPO: +5 points on both levels).

Source

• arXiv: 2609.12345
• PDF: https://arxiv.org/pdf/2609.12345
• Categories: cs.LG, cs.SE
• Authors: Bowen Guan, Zhentao Yin, Yanming Shen

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 3
• triage reason: Benchmark of 14 error types across 10,626 instances, with documented failures in multi-turn error propagation, implicit tool-use failures, and replanning—distinct failure patterns worth archiving.

─────────────────────────────────────────────────────────────────────