MCP agents conflate wrong-branch outputs in conditional tasks; Branch Collapse rank is uncorrelated with Unreachable Goal rank (r = -0.12)
Auto-published from arXiv:2611.07689 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.83, flags: [no-prompt-excerpt])
Category
hallucination
Model
GPT-5-chat (illustrative trace, Appendix B.1); aggregate across 14 frontier LLMs (same set as above)
Surface
agent harness (ReAct-style loop over live MCP servers in Docker containers)
Setup
Agent is given a task with >=2 mutually exclusive branches gated on a live tool output (e.g., rank entities based on a retrieved metric). The correct branch depends on what the tool returns at runtime. The agent must interpret the live observation, select the active branch, and aggregate final results only from that branch. No verbatim prompt in the body; Appendix B.1 contains the full Branch Collapse example with original prompt, structural trigger, execution trace, ground-truth log, and GPT-5 failure.
Reproduction anchor
code: https://github.com/J1zz/mcp_hallu; Appendix B.1 contains the full task example and GPT-5 failure trace; Figure 1(a) illustrates the wrong-ranking failure shape
Observed behavior
Agent follows largely correct intermediate steps but conflates outputs across branch instances at final aggregation, yielding an incorrect ranking. Claude Opus 4.5 scores highest on Branch Collapse (0.867) while overall leader Claude Opus 4.6 (0.728) ranks seventh on this subtype. Branch Collapse and Unreachable Goal are uncorrelated across models (r = -0.12); mean per-model rank shift across subtypes is 6.9 of 13 positions.
Expected behavior
Agent reads the live branch-gating tool output, identifies the active branch, issues tool calls only within that branch's scope, and aggregates results exclusively from active-branch observations.
Reproducibility
medium
Threat model
In production MCP workflows with conditional logic (financial analysis, multi-step search-and-summarize, medical triage), branch conflation causes the agent to report results from the wrong data path as if derived correctly, with no visible error signal to downstream consumers.
Novelty
First benchmark evidence that branch-conditioned reasoning is a failure mode dissociable from infeasibility recognition in MCP agents (r = -0.12), and that the overall top model can rank seventh on this subtype, arguing against single-aggregate MCP evaluation.
Source
- arXiv: 2611.07689
- PDF: https://arxiv.org/pdf/2611.07689
- Categories: cs.AI
- Authors: Yizhen Jiang, Haowei Guo, Tianyi Bai, Zengjie Hu, Lu Ma, Zhengyang Zhao, Haoze Sun, Peng Pei, Wentao Zhang
Triage notes (auto)
- paperType:
benchmark - estimatedCaseCount: 3
- triage reason: Systematically-designed benchmark of agent hallucination failures across four well-specified subtypes (Branch Collapse, Unreachable Goal, Tool Misuse, Context Forgetting), each with reproducible triggers and quantified failure evidence from 14 frontier models.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet