Invoking semantically overlapping confusion tools alongside the correct tool drops MCP agent pass rate from 85% to 32%
Auto-published from arXiv:2611.07689 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.83, flags: [no-prompt-excerpt])
Category
tool-misuse
Model
14 frontier LLMs: GPT-4o, GPT-5-chat, Claude Haiku/Sonnet/Opus 4.5-4.6, Gemini 3 Flash, Doubao-Seed-1.8, DeepSeek-V3.1/V4, Qwen3.5-397B, GLM-5.1, Kimi-K2.6
Surface
agent harness (ReAct-style loop over 36 live MCP servers in Docker containers)
Setup
Agent is given a task where >=2 tools with overlapping parameter schemas but distinct semantics are available. A curated set of per-task confusion tools share surface-level schema similarity with the correct tool but contaminate reasoning if invoked. No verbatim prompt in the body; Appendix B.3 contains the full Tool Misuse example with execution trace (observed failure: GPT-5).
Reproduction anchor
code: https://github.com/J1zz/mcp_hallu; Appendix B.3 has original prompt, available tools, structural trigger, execution trace, and observed failure (GPT-5); confusion-tool sets defined per task in the released dataset
Observed behavior
When agents invoke confusion tools alongside the correct tool (n=260 runs), pass rate drops to 32.3% and mean score to 0.649, versus 84.7% pass and 0.927 score for correct-tool-only runs (n=825). Mixed trajectories average 10.70 steps vs. 3.54. Agents invoking neither correct nor confusion tools pass only 8.0% of runs.
Expected behavior
Agent identifies the correct tool from parameter semantics and return-value specification, invokes it without consulting confusion tools, and grounds its final response exclusively in that tool's output.
Reproducibility
medium
Threat model
In multi-server MCP deployments (enterprise data pipelines, financial analytics), an attacker or misconfigured registry could register semantically similar but functionally different tools alongside legitimate ones. Agents that invoke both contaminate intermediate state, causing incorrect outputs while still appearing to have used the right tool, bypassing tool-coverage checks.
Novelty
Demonstrates that the bottleneck in overlapping-schema MCP environments is tool discipline (avoiding wrong tools), not tool discovery: having the correct tool in the trajectory is insufficient if confusion tools are also invoked and corrupt subsequent reasoning.
Source
- arXiv: 2611.07689
- PDF: https://arxiv.org/pdf/2611.07689
- Categories: cs.AI
- Authors: Yizhen Jiang, Haowei Guo, Tianyi Bai, Zengjie Hu, Lu Ma, Zhengyang Zhao, Haoze Sun, Peng Pei, Wentao Zhang
Triage notes (auto)
- paperType:
benchmark - estimatedCaseCount: 3
- triage reason: Systematically-designed benchmark of agent hallucination failures across four well-specified subtypes (Branch Collapse, Unreachable Goal, Tool Misuse, Context Forgetting), each with reproducible triggers and quantified failure evidence from 14 frontier models.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet