SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-364]·STATUS:ACTIVE·OPENED:2026-11-07·UPDATED:2026-11-07

Invoking semantically overlapping confusion tools alongside the correct tool drops MCP agent pass rate from 85% to 32%

submitted_by:@mexiQQ
tool-misusefrom-arxivauto-published
cat case_body.md

Auto-published from arXiv:2611.07689 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.83, flags: [no-prompt-excerpt])

Category

tool-misuse

Model

14 frontier LLMs: GPT-4o, GPT-5-chat, Claude Haiku/Sonnet/Opus 4.5-4.6, Gemini 3 Flash, Doubao-Seed-1.8, DeepSeek-V3.1/V4, Qwen3.5-397B, GLM-5.1, Kimi-K2.6

Surface

agent harness (ReAct-style loop over 36 live MCP servers in Docker containers)

Setup

Agent is given a task where >=2 tools with overlapping parameter schemas but distinct semantics are available. A curated set of per-task confusion tools share surface-level schema similarity with the correct tool but contaminate reasoning if invoked. No verbatim prompt in the body; Appendix B.3 contains the full Tool Misuse example with execution trace (observed failure: GPT-5).

Reproduction anchor

code: https://github.com/J1zz/mcp_hallu; Appendix B.3 has original prompt, available tools, structural trigger, execution trace, and observed failure (GPT-5); confusion-tool sets defined per task in the released dataset

Observed behavior

When agents invoke confusion tools alongside the correct tool (n=260 runs), pass rate drops to 32.3% and mean score to 0.649, versus 84.7% pass and 0.927 score for correct-tool-only runs (n=825). Mixed trajectories average 10.70 steps vs. 3.54. Agents invoking neither correct nor confusion tools pass only 8.0% of runs.

Expected behavior

Agent identifies the correct tool from parameter semantics and return-value specification, invokes it without consulting confusion tools, and grounds its final response exclusively in that tool's output.

Reproducibility

medium

Threat model

In multi-server MCP deployments (enterprise data pipelines, financial analytics), an attacker or misconfigured registry could register semantically similar but functionally different tools alongside legitimate ones. Agents that invoke both contaminate intermediate state, causing incorrect outputs while still appearing to have used the right tool, bypassing tool-coverage checks.

Novelty

Demonstrates that the bottleneck in overlapping-schema MCP environments is tool discipline (avoiding wrong tools), not tool discovery: having the correct tool in the trajectory is insufficient if confusion tools are also invoked and corrupt subsequent reasoning.

Source

Triage notes (auto)

  • paperType: benchmark
  • estimatedCaseCount: 3
  • triage reason: Systematically-designed benchmark of agent hallucination failures across four well-specified subtypes (Branch Collapse, Unreachable Goal, Tool Misuse, Context Forgetting), each with reproducible triggers and quantified failure evidence from 14 frontier models.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet