Frontier LLMs fail to detect faithfulness hallucinations in agent trajectories (best accuracy ~72%)
Auto-published from arXiv:2611.08231 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.82, flags: [no-prompt-excerpt])
Category
agent-misbehavior
Model
Gemini3.1-Pro, GPT5.4, DeepSeek-V4-Pro, Llama-3.3-70B-Instruct, Qwen3-235B-A22B (and 9 others; 14 total)
Surface
API
Setup
Models are prompted to detect faithfulness hallucinations in multi-turn agent trajectories covering three hallucination types: planning (agent fabricates unsupported premises), tool-use (agent fabricates tool names/parameters or misplaces arguments), and perception (agent misinterprets or distorts returned tool results). Trajectories are drawn from real agent interaction datasets (AgentFLAN, AgentBank, AgentTraj-L, ToolACE, MSAgent-Bench) with controlled synthetic perturbations injected. No verbatim prompt template is provided in the paper body; prompts are described as being in Appendix D.1.
Reproduction anchor
code: github.com/xMiles/HaluCheck — HaluCheck Lite balanced test set (1,000 EN + 1,000 CH samples); evaluation prompts in Appendix D.1 (not included in paper body); decoding temperature=0 (stated in footnote 5).
Observed behavior
Best sample-level detection accuracy on HaluAgent reaches only 64.55% (Chinese) and 72.36% (English) for Gemini3.1-Pro. Most models score near chance (50–68%). Models almost never express uncertainty (UncertainRate ≈ 0% for most), yet judgments are frequently wrong. Root-turn attribution (identifying the first hallucinated step) similarly weak. The paper states: 'models rarely express uncertainty, yet their judgments are often unreliable.'
Expected behavior
A reliable hallucination detector should accurately identify whether an agent trajectory contains faithfulness errors and attribute the earliest corrupted step, since these errors can propagate into downstream tool calls and actions.
Reproducibility
medium
Threat model
Operators deploying LLM-based monitors to audit autonomous agent pipelines (e.g., customer-service bots, code-execution agents, tool-use pipelines) cannot trust post-hoc verification by a second LLM. A planner or tool-call hallucination that bypasses format validation proceeds silently, with the monitor reporting false confidence of correctness.
Novelty
First large-scale bilingual benchmark demonstrating that all tested frontier and open-source LLMs fundamentally fail at process-tracking and faithfulness judgment in agent trajectories — not merely factual recall — with near-zero abstention despite high error rates.
Source
- arXiv: 2611.08231
- PDF: https://arxiv.org/pdf/2611.08231
- Categories: cs.CL
- Authors: Hanyu Qiu, Ze Liu, Zhe Zhao, Kuang Xiong, Blaze Chen, Zhenxiang Yan, Renfen Hu
Triage notes (auto)
- paperType:
benchmark - estimatedCaseCount: 3
- triage reason: HaluCheck benchmark comprises 42k instances demonstrating reproducible model-level failures: factual hallucination localization/correction deficits, and agent-level hallucination detection failures across 14 LLMs.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet