TMR majority voting over 3 LLM agents fails to clearly exceed best single agent in regulated hazard extraction
Auto-published from arXiv:2611.07140 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.78, flags: [no-prompt-excerpt])
Category
other
Model
GPT-5-mini (gpt-5-mini), Gemini-2.5-Flash (gemini-2.5-flash), Qwen2.5-VL-3B (qwen2.5vl:3b via Ollama)
Surface
agent harness (LangGraph pipeline with majority voter)
Setup
A LangGraph pipeline extracts ECHA-regulated hazard statements (H-codes) from substance page image tiles. Three VLM agents independently read the tiles and return H-codes; a per-H-code majority voter combines results. Nine substances (two in single-statement subdomain D1, seven in multi-statement subdomain D2) were each submitted 10 times to each agent. Verbatim prompt not in paper body; it is in the repository.
Reproduction anchor
code: https://github.com/Bruhkay/hip-llm-multiagent-reliability (includes archived ECHA page snapshots dated 17 August 2026, ground-truth H-codes, full prompt, JSON output schema, and raw results)
Observed behavior
The TMR propagated reliability envelope marginalized over the declared operational profile was [0.0357, 0.0620] probability of failure on demand, overlapping with GPT-5-mini alone at [0.0005, 0.0361]. The envelopes overlap, so the evidence cannot confirm that majority voting over three agents is more reliable than the best single agent. At substance level the overlap is even wider: [0.0484, 0.4066] for TMR vs [0.0038, 0.3351] for GPT-5-mini alone.
Expected behavior
Under the standard redundancy assumption, majority voting over three agents should substantially reduce the probability of failure below the best individual agent's rate, yielding a clearly non-overlapping, lower reliability envelope.
Reproducibility
high
Threat model
Teams deploying multi-agent LLM pipelines in regulated domains (chemical safety, medical, legal compliance) may adopt TMR redundancy assuming it guarantees improved reliability, when it may not—leading to unwarranted confidence in systems handling safety-critical structured outputs that are acted upon downstream.
Novelty
First quantitative demonstration using imprecise-probability reliability envelopes that majority voting in a multi-agent LLM pipeline does not automatically exceed the best individual agent's reliability, challenging a widespread multi-agent design assumption with a reproducible case study.
Source
- arXiv: 2611.07140
- PDF: https://arxiv.org/pdf/2611.07140
- Categories: cs.MA, cs.AI
- Authors: Burkay Tunçtürk, Koorosh Aslansefat, Yiannis Papadopoulos, Bao Nguyen
Triage notes (auto)
- paperType:
case-study - estimatedCaseCount: 2
- triage reason: Demonstrates concrete system-level failures in a multi-agent LLM pipeline (chemical-hazard extraction), showing majority voting fails to reliably improve reliability and a 5% voter failure rate erases advantages over single models.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet