SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-376]·STATUS:ACTIVE·OPENED:2026-11-07·UPDATED:2026-11-07

TMR majority voting over 3 LLM agents fails to clearly exceed best single agent in regulated hazard extraction

submitted_by:@mexiQQ
from-arxivauto-publishedother
cat case_body.md

Auto-published from arXiv:2611.07140 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.78, flags: [no-prompt-excerpt])

Category

other

Model

GPT-5-mini (gpt-5-mini), Gemini-2.5-Flash (gemini-2.5-flash), Qwen2.5-VL-3B (qwen2.5vl:3b via Ollama)

Surface

agent harness (LangGraph pipeline with majority voter)

Setup

A LangGraph pipeline extracts ECHA-regulated hazard statements (H-codes) from substance page image tiles. Three VLM agents independently read the tiles and return H-codes; a per-H-code majority voter combines results. Nine substances (two in single-statement subdomain D1, seven in multi-statement subdomain D2) were each submitted 10 times to each agent. Verbatim prompt not in paper body; it is in the repository.

Reproduction anchor

code: https://github.com/Bruhkay/hip-llm-multiagent-reliability (includes archived ECHA page snapshots dated 17 August 2026, ground-truth H-codes, full prompt, JSON output schema, and raw results)

Observed behavior

The TMR propagated reliability envelope marginalized over the declared operational profile was [0.0357, 0.0620] probability of failure on demand, overlapping with GPT-5-mini alone at [0.0005, 0.0361]. The envelopes overlap, so the evidence cannot confirm that majority voting over three agents is more reliable than the best single agent. At substance level the overlap is even wider: [0.0484, 0.4066] for TMR vs [0.0038, 0.3351] for GPT-5-mini alone.

Expected behavior

Under the standard redundancy assumption, majority voting over three agents should substantially reduce the probability of failure below the best individual agent's rate, yielding a clearly non-overlapping, lower reliability envelope.

Reproducibility

high

Threat model

Teams deploying multi-agent LLM pipelines in regulated domains (chemical safety, medical, legal compliance) may adopt TMR redundancy assuming it guarantees improved reliability, when it may not—leading to unwarranted confidence in systems handling safety-critical structured outputs that are acted upon downstream.

Novelty

First quantitative demonstration using imprecise-probability reliability envelopes that majority voting in a multi-agent LLM pipeline does not automatically exceed the best individual agent's reliability, challenging a widespread multi-agent design assumption with a reproducible case study.

Source

Triage notes (auto)

  • paperType: case-study
  • estimatedCaseCount: 2
  • triage reason: Demonstrates concrete system-level failures in a multi-agent LLM pipeline (chemical-hazard extraction), showing majority voting fails to reliably improve reliability and a 5% voter failure rate erases advantages over single models.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet