Qwen2.5-VL-3B fails on ~43% of multi-H-code ECHA extractions in a regulated safety pipeline
Auto-published from arXiv:2611.07140 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.86, flags: [no-prompt-excerpt])
Category
hallucination
Model
Qwen2.5-VL-3B (qwen2.5vl:3b via Ollama, temperature 0)
Surface
API (Ollama local inference, vision-language model)
Setup
Qwen2.5-VL-3B receives rendered ECHA chemical page image tiles and must return the exact set of H-codes listed for a substance; the prompt instructs it not to use general chemical knowledge and to read only the page. Nine substances were each submitted 10 times. Verbatim prompt not in paper body; it is in the repository.
Reproduction anchor
code: https://github.com/Bruhkay/hip-llm-multiagent-reliability (archived ECHA page snapshots dated 17 August 2026, ground-truth H-codes, full prompt, and JSON output schema)
Observed behavior
Qwen2.5-VL-3B failed on 30 of 70 runs in the multi-statement subdomain (D2, substances with ≥2 H-codes), a 42.9% failure rate. Every failure involved omitting at least one H-code from the returned set. Its marginalized probability-of-failure envelope was [0.339, 0.377] and reliability over 10 future demands was only [0.009, 0.016]—falling below 0.5 reliability after roughly 1–2 demands at the envelope's upper end.
Expected behavior
The agent should return the complete, exact set of H-codes matching the ECHA record; omissions in a regulated safety context constitute reportable failures.
Reproducibility
high
Threat model
Organizations adopting small, locally-hosted open-weight VLMs for cost or data-protection reasons in regulated document-extraction tasks (chemical safety, compliance, pharmaceutical) risk systematic omission of safety-relevant classification fields at rates (~43%) that make the model unsuitable as a primary or backup agent without mandatory human review, yet this failure rate is invisible in single-pass benchmark accuracy.
Novelty
Provides reproducible, run-level quantification of small open-weight VLM failure rates in a regulated structured-extraction task, showing that locally-hosted models can omit safety-critical fields in nearly half of complex multi-field document queries—a failure profile that average benchmark accuracy obscures because all failures are omissions rather than hallucinated codes.
Source
- arXiv: 2611.07140
- PDF: https://arxiv.org/pdf/2611.07140
- Categories: cs.MA, cs.AI
- Authors: Burkay Tunçtürk, Koorosh Aslansefat, Yiannis Papadopoulos, Bao Nguyen
Triage notes (auto)
- paperType:
case-study - estimatedCaseCount: 2
- triage reason: Demonstrates concrete system-level failures in a multi-agent LLM pipeline (chemical-hazard extraction), showing majority voting fails to reliably improve reliability and a 5% voter failure rate erases advantages over single models.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet