SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-378]·STATUS:ACTIVE·OPENED:2026-11-07·UPDATED:2026-11-07

Qwen2.5-VL-3B fails on ~43% of multi-H-code ECHA extractions in a regulated safety pipeline

submitted_by:@mexiQQ
hallucinationfrom-arxivauto-published
cat case_body.md

Auto-published from arXiv:2611.07140 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.86, flags: [no-prompt-excerpt])

Category

hallucination

Model

Qwen2.5-VL-3B (qwen2.5vl:3b via Ollama, temperature 0)

Surface

API (Ollama local inference, vision-language model)

Setup

Qwen2.5-VL-3B receives rendered ECHA chemical page image tiles and must return the exact set of H-codes listed for a substance; the prompt instructs it not to use general chemical knowledge and to read only the page. Nine substances were each submitted 10 times. Verbatim prompt not in paper body; it is in the repository.

Reproduction anchor

code: https://github.com/Bruhkay/hip-llm-multiagent-reliability (archived ECHA page snapshots dated 17 August 2026, ground-truth H-codes, full prompt, and JSON output schema)

Observed behavior

Qwen2.5-VL-3B failed on 30 of 70 runs in the multi-statement subdomain (D2, substances with ≥2 H-codes), a 42.9% failure rate. Every failure involved omitting at least one H-code from the returned set. Its marginalized probability-of-failure envelope was [0.339, 0.377] and reliability over 10 future demands was only [0.009, 0.016]—falling below 0.5 reliability after roughly 1–2 demands at the envelope's upper end.

Expected behavior

The agent should return the complete, exact set of H-codes matching the ECHA record; omissions in a regulated safety context constitute reportable failures.

Reproducibility

high

Threat model

Organizations adopting small, locally-hosted open-weight VLMs for cost or data-protection reasons in regulated document-extraction tasks (chemical safety, compliance, pharmaceutical) risk systematic omission of safety-relevant classification fields at rates (~43%) that make the model unsuitable as a primary or backup agent without mandatory human review, yet this failure rate is invisible in single-pass benchmark accuracy.

Novelty

Provides reproducible, run-level quantification of small open-weight VLM failure rates in a regulated structured-extraction task, showing that locally-hosted models can omit safety-critical fields in nearly half of complex multi-field document queries—a failure profile that average benchmark accuracy obscures because all failures are omissions rather than hallucinated codes.

Source

Triage notes (auto)

  • paperType: case-study
  • estimatedCaseCount: 2
  • triage reason: Demonstrates concrete system-level failures in a multi-agent LLM pipeline (chemical-hazard extraction), showing majority voting fails to reliably improve reliability and a 5% voter failure rate erases advantages over single models.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet