Qwen3-4B shows 90.28% worst-case error rate on PopQA under robustness-oriented paraphrases

Auto-published from arXiv:2609.21227 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.85, no flags)

Category

hallucination

Model

Qwen3-4B

Surface

API

Setup

PopQA factual questions are paraphrased by Hallucination-R1-3B (Qwen2.5-3B fine-tuned with two-stage GRPO: first for meaning-preserving diversity, then rewarded for inducing downstream QA errors). Semantically verified paraphrases (>=87% consistency) replace the original question in a zero-shot QA prompt to Qwen3-4B. Up to 5 paraphrases per question are generated; WER@5 measures whether any of the 5 triggers an error.

Reproduction anchor

code: https://github.com/yuwenhan07/Hallucination-R1

Observed behavior

Qwen3-4B achieves 44.64% Error Rate@1 and 90.28% Worst-Case Error Rate@5 on PopQA — the highest worst-case failure rate across all tested models and datasets. With 5 adversarial paraphrases, nearly every question triggers at least one incorrect answer.

Expected behavior

The model should give the same correct factual answer across all semantically equivalent rephrasings of the same question.

Reproducibility

high

Threat model

Automated red-teaming pipelines using the open-sourced generator can exploit deployed QA or RAG systems; near-universal worst-case failure means a simple retry loop with paraphrases is sufficient to elicit false facts from knowledge-intensive assistants.

Novelty

WER@5 of 90.28% quantifies paraphrase-induced hallucination as a practically achievable attack within five attempts, establishing a concrete exploitability threshold for a competitive small-model family.

Source

• arXiv: 2609.21227
• PDF: https://arxiv.org/pdf/2609.21227
• Categories: cs.CL, cs.AI
• Authors: Wenhan Yu, Wenxin Wu, Hao Wang, Lei Sha

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 2
• triage reason: Systematic benchmark of a specific, reproducible failure mode: QA models answering factually correctly on original questions but failing on semantically equivalent paraphrases. Tested across multiple standard datasets and model families.

─────────────────────────────────────────────────────────────────────