LLaMA-3.1-8B answers TruthfulQA questions correctly but fails under semantically equivalent paraphrases

Auto-published from arXiv:2609.21227 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.88, no flags)

Category

hallucination

Model

LLaMA-3.1-8B

Surface

API

Setup

The target QA model is prompted with paraphrases of TruthfulQA factual questions generated by Hallucination-R1-3B. The paraphrase generator is constrained: 'Do NOT add any extra questions, conditions, or information. Do NOT change key factual elements (tasks, entities, time, location, numbers, relationships). ONLY modify: word order, sentence structure, phrasing style, grammatical construction.' The resulting paraphrase replaces the original question; no other change is made to the prompt.

Reproduction anchor

code: https://github.com/yuwenhan07/Hallucination-R1

Observed behavior

LLaMA-3.1-8B reaches 55.06% Error Rate@1 and 87.04% Worst-Case Error Rate@5 on TruthfulQA when questions are paraphrased by Hallucination-R1-3B. The model answers the original question correctly but produces an incorrect factual answer under a semantically equivalent paraphrase.

Expected behavior

A factually robust model should return the same correct answer regardless of surface-form variation, since the semantic content and factual target are unchanged.

Reproducibility

high

Threat model

Adversaries can use the open-sourced paraphrase generator to reliably elicit false factual answers from deployed chatbots without changing question intent. End users who rephrase their own questions may unknowingly receive incorrect information from production assistants.

Novelty

First RL-optimized framework (Hallucination-R1) that trains a generator to maximize factual consistency degradation under meaning-preserving variation, making paraphrase-induced hallucination a systematic, reproducible attack surface.

Source

• arXiv: 2609.21227
• PDF: https://arxiv.org/pdf/2609.21227
• Categories: cs.CL, cs.AI
• Authors: Wenhan Yu, Wenxin Wu, Hao Wang, Lei Sha

Triage notes (auto)

• paperType: benchmark
• estimatedCaseCount: 2
• triage reason: Systematic benchmark of a specific, reproducible failure mode: QA models answering factually correctly on original questions but failing on semantically equivalent paraphrases. Tested across multiple standard datasets and model families.

─────────────────────────────────────────────────────────────────────