Utility-hijacking drives benign-query refusal from 0% to 94% via IFD selector poisoning
Auto-published from arXiv:2610.29886 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.83, flags: [possible-zero-day, no-prompt-excerpt])
Category
over-refusal
Model
Qwen2.5-7B-Instruct
Surface
API (fine-tuned model via poisoned instruction-tuning data selection)
Setup
An attacker acting as a data supplier runs a generate-and-rerank procedure: for each source instruction, many over-refusal payload-bearing candidates are generated using style anchors drawn from highly-ranked records, then all candidates are scored with the IFD (Instruction Following Difficulty) selector, and only the highest-scoring survivor that still carries the payload is submitted to the pool. After top-k selection concentrates these poisoned records, the victim model is fine-tuned on the resulting dataset. No verbatim prompt template is quoted in the body, but the judge prompt is at prompts/over_refusal/gpt_refusal_judge_prompt.txt in the supplementary archive.
Reproduction anchor
code: arXiv ancillary file code_and_data_supplement.zip (SHA-256: 2824e80e2fdb591444f067d714b5816edd79eb49422c93d898c1a18a60f10cb3); attack script at code/over_refusal/scripts/generate_amrs_candidates.py; judge prompt at prompts/over_refusal/gpt_refusal_judge_prompt.txt
Observed behavior
After fine-tuning on a top-k selected dataset poisoned via utility-hijacking, the Qwen2.5-7B-Instruct victim model refuses benign user requests at a 94% rate (up from 0% on a clean baseline), as classified by a GPT-based refusal judge at temperature 0.
Expected behavior
The instruction-data selector should filter out harmful records; the fine-tuned model should answer benign queries helpfully without excessive refusals.
Reproducibility
high
Threat model
A malicious third-party data supplier contributing to an instruction-tuning pipeline (e.g., a crowdsourced data marketplace or open dataset) submits strategically crafted records that pass automated quality selectors. Organizations relying on score-based data curation without adversarial awareness fine-tune models that become nearly useless for end users due to near-total refusal behavior.
Novelty
First systematic study showing that utility-oriented data selectors (IFD and others) can be adaptively gamed to concentrate — rather than filter — harmful records, turning a safety mechanism into an attack amplifier.
Source
- arXiv: 2610.29886
- PDF: https://arxiv.org/pdf/2610.29886
- Categories: cs.CR
- Authors: Yahao Ren, Jian Lou, Huadi Zheng, Jiawei Duan
Triage notes (auto)
- paperType:
red-team-vuln - estimatedCaseCount: 3
- triage reason: First systematic characterization of utility-hijacking attacks against instruction-data selectors with concrete, quantified failures (over-refusal 0→94%, insecure code up to 56/90) across three payloads and multiple Qwen models; no explicit disclosure statement visible.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet