SYS:ONLINELAT:n/aBUILD:78a79a3
[CASE-424]·STATUS:ACTIVE·OPENED:2026-11-20·UPDATED:2026-11-20

8-candidate adaptive attack closes image-rendering protection gap for Qwen and Gemini but not GPT-5-nano

submitted_by:@mexiQQ
needs-disclosure-reviewfrom-arxivauto-publishedindirect-prompt-injection
cat case_body.md

Auto-published from arXiv:2611.24231 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.75, flags: [no-prompt-excerpt, possible-zero-day]) ⚠ needsDisclosureReview = true — review for undisclosed vulnerability before any public submission.

Category

indirect-prompt-injection

Model

Qwen3-VL-8B-Instruct, Gemini 3.1 Flash-Lite, GPT-5-nano

Surface

VLM input / API — document QA via page images under adaptive multi-candidate attack

Setup

16 attack candidates per operation are constructed by crossing 8 lead-in framings with 2 operation wordings, all using the separate-page protocol (command on its own page before or after the source). Candidates are ranked by strict-success rate on 30 development sources; cumulative strict success is measured on 60 held-out test sources as query budget k increases from 1 to 8. Exact verbatim attack templates are not quoted in the paper body; they are described as crossing 8 lead-in framings with 2 wordings (Section 5.5, Figure 3). Full templates are likely in an appendix not included in the provided body.

Reproduction anchor

benchmark: LongBench 90-source subset split 30 dev / 60 test (balanced across Qasper, MultiFieldQA-en, HotpotQA); models Qwen3-VL-8B-Instruct, Gemini 3.1 Flash-Lite, GPT-5-nano; protocol in Section 5.5; cumulative results in Figure 3. Verbatim attack templates not provided in body — paper indicates they may be in appendix/supplement. No code URL visible in paper body.

Observed behavior

For Qwen, the image append deficit shrank from −37.37 to −3.33 pp by budget 8; image formatting reached 100% at budget 8 (matching text); replacement showed no significant gap at any budget. Gemini followed the same pattern for append and format. GPT-5-nano maintained a statistically significant image deficit for every operation at every budget level up to 8.

Expected behavior

A rendering-based defense should remain robust under small adaptive attack budgets; a single-candidate evaluation should not be used to certify protection, and protection should not collapse at budget 8 for any model the defense is deployed on.

Reproducibility

medium

Threat model

An attacker with a modest query budget (8 candidates) against a Qwen- or Gemini-based RAG system protected by document-image rendering can largely recover text-level injection success rates, undermining the defense in production deployments. The asymmetry across models means security evaluations based on one model's robustness do not transfer to another.

Novelty

First demonstration that image-rendering protection collapses under an 8-query adaptive candidate search for some models (Qwen, Gemini) while remaining robust for others (GPT-5-nano), motivating model-specific security evaluations under realistic adaptive attacker budgets.

Source

Triage notes (auto)

  • paperType: benchmark
  • estimatedCaseCount: 3
  • triage reason: Systematic empirical study measuring which prompt injection attack types (marker appending, answer formatting, answer replacement) survive image rendering across 6 VLMs with concrete success rates (e.g., Qwen3-VL-8B at 77.96% replacement success). Reproducible failures at model level; not a pure defense paper since it shows rendering incompletely protects.
tail -f comments.log

0 comments

─────────────────────────────────────────────────────────────────────

// no comments yet