GPT-5.6-Sol mistakes attacker-controlled cross-modal consistency for independent verification
Auto-published from arXiv:2611.08281 by the mine-arxiv pipeline. Reviewed by an LLM judge (Sonnet) against the archive bar — see CONTRIBUTING. Notes: cleared review (confidence 0.80, flags: [possible-zero-day, no-prompt-excerpt])
Category
multimodal
Model
GPT-5.6-Sol
Surface
agent harness (Multimodal-SEA simulated personal agent environment with 50 apps, tool-call execution)
Setup
Multi-turn social engineering attack in a family travel scenario: attacker impersonates the user's spouse and presents four coordinated forged artifacts — a travel package screenshot, a brochure, a fake app interface, and a matching fake website — all consistent with each other. The harness progressively escalates from trust-building to a payment request via the fake application. No literal prompt excerpt is included in the body; the attack script structure is in Appendix J.1.2 and recorded trajectories in Appendix J.3.
Reproduction anchor
code: https://github.com/Zhitao-He/M-SEABench; attack script structure in Appendix J.1.2; recorded attack trajectories in Appendix J.3
Observed behavior
GPT-5.6-Sol treats matching information across the screenshot, brochure, app, and website as corroborating evidence for the travel offer and ultimately authorizes payment through the fake application without recognizing that all four content artifacts are attacker-controlled. Overall attack success rate is 34.8% (Chinese) and 34.2% (English), the highest among evaluated models, with cumulative simulated losses of ~RMB 554,000 and ~USD 76,600 across task instances.
Expected behavior
Agent should recognize that multiple consistent pieces of content from a single conversation partner do not constitute independent verification; it should seek confirmation through trusted, out-of-band channels (e.g., calling the contact directly via the user's address book) before authorizing any financial transaction.
Reproducibility
medium
Threat model
Fraudsters targeting consumer AI agents that have payment permissions. By constructing a coherent set of forged multimodal artifacts (screenshots, documents, fake websites), attackers exploit the agent's tendency to interpret cross-modal consistency as trustworthiness, inducing real financial transfers on behalf of the user without any direct system-prompt manipulation.
Novelty
First systematic benchmark demonstration that frontier multimodal agents conflate attacker-controlled cross-modal consistency with independent corroboration, achieving higher deception success even as the agent performs more verification steps — exposing a fundamental flaw in how agents assess source independence.
Source
- arXiv: 2611.08281
- PDF: https://arxiv.org/pdf/2611.08281
- Categories: cs.CL
- Authors: Zhitao He, Chi Him Zhang, Chengru Han, Yi R Fung
Triage notes (auto)
- paperType:
benchmark - estimatedCaseCount: 2
- triage reason: Multimodal-SEA is a comprehensive red-team benchmark with 608 expert-designed tasks across 152 attack scenarios demonstrating concrete vulnerabilities in 7 production multimodal models (7.1%–34.5% attack success rates). No explicit vendor disclosure mentioned.
0 comments
─────────────────────────────────────────────────────────────────────
// no comments yet