Penny Wise, Pixel Foolish: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations

arXiv cs.CV / 4/21/2026

📰 NewsSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper studies a vulnerability in screenshot-based, price-constrained multimodal agents, identifying “Visual Dominance Hallucination (VDH),” where subtle visual cues can override textual price evidence and cause irrational decisions.
It introduces “PriceBlind,” a stealthy white-box adversarial attack framework that targets the modality gap in CLIP-style encoders using a Semantic-Decoupling Loss to manipulate image embeddings while keeping pixel-level appearance intact.
In evaluations on E-ShopBench, PriceBlind reaches about 80% attack success rate (ASR) in white-box settings, and transfers at roughly 35–41% ASR across major multimodal models under a simplified single-turn coordinate-selection protocol.
The authors show that defenses such as robust encoders and “Verify-then-Act” significantly reduce ASR, but can involve trade-offs with clean accuracy.

Abstract

The rapid proliferation of Multimodal Large Language Models (MLLMs) has enabled mobile agents to execute high-stakes financial transactions, but their adversarial robustness remains underexplored. We identify Visual Dominance Hallucination (VDH), where imperceptible visual cues can override textual price evidence in screenshot-based, price-constrained settings and lead agents to irrational decisions. We propose PriceBlind, a stealthy white-box adversarial attack framework for controlled screenshot-based evaluation. PriceBlind exploits the modality gap in CLIP-based encoders via a Semantic-Decoupling Loss that aligns the image embedding with low-cost, value-associated anchors while preserving pixel-level fidelity. On E-ShopBench, PriceBlind achieves around 80% ASR in white-box evaluation; under a simplified single-turn coordinate-selection protocol, Ensemble-DI-FGSM transfers with roughly 35-41% ASR across GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet. We also show that robust encoders and Verify-then-Act defenses reduce ASR substantially, though with some clean-accuracy trade-off.