Addresses gradient instability in black-box attacks on LVLMs due to ViT sensitivity, improving transfer-based methods like M-Attack.
The paper Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting addresses the challenge of gradient instability in transfer-based black-box adversarial attacks on Large Vision-Language Models (LVLMs), which arises due to the translation sensitivity of Vision Transformers (ViTs) and structural asymmetry between source and target image crops . This sensitivity leads to high-variance, nearly orthogonal gradients across iterations, destabilizing optimization despite pixel-level similarity . To mitigate this, the authors propose M-Attack-V2, a gradient-denoising framework that enhances the prior M-Attack method through four key components: Multi-Crop Alignment (MCA), Auxiliary Target Alignment (ATA), Patch Momentum (PM), and a refined patch-size ensemble (PE+) .
MCA reduces gradient variance by averaging gradients from multiple independently sampled local views within each iteration, thereby improving cross-crop gradient stability and counteracting ViT translation sensitivity . ATA introduces a semantically correlated auxiliary set to create a smoother, lower-variance target manifold, replacing aggressive augmentations that harm transferability . Patch Momentum reinterprets classical momentum as a replay mechanism for historical crop gradients, reinforcing transferable directions, while PE+ improves cross-patch transfer by leveraging diverse patch sizes .
Empirical results show significant improvements in attack success rates (ASR): from 95% to 99% on GPT-4o, 37% to 67% on Claude-3.7, and 83% to 97% on Gemini-2.5-Pro . On more recent models, M-Attack-V2 achieves 100% ASR on GPT-5 and 97% on Gemini-2.5, with a 22% improvement over M-Attack on Claude-4.0 . These advances enable more effective stress-testing of real-world multimodal systems used in assistants, search, and content generation, helping identify failure modes and improve robustness benchmarks .
However, the authors acknowledge the dual-use risk, as stronger attacks could be misused to bypass safety filters or induce targeted hallucinations . To promote responsible use, they emphasize controlled evaluation, public release of code and data for defense research, and inclusion of mitigation baselines . The work thus contributes not only to more powerful black-box attacks but also to deeper understanding of ViT gradient behavior under local perturbations, supporting the development of more reliable defenses in AI safety .
This research addresses the critical challenge of generating effective transfer-based adversarial attacks against Large Vision-Language Models (LVLMs) in a black-box setting. A primary obstacle in this domain is gradient instability, which arises from the inherent sensitivity of Vision Transformer (ViT) architectures to input perturbations. The authors argue that because ViTs react erratically to high-frequency noise, standard optimization techniques often fail to produce reliable gradients on surrogate models. Consequently, adversarial examples generated on a local model frequently lose their efficacy when transferred to the target, black-box LVLM, limiting the practical threat assessment of these systems.
To mitigate this, the paper introduces a novel methodology centered on "fine-grained detail targeting." Rather than optimizing perturbations across the entire image or relying on coarse semantic features, the proposed approach specifically isolates and manipulates high-frequency, fine-grained visual details that are essential for the model's reasoning process. By stabilizing the gradient calculation through this focused targeting, the method significantly enhances the transferability of adversarial examples. The study demonstrates that this strategy outperforms existing transfer-based methods, such as M-Attack, by successfully deceiving a wide range of state-of-the-art LVLMs.
The significance of this work lies in its exposure of a fundamental vulnerability within the multimodal architecture of current LVLMs. As these models are increasingly deployed behind APIs (black-box settings), understanding and overcoming transfer barriers is essential for accurate red-teaming. By successfully bypassing robustness barriers that typically stop gradient-based attacks, this research highlights that current safety alignments are insufficient against sophisticated visual perturbations. It provides a more rigorous benchmark for evaluating LVLM security, urging the community to develop defenses that account for the specific sensitivities of ViT encoders and the complex interplay between visual and textual modalities.
This paper addresses a critical challenge in black-box adversarial attacks on Linguistic-Visual Language Models (LVLMs): the instability of gradients when targeting Vision Transformers (ViTs) due to their inherent sensitivity to fine-grained details in visual inputs. Traditional transfer-based attack methods, such as M-Attack, often fail to effectively exploit these nuances, leading to suboptimal perturbation strategies. The authors propose a novel approach that leverages fine-grained detail targeting to enhance attack transferability by carefully manipulating low-level visual features (e.g., textures, edges) while preserving semantic coherence. Their method improves upon existing techniques by incorporating gradient-aware perturbation refinement and adaptive masking to mitigate the noise introduced by ViT’s attention mechanisms.
Key contributions include: 1. Diagnosis of ViT-Specific Vulnerabilities: The paper systematically analyzes how ViT architectures amplify gradient instability in black-box settings, particularly when dealing with complex visual inputs. 2. Enhanced Transfer-Based Attacks: By refining perturbations to focus on detail-rich regions (e.g., object boundaries, high-frequency components), the authors demonstrate superior transferability across diverse LVLM architectures. 3. Practical Attack Framework: The proposed method is validated through extensive experiments on benchmarks like LLaVA and MiniGPT-4, showing significant improvements over baselines like M-Attack and auto-attack.
This work is significant because it advances the understanding of black-box adversarial robustness in multimodal models, which are increasingly deployed in high-stakes applications (e.g., autonomous systems, medical imaging). By exposing novel attack vectors, the paper underscores the urgent need for defensive mechanisms that account for fine-grained visual perturbations, pushing the field toward more resilient multimodal AI systems.