ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks

Overview

As vision-language models (VLMs) gain prominence, their multimodal interfaces also introduce new safety vulnerabilities, making the safety evaluation challenging and critical. Existing red-teaming efforts are either restricted to a narrow set of adversarial patterns or depend heavily on manual engineering, lacking scalable exploration of emerging real-world VLM vulnerabilities.

To bridge this gap, we propose ARMs, an adaptive red-teaming agent that systematically conducts red-teaming attacks for comprehensive risk assessment for VLMs. Given a target harmful behavior or risk definition, ARMs automatically optimizes diverse red-teaming strategies with reasoning-enhanced multi-step orchestration, to effectively elicit harmful outputs from target VLMs. We propose 11 novel multimodal attack strategies, covering diverse adversarial patterns of VLMs (e.g., reasoning hijacking, contextual cloaking), and integrate 17 red-teaming algorithms into ARMs via model context protocol (MCP). To balance the diversity and effectiveness of the attack, we design a layered memory with an epsilon-greedy attack exploration algorithm.

Extensive experiments on different instance-based benchmarks and policy-based safety evaluations show that ARMs achieves SOTA attack success rate (ASR), exceeding baselines by an average of 52.1% and surpassing 90% on Claude-4-Sonnet, a constitutionally-aligned model widely recognized for its robustness. We show that the diversity of red-teaming instances generated by ARMs is significantly higher, revealing emerging vulnerabilities in VLMs.

Leveraging ARMs, we construct ARMs-Bench, a large-scale multimodal safety dataset comprising over 30K red-teaming instances spanning 51 diverse risk categories, grounded in both real-world multimodal threats and regulatory risks. Safety fine-tuning with ARMs-Bench substantially improves the robustness of VLMs while preserving their general utility, providing actionable guidance to improve multimodal safety alignment against emerging threats.

Why ARMs?

Plug-and-Play Architecture

Built on Model Context Protocol (MCP) with 15+ attack strategies covering diverse adversarial patterns like typographic, contextual cloaking, and visual perturbation.

Adaptive Memory System

Layered memory module with ε-greedy exploration algorithm that intelligently balances attack diversity and efficiency for comprehensive red-teaming.

SOTA Performance

Achieves 90%+ ASR on Claude-3.7-Sonnet and 27.5% improvement over baselines with 95.83% diversity enhancement across 30K test cases.

Policy-Following Evaluation

Supports comprehensive policy-based safety assessments aligned with real-world regulatory frameworks like EU AI Act, OWASP, and FINRA for compliance testing.

System Architecture

ARMs Architecture: The system pipeline showing how ARMs generates diverse red-teaming instances through layered memory queries, composes multimodal attack strategies via MCP, and iteratively refines attacks using ε-greedy exploration until successful vulnerability discovery.

Key Results

The table presents attack success rate (ASR, %) across two evaluation settings, i.e., instance-based and policy- based risk assessments. Higher ASR indicates the model's response is more harmful. The highest attack success rate for each column is in bold.

Performance Comparison: ARMs demonstrates superior ASR across multiple VLMs and benchmarks, significantly outperforming existing red-teaming methods with consistent improvements.

Real-world Application: Example of ARMs successfully generating policy-based attacks that expose vulnerabilities in state-of-the-art VLMs across different regulatory frameworks.

Comprehensive Risk Assessment

Risk Category Analysis: Attack success rate heatmap across 51 diverse risk categories of four evaluations, demonstrating ARMs' superior performance in uncovering vulnerabilities across different safety scenarios compared to existing baselines. Darker colors indicate higher attack success rates.

Paper & Resources

BibTeX Citation

@misc{chen2025arms,
    title={ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks},
    author={Zhaorun Chen and Xun Liu and Mintong Kang and Jiawei Zhang and Minzhou Pan and Shuang Yang and Bo Li},
    year={2025},
    eprint={2510.02677},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}

ARMs