{"ID":2822919,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2601.01592","arxiv_id":"2601.01592","title":"OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs","abstract":"The rapid integration of Multimodal Large Language Models (MLLMs) into critical applications is increasingly hindered by persistent safety vulnerabilities. However, existing red-teaming benchmarks are often fragmented, limited to single-turn text interactions, and lack the scalability required for systematic evaluation. To address this, we introduce OpenRT, a unified, modular, and high-throughput red-teaming framework designed for comprehensive MLLM safety evaluation. At its core, OpenRT architects a paradigm shift in automated red-teaming by introducing an adversarial kernel that enables modular separation across five critical dimensions: model integration, dataset management, attack strategies, judging methods, and evaluation metrics. By standardizing attack interfaces, it decouples adversarial logic from a high-throughput asynchronous runtime, enabling systematic scaling across diverse models. Our framework integrates 37 diverse attack methodologies, spanning white-box gradients, multi-modal perturbations, and sophisticated multi-agent evolutionary strategies. Through an extensive empirical study on 20 advanced models (including GPT-5.2, Claude 4.5, and Gemini 3 Pro), we expose critical safety gaps: even frontier models fail to generalize across attack paradigms, with leading models exhibiting average Attack Success Rates as high as 49.14%. Notably, our findings reveal that reasoning models do not inherently possess superior robustness against complex, multi-turn jailbreaks. By open-sourcing OpenRT, we provide a sustainable, extensible, and continuously maintained infrastructure that accelerates the development and standardization of AI safety.","short_abstract":"The rapid integration of Multimodal Large Language Models (MLLMs) into critical applications is increasingly hindered by persistent safety vulnerabilities. However, existing red-teaming benchmarks are often fragmented, limited to single-turn text interactions, and lack the scalability required for systematic evaluation...","url_abs":"https://arxiv.org/abs/2601.01592","url_pdf":"https://arxiv.org/pdf/2601.01592v2","authors":"[\"Xin Wang\",\"Yunhao Chen\",\"Juncheng Li\",\"Yixu Wang\",\"Yang Yao\",\"Tianle Gu\",\"Jie Li\",\"Yan Teng\",\"Yingchun Wang\",\"Xia Hu\"]","published":"2026-01-04T16:41:33Z","proceeding":"cs.CR","tasks":"[\"cs.CR\",\"cs.CV\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}