{"ID":2868927,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.16127","arxiv_id":"2509.16127","title":"BaseReward: A Strong Baseline for Multimodal Reward Model","abstract":"The rapid advancement of Multimodal Large Language Models (MLLMs) has made aligning them with human preferences a critical challenge. Reward Models (RMs) are a core technology for achieving this goal, but a systematic guide for building state-of-the-art Multimodal Reward Models (MRMs) is currently lacking in both academia and industry. Through exhaustive experimental analysis, this paper aims to provide a clear ``recipe'' for constructing high-performance MRMs. We systematically investigate every crucial component in the MRM development pipeline, including \\textit{reward modeling paradigms} (e.g., Naive-RM, Critic-based RM, and Generative RM), \\textit{reward head architecture}, \\textit{training strategies}, \\textit{data curation} (covering over ten multimodal and text-only preference datasets), \\textit{backbone model} and \\textit{model scale}, and \\textit{ensemble methods}. Based on these experimental insights, we introduce \\textbf{BaseReward}, a powerful and efficient baseline for multimodal reward modeling. BaseReward adopts a simple yet effective architecture, built upon a {Qwen2.5-VL} backbone, featuring an optimized two-layer reward head, and is trained on a carefully curated mixture of high-quality multimodal and text-only preference data. Our results show that BaseReward establishes a new SOTA on major benchmarks such as MM-RLHF-Reward Bench, VL-Reward Bench, and Multimodal Reward Bench, outperforming previous models. Furthermore, to validate its practical utility beyond static benchmarks, we integrate BaseReward into a real-world reinforcement learning pipeline, successfully enhancing an MLLM's performance across various perception, reasoning, and conversational tasks. This work not only delivers a top-tier MRM but, more importantly, provides the community with a clear, empirically-backed guide for developing robust reward models for the next generation of MLLMs.","short_abstract":"The rapid advancement of Multimodal Large Language Models (MLLMs) has made aligning them with human preferences a critical challenge. Reward Models (RMs) are a core technology for achieving this goal, but a systematic guide for building state-of-the-art Multimodal Reward Models (MRMs) is currently lacking in both acade...","url_abs":"https://arxiv.org/abs/2509.16127","url_pdf":"https://arxiv.org/pdf/2509.16127v1","authors":"[\"Yi-Fan Zhang\",\"Haihua Yang\",\"Huanyu Zhang\",\"Yang Shi\",\"Zezhou Chen\",\"Haochen Tian\",\"Chaoyou Fu\",\"Haotian Wang\",\"Kai Wu\",\"Bo Cui\",\"Xu Wang\",\"Jianfei Pan\",\"Haotian Wang\",\"Zhang Zhang\",\"Liang Wang\"]","published":"2025-09-19T16:25:26Z","proceeding":"cs.CV","tasks":"[\"cs.CV\"]","methods":"[\"Reinforcement Learning\",\"Large Language Model\",\"Language Model\",\"RLHF\"]","has_code":false}