{"ID":2830239,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.10575","arxiv_id":"2512.10575","title":"RoleRMBench \u0026 RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems","abstract":"Reward modeling has become a cornerstone of aligning large language models (LLMs) with human preferences. Yet, when extended to subjective and open-ended domains such as role play, existing reward models exhibit severe degradation, struggling to capture nuanced and persona-grounded human judgments. To address this gap, we introduce RoleRMBench, the first systematic benchmark for reward modeling in role-playing dialogue, covering seven fine-grained capabilities from narrative management to role consistency and engagement. Evaluation on RoleRMBench reveals large and consistent gaps between general-purpose reward models and human judgment, particularly in narrative and stylistic dimensions. We further propose RoleRM, a reward model trained with Continuous Implicit Preferences (CIP), which reformulates subjective evaluation as continuous consistent pairwise supervision under multiple structuring strategies. Comprehensive experiments show that RoleRM surpasses strong open- and closed-source reward models by over 24% on average, demonstrating substantial gains in narrative coherence and stylistic fidelity. Our findings highlight the importance of continuous preference representation and annotation consistency, establishing a foundation for subjective alignment in human-centered dialogue systems.","short_abstract":"Reward modeling has become a cornerstone of aligning large language models (LLMs) with human preferences. Yet, when extended to subjective and open-ended domains such as role play, existing reward models exhibit severe degradation, struggling to capture nuanced and persona-grounded human judgments. To address this gap,...","url_abs":"https://arxiv.org/abs/2512.10575","url_pdf":"https://arxiv.org/pdf/2512.10575v1","authors":"[\"Hang Ding\",\"Qiming Feng\",\"Dongqi Liu\",\"Qi Zhao\",\"Tao Yao\",\"Shuo Wang\",\"Dongsheng Chen\",\"Jian Li\",\"Zhenye Gan\",\"Jiangning Zhang\",\"Chengjie Wang\",\"Yabiao Wang\"]","published":"2025-12-11T12:04:46Z","proceeding":"cs.CL","tasks":"[\"cs.CL\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}