{"ID":2844438,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.07483","arxiv_id":"2511.07483","title":"Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning","abstract":"Recent advancements in large language models (LLMs) have shifted the post-training paradigm from traditional instruction tuning and human preference alignment toward reinforcement learning (RL) focused on reasoning capabilities. However, numerous technical reports indicate that purely rule-based reward RL frequently results in poor-quality reasoning chains or inconsistencies between reasoning processes and final answers, particularly when the base model is of smaller scale. During the RL exploration process, models might employ low-quality reasoning chains due to the lack of knowledge, occasionally producing correct answers randomly and receiving rewards based on established rule-based judges. This constrains the potential for resource-limited organizations to conduct direct reinforcement learning training on smaller-scale models. We propose a novel confidence-based reward model tailored for enhancing STEM reasoning capabilities. Unlike conventional approaches, our model penalizes not only incorrect answers but also low-confidence correct responses, thereby promoting more robust and logically consistent reasoning. We validate the effectiveness of our approach through static evaluations, Best-of-N inference tests, and PPO-based RL training. Our method outperforms several state-of-the-art open-source reward models across diverse STEM benchmarks. We release our codes and model in https://github.com/qianxiHe147/C2RM.","short_abstract":"Recent advancements in large language models (LLMs) have shifted the post-training paradigm from traditional instruction tuning and human preference alignment toward reinforcement learning (RL) focused on reasoning capabilities. However, numerous technical reports indicate that purely rule-based reward RL frequently re...","url_abs":"https://arxiv.org/abs/2511.07483","url_pdf":"https://arxiv.org/pdf/2511.07483v1","authors":"[\"Qianxi He\",\"Qingyu Ren\",\"Shanzhe Lei\",\"Xuhong Wang\",\"Yingchun Wang\"]","published":"2025-11-09T17:58:40Z","proceeding":"cs.AI","tasks":"[\"cs.AI\",\"cs.LG\"]","methods":"[\"Reinforcement Learning\",\"Large Language Model\",\"Language Model\",\"LoRA\",\"Generative Adversarial Network\"]","has_code":false,"code_links":[{"ID":607294,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2844438,"paper_url":"https://arxiv.org/abs/2511.07483","paper_title":"Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning","repo_url":"https://github.com/qianxiHe147/C2RM","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}