{"ID":2862236,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.01167","arxiv_id":"2510.01167","title":"Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards","abstract":"Aligning large language models to human preferences is inherently multidimensional, yet most pipelines collapse heterogeneous signals into a single optimizeable objective. We seek to answer what it would take to simultaneously align a model across various domains spanning those with: verifiable rewards (mathematical accuracy), non-verifiable subjective preferences (human values), and complex interactive scenarios (multi-turn AI tutoring dialogues). Such multi-objective reinforcement learning setups are often plagued by the individual objectives being at odds with each other, resulting in inefficient training and little user control during inference. We propose a unified framework that: (i) standardizes {process reward model} (PRM) training across both verifiable and non-verifiable settings to better supervise models' chain-of-thought reasoning; (ii) performs {multi-objective alignment} by training the LLM with our $\\textbf{M}$ulti-$\\textbf{A}$ction-$\\textbf{H}$ead $\\textbf{DPO}$ (MAH-DPO) and a vectorized reward where the dimensions of the vector correspond to the various objectives instead of a single scalar; and (iii) demonstrates how such a system provides fine-grained inference-time user control. Experiments across math reasoning, value alignment, and multi-turn dialogue show that our framework improves performance across multiple objectives simultaneously, while minimizing cross-objective trade-offs and enabling flexible inference time user control. The code can be found at https://github.com/pearls-lab/multiobj-align.","short_abstract":"Aligning large language models to human preferences is inherently multidimensional, yet most pipelines collapse heterogeneous signals into a single optimizeable objective. We seek to answer what it would take to simultaneously align a model across various domains spanning those with: verifiable rewards (mathematical ac...","url_abs":"https://arxiv.org/abs/2510.01167","url_pdf":"https://arxiv.org/pdf/2510.01167v1","authors":"[\"Yiran Shen\",\"Yu Xia\",\"Jonathan Chang\",\"Prithviraj Ammanabrolu\"]","published":"2025-10-01T17:54:15Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.CL\"]","methods":"[\"Reinforcement Learning\",\"Large Language Model\",\"Language Model\"]","has_code":false,"code_links":[{"ID":608882,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2862236,"paper_url":"https://arxiv.org/abs/2510.01167","paper_title":"Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards","repo_url":"https://github.com/pearls-lab/multiobj-align","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}