{"ID":5346754,"CreatedAt":"2026-06-30T04:09:55.830587294Z","UpdatedAt":"2026-07-02T14:12:34.668891255Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.30339","arxiv_id":"2606.30339","title":"REAR: Test-time Preference Realignment through Reward Decomposition","abstract":"Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task. While post-training methods can adapt models to specific needs, they often require costly data curation and additional training. Test-time scaling (TTS) presents an efficient, training-free alternative, but its application has been largely limited to verifiable domains like mathematics and coding, where response correctness is easily judged. To extend TTS to preference alignment, we introduce a novel framework that models the task as a realignment problem, since the base model often fails to sufficiently align with the stated preference. Our key insight is to decompose the underlying reward function into two components: one related to the question and the other to preference information. This allows us to derive a REAlignment Reward (REAR) that selectively rescales the proportions of these two reward terms. We then show that REAR can be formulated as a linear combination of token-level policy log-probabilities, making it computationally efficient and easy to integrate with various TTS algorithms such as best-of-$N$ sampling and tree search. Experiments show that compared to other test-time baselines, REAR not only enables scalable test-time realignment for preference alignment tasks under diverse user requirements, but also generalizes to mathematical and visual tasks under appropriate preference settings.","short_abstract":"Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task. While post-training methods can adapt models to specific needs, they often require costly data curation and additional training. Test-time scaling (TTS) presents an efficient, training-free alternative, but its appli...","url_abs":"https://arxiv.org/abs/2606.30339","url_pdf":"https://arxiv.org/pdf/2606.30339v1","authors":"[\"Fuxiang Zhang\",\"Pengcheng Wang\",\"Chenran Li\",\"Yi-Chen Li\",\"Yuxin Chen\",\"Lang Feng\",\"Chenfeng Xu\",\"Masayoshi Tomizuka\",\"Bo An\"]","published":"2026-06-29T14:17:53Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.LG\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}