{"ID":5438652,"CreatedAt":"2026-07-01T01:17:58.482524686Z","UpdatedAt":"2026-07-03T05:54:49.125664311Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.31178","arxiv_id":"2606.31178","title":"AETDICE: Unified Framework and Offline Optimization for Nonlinear Multi-Objective RL","abstract":"Optimizing nonlinear preferences in multi-objective reinforcement learning (MORL) is essential for capturing complex trade-offs like risk aversion or fairness. However, such non-linearity has historically bifurcated nonlinear MORL objectives into two distinct paradigms: Scalarized Expected Return (SER) and Expected Scalarized Return (ESR). While SER requires global-level optimization and ESR requires non-Markovian policies, leading to fragmented optimization strategies, we bridge this divide through the Aggregation-Expectation-Transformation (AET) framework. By unifying both criteria through a tripartite decomposition of scalarization, AET provides a principled foundation for general nonlinear MORL. Building on this framework, we propose AETDICE, a tractable offline RL algorithm for AET objectives. By utilizing DICE-style density-ratio estimation in an augmented state space, AETDICE enables sample-based optimization from static datasets. Our framework resolves long-standing barriers and captures respective trade-offs induced by AET framework, which existing methods fail to address.","short_abstract":"Optimizing nonlinear preferences in multi-objective reinforcement learning (MORL) is essential for capturing complex trade-offs like risk aversion or fairness. However, such non-linearity has historically bifurcated nonlinear MORL objectives into two distinct paradigms: Scalarized Expected Return (SER) and Expected Sca...","url_abs":"https://arxiv.org/abs/2606.31178","url_pdf":"https://arxiv.org/pdf/2606.31178v1","authors":"[\"Woosung Kim\",\"Youngjun Suh\",\"Jinho Lee\",\"Jongmin Lee\",\"Byung-Jun Lee\"]","published":"2026-06-30T06:08:46Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\"]","methods":"[\"Reinforcement Learning\"]","has_code":false}