{"ID":2847195,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.00423","arxiv_id":"2511.00423","title":"Bootstrap Off-policy with World Model","abstract":"Online planning has proven effective in reinforcement learning (RL) for improving sample efficiency and final performance. However, using planning for environment interaction inevitably introduces a divergence between the collected data and the policy's actual behaviors, degrading both model learning and policy improvement. To address this, we propose BOOM (Bootstrap Off-policy with WOrld Model), a framework that tightly integrates planning and off-policy learning through a bootstrap loop: the policy initializes the planner, and the planner refines actions to bootstrap the policy through behavior alignment. This loop is supported by a jointly learned world model, which enables the planner to simulate future trajectories and provides value targets to facilitate policy improvement. The core of BOOM is a likelihood-free alignment loss that bootstraps the policy using the planner's non-parametric action distribution, combined with a soft value-weighted mechanism that prioritizes high-return behaviors and mitigates variability in the planner's action quality within the replay buffer. Experiments on the high-dimensional DeepMind Control Suite and Humanoid-Bench show that BOOM achieves state-of-the-art results in both training stability and final performance. The code is accessible at https://github.com/molumitu/BOOM_MBRL.","short_abstract":"Online planning has proven effective in reinforcement learning (RL) for improving sample efficiency and final performance. However, using planning for environment interaction inevitably introduces a divergence between the collected data and the policy's actual behaviors, degrading both model learning and policy improve...","url_abs":"https://arxiv.org/abs/2511.00423","url_pdf":"https://arxiv.org/pdf/2511.00423v3","authors":"[\"Guojian Zhan\",\"Likun Wang\",\"Xiangteng Zhang\",\"Jiaxin Gao\",\"Masayoshi Tomizuka\",\"Shengbo Eben Li\"]","published":"2025-11-01T06:33:04Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.RO\"]","methods":"[\"Reinforcement Learning\"]","has_code":false,"code_links":[{"ID":607495,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2847195,"paper_url":"https://arxiv.org/abs/2511.00423","paper_title":"Bootstrap Off-policy with World Model","repo_url":"https://github.com/molumitu/BOOM_MBRL","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}