{"ID":5346729,"CreatedAt":"2026-06-30T04:09:55.830587294Z","UpdatedAt":"2026-07-02T14:28:49.359749133Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.30376","arxiv_id":"2606.30376","title":"FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification","abstract":"Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods. Existing density-approximated policy gradient methods rely on stochastic SDE samplers to construct tractable transition kernels, which introduce training-inference inconsistencies and necessitates Classifier-Free Guidance (CFG). While implicit frameworks such as DiffusionNFT directly optimize forward-process velocity fields, its heuristic fixed-magnitude corrections prevent optimization strength from relative intra-group quality. We propose \\textit{Flow Advantage-Weighted Rectification} (\\textbf{FlowAWR}), a paradigm that recasts continuous generative policy optimization as supervised regression toward a theoretically optimal velocity field. Starting from the optimal policy of a KL-constrained reward maximization, FlowAWR derives the optimal velocity field that admits a magnitude-aware, advantage-weighted rectification form, yielding SDE-free optimization and CFG-free generation. In comparative evaluations on SD3.5-Medium, FlowAWR achieves improved alignment performance alongside a 2$\\times$ to 5$\\times$ convergence acceleration over DiffusionNFT (e.g., reaching a 24.12 PickScore in 1.2k steps, versus 23.82 in 2.0k steps for DiffusionNFT and 23.50 in $\u003e$4k steps for FlowGRPO). Under multi-reward constraints, FlowAWR sustains generation quality, satisfying structural rules while maintaining stable out-of-domain performance.","short_abstract":"Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods. Existing density-approximated policy gradient methods rely on stochastic SDE samplers to construct tractable transition kernels, which introduce training-inference inconsistencies...","url_abs":"https://arxiv.org/abs/2606.30376","url_pdf":"https://arxiv.org/pdf/2606.30376v1","authors":"[\"Zheming Fu\",\"Ruizhe He\",\"Wei Shang\",\"Xiaoxiao Ma\",\"Lei Wang\",\"Chang Liu\",\"Siming Fu\"]","published":"2026-06-29T14:37:36Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.CV\"]","methods":"[\"Reinforcement Learning\",\"Diffusion Model\"]","has_code":false}