{"ID":2862511,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.25774","arxiv_id":"2509.25774","title":"PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Models","abstract":"While reinforcement learning has advanced the alignment of text-to-image (T2I) models, state-of-the-art policy gradient methods are still hampered by training instability and high variance, hindering convergence speed and compromising image quality. Our analysis identifies a key cause of this instability: disproportionate credit assignment, in which the mathematical structure of the generative sampler produces volatile and non-proportional feedback across timesteps. To address this, we introduce Proportionate Credit Policy Optimization (PCPO), a framework that enforces proportional credit assignment through a stable objective reformulation and a principled reweighting of timesteps. This correction stabilizes the training process, leading to significantly accelerated convergence and superior image quality. The improvement in quality is a direct result of mitigating model collapse, a common failure mode in recursive training. PCPO substantially outperforms existing policy gradient baselines on all fronts, including the state-of-the-art DanceGRPO. Code is available at https://github.com/jaylee2000/pcpo/.","short_abstract":"While reinforcement learning has advanced the alignment of text-to-image (T2I) models, state-of-the-art policy gradient methods are still hampered by training instability and high variance, hindering convergence speed and compromising image quality. Our analysis identifies a key cause of this instability: disproportion...","url_abs":"https://arxiv.org/abs/2509.25774","url_pdf":"https://arxiv.org/pdf/2509.25774v3","authors":"[\"Jeongjae Lee\",\"Jong Chul Ye\"]","published":"2025-09-30T04:43:58Z","proceeding":"cs.CV","tasks":"[\"cs.CV\",\"cs.AI\",\"cs.LG\"]","methods":"[\"Reinforcement Learning\"]","has_code":false,"code_links":[{"ID":608902,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2862511,"paper_url":"https://arxiv.org/abs/2509.25774","paper_title":"PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Models","repo_url":"https://github.com/jaylee2000/pcpo","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}