{"ID":2845725,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.03317","arxiv_id":"2511.03317","title":"Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models","abstract":"Text-to-image diffusion models deliver high-quality images, yet aligning them with human preferences remains challenging. We revisit diffusion-based Direct Preference Optimization (DPO) for these models and identify a critical pathology: enlarging the preference margin does not necessarily improve generation quality. In particular, the standard Diffusion-DPO objective can increase the reconstruction error of both winner and loser branches. Consequently, degradation of the less-preferred outputs can become sufficiently severe that the preferred branch is also adversely affected even as the margin grows. To address this, we introduce Diffusion-SDPO, a safeguarded update rule that preserves the winner by adaptively scaling the loser gradient according to its alignment with the winner gradient. A first-order analysis yields a closed-form scaling coefficient that guarantees the error of the preferred output is non-increasing at each optimization step. Our method is simple, model-agnostic, broadly compatible with existing DPO-style alignment frameworks and adds only marginal computational overhead. Across standard text-to-image benchmarks, Diffusion-SDPO delivers consistent gains over preference-learning baselines on automated preference, aesthetic, and prompt alignment metrics. Code is publicly available at https://github.com/AIDC-AI/Diffusion-SDPO.","short_abstract":"Text-to-image diffusion models deliver high-quality images, yet aligning them with human preferences remains challenging. We revisit diffusion-based Direct Preference Optimization (DPO) for these models and identify a critical pathology: enlarging the preference margin does not necessarily improve generation quality. I...","url_abs":"https://arxiv.org/abs/2511.03317","url_pdf":"https://arxiv.org/pdf/2511.03317v2","authors":"[\"Minghao Fu\",\"Guo-Hua Wang\",\"Tianyu Cui\",\"Qing-Guo Chen\",\"Zhao Xu\",\"Weihua Luo\",\"Kaifu Zhang\"]","published":"2025-11-05T09:30:49Z","proceeding":"cs.CV","tasks":"[\"cs.CV\"]","methods":"[\"Diffusion Model\"]","has_code":false,"code_links":[{"ID":607377,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2845725,"paper_url":"https://arxiv.org/abs/2511.03317","paper_title":"Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models","repo_url":"https://github.com/AIDC-AI/Diffusion-SDPO","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}