{"ID":2848927,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.24088","arxiv_id":"2510.24088","title":"Information-Theoretic Discrete Diffusion","abstract":"We present an information-theoretic framework for discrete diffusion models that yields principled estimators of log-likelihood using score-matching losses. Inspired by the I-MMSE identity for the Gaussian setup, we derive analogous results for the discrete setting. Specifically, we introduce the Information-Minimum Denoising Score Entropy (I-MDSE) relation, which links mutual information between data and its diffused version to the minimum denoising score entropy (DSE) loss. We extend this theory to masked diffusion and establish the Information-Minimum Denoising Cross-Entropy (I-MDCE) relation, connecting cross-entropy losses to mutual information in discrete masked processes. These results provide a time-integral decomposition of the log-likelihood of the data in terms of optimal score-based losses, showing that commonly used losses such as DSE and DCE are not merely variational bounds but tight and principled estimators of log-likelihood. The I-MDCE decomposition further enables practical extensions, including time-free formula, conditional likelihood estimation in prompt-response tasks, and coupled Monte Carlo estimation of likelihood ratios. Experiments on synthetic and real-world data confirm the accuracy, variance stability, and utility of our estimators. The code is publicly available at https://github.com/Dongjae0324/infodis.","short_abstract":"We present an information-theoretic framework for discrete diffusion models that yields principled estimators of log-likelihood using score-matching losses. Inspired by the I-MMSE identity for the Gaussian setup, we derive analogous results for the discrete setting. Specifically, we introduce the Information-Minimum De...","url_abs":"https://arxiv.org/abs/2510.24088","url_pdf":"https://arxiv.org/pdf/2510.24088v1","authors":"[\"Moongyu Jeon\",\"Sangwoo Shin\",\"Dongjae Jeon\",\"Albert No\"]","published":"2025-10-28T05:59:05Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.IT\"]","methods":"[\"Diffusion Model\"]","has_code":false,"code_links":[{"ID":607656,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2848927,"paper_url":"https://arxiv.org/abs/2510.24088","paper_title":"Information-Theoretic Discrete Diffusion","repo_url":"https://github.com/Dongjae0324/infodis","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}