{"ID":2921588,"CreatedAt":"2026-06-02T02:42:49.606572591Z","UpdatedAt":"2026-06-03T05:23:15.23124536Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.01024","arxiv_id":"2606.01024","title":"DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs","abstract":"Discrete Masked diffusion language models generate text by iterative parallel decoding, but few-step decoding suffers from a tradeoff between length and quality: with a fixed step budget, standard methods can generate a short, high-quality output, or they can produce long but repetitive text. Continuous denoising can sidestep this tradeoff by evolving all positions jointly in embedding space, but building such a model from scratch at scale remains an open problem. We show that a pretrained masked DLM can instead be lightly adapted to support continuous embedding-space denoising. Starting from LLaDA-8B-Instruct, we continue-pretrain for only 1,000 steps with Discrete Stochastic Localization (DSL), replacing binary masking with continuous per-token Gaussian noise as a soft mask. The adapted model supports continuous inference that evolves all positions jointly in embedding space and defers hard token commitment to the final step. On zero-shot summarization at low step budgets (\u003c=16 forward passes), DSL-LLaDA-SDE achieves the best ROUGE-1 on all four benchmarks and largely avoids the premature-termination / repetition tradeoff of iterative unmasking. The same adaptation also yields selective noisy-state robustness: the model corrects corrupted tokens while preserving clean ones. Control experiments using standard masked diffusion training with the same compute demonstrate neither behavior.","short_abstract":"Discrete Masked diffusion language models generate text by iterative parallel decoding, but few-step decoding suffers from a tradeoff between length and quality: with a fixed step budget, standard methods can generate a short, high-quality output, or they can produce long but repetitive text. Continuous denoising can s...","url_abs":"https://arxiv.org/abs/2606.01024","url_pdf":"https://arxiv.org/pdf/2606.01024v1","authors":"[\"Longxuan Yu\",\"Yunshu Wu\",\"Yu Fu\",\"Siheng Xiong\",\"Rob Brekelmans\",\"Hui Liu\",\"Yue Dong\",\"Greg Ver Steeg\"]","published":"2026-05-31T05:27:01Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\"]","methods":"[\"Diffusion Model\",\"Language Model\"]","has_code":false}