Session 8 // Feb 9: The Diffusion Duality
Diffusion LLM
Hello folks,
Uniform-state discrete diffusion (USDM) promises self-correcting text generation but still lags behind AR and masked diffusion. Diffusion Duality (Duo) reveals that USDM arises from an underlying Gaussian diffusion, unlocking: (1) Curriculum learning → 2x faster training (2) Discrete Consistency Distillation → 100x faster few-step generation.
This Monday, Subham Sahoo (IFM), Justin Deschenaux (EPFL), and Zhihan Yang (Cornell) will jointly present the paper The Diffusion Duality.
Title: The Diffusion Duality
Meeting Link: click here
Time: Feb 9 (Monday) 1pm ET / 10am PT / 7pm CET / 11:30pm IST
Paper: https://arxiv.org/abs/2506.10892
Prior knowledge:
Fundamentals of discrete diffusion (video by Sasha Rush)
Uniform-state discrete diffusion (video by Yair Schiff)
Abstract: Uniform-state discrete diffusion models hold the promise of fast text generation due to their inherent ability to self-correct. However, they are typically outperformed by autoregressive models and masked diffusion models. In this work, we narrow this performance gap by leveraging a key insight: Uniform-state diffusion processes naturally emerge from an underlying Gaussian diffusion. Our method, Duo, transfers powerful techniques from Gaussian diffusion to improve both training and sampling. First, we introduce a curriculum learning strategy guided by the Gaussian process, doubling training speed by reducing variance. Models trained with curriculum learning surpass autoregressive models in zero-shot perplexity on 3 of 7 benchmarks. Second, we present Discrete Consistency Distillation, which adapts consistency distillation from the continuous to the discrete setting. This algorithm unlocks few-step generation in diffusion language models by accelerating sampling by two orders of magnitude. We provide the code, model checkpoints, and video tutorials on the project page: this http URL.
Yours truly,
Subham, Justin, Zhihan
Website, Twitter, Discord, YouTubeDiffusion LLM
Gentle reminder: See you all at 1pm ET / 10am PT / 7pm CET / 11:30pm IST
Meeting Link: click here
Today's paper: https://arxiv.org/abs/2506.10892