Session 23: Continuous Diffusion Scales Competitively with Discrete Diffusion on Language
Diffusion LLM
Hello folks,
While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches.
To challenge this belief, the authors revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs.
In this unified setting, they establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only 20× compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime.
The authors benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of 22.1 among continuous DLMs and superior generation quality.
These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs.
Moreover, they offer theoretical insights to understand the advantage of likelihood-based training. They show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, the authors find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.
This Monday, Zhihan Yang from Cornell University will present his latest work: Continuous Diffusion Scales Competitively with Discrete Diffusion on Language.
Title: Continuous Diffusion Scales Competitively with Discrete Diffusion on Language
Meeting Link: click here
Time: July 13 (Monday) 1pm ET / 10am PT / 7pm CET / 10:30pm IST
Paper: [2605.18530] Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
Prior knowledge:
Fundamentals of discrete diffusion (video by Sasha Rush)
The Diffusion Duality (video by our reading group)
Scaling Uniform and Esoteric d-LLMs (video by our reading group)
Abstract:
While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only 20× compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of 22.1 among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.
Yours truly,
Subham, Justin, Zhihan
Diffusion LLM
Gentle reminder: See you all at 1pm ET / 10am PT / 7pm CET / 10:30pm IST
Meeting Link: click here
Today's paper: [2605.18530] Continuous Diffusion Scales Competitively with Discrete Diffusion for Language