Diffusion Is the Future of LLMs — Join Us Monday
123 views
Skip to first unread message
Diffusion LLM
Nov 13, 2025, 5:03:29 PM11/13/25
to Diffusion-llms
Hello folks,
Diffusion is emerging as the future of LLMs because, as Ilya Sutskever puts it, “we have only one internet.” In other words, our training data is fundamentally finite — and that has big implications for how we scale models.
This Monday, Mihir Prabhudesai (Ph.D. student, CMU) will unpack this idea in depth at our Discrete Diffusion Reading Group. If you’re curious about where LLMs are headed (and why diffusion might win), you won’t want to miss it.
Hope to see you there!
Abstract: Autoregressive (AR) models have long dominated the landscape of large language models, driving progress across a wide range of tasks. Recently, diffusion-based language models have emerged as a promising alternative, though their advantages over AR models remain underexplored. In this paper, we systematically study masked diffusion models in data-constrained settings where training involves repeated passes over limited data and find that they significantly outperform AR models when compute is abundant but data is scarce. Diffusion models make better use of repeated data, achieving lower validation loss and superior downstream performance. We find new scaling laws for diffusion models and derive a closed-form expression for the critical compute threshold at which diffusion begins to outperform AR. Finally, we explain why diffusion models excel in this regime: their randomized masking objective implicitly trains over a rich distribution of token orderings, acting as an implicit data augmentation that AR's fixed left-to-right factorization lacks. Our results suggest that when data, not compute, is the bottleneck, diffusion models offer a compelling alternative to the standard AR paradigm.
Yours truly,
Subham, Justin, Zhihan
Diffusion LLM
Nov 13, 2025, 5:07:03 PM11/13/25
to Diffusion-llms
Time: Nov 17 (Monday) 10 am ET / 4 pm CET
Diffusion LLM
Nov 16, 2025, 8:39:11 PM11/16/25
to Diffusion-llms
Gentle reminder: See you all at 10 AM ET / 4 PM CET.
As preparation, please take some time to familiarize yourself with discrete diffusion models. A helpful resource: https://www.youtube.com/watch?v=WjAUX23vgfg
Here's the paper that will be covered today: https://arxiv.org/abs/2507.15857
Diffusion LLM
Nov 21, 2025, 5:08:19 PM11/21/25
to Diffusion-llms
Hello folks, the recording for Mihir's talk can be found here: https://www.youtube.com/watch?v=f6xHZEKEkU0&list=PLUeHGh5tvjZ890GBO1aGcmHqNulwoiJyG
Reply all
Reply to author
Forward
0 new messages