Session 17: IDLM: Inverse-distilled Diffusion Language Models
Diffusion LLM
Hello folks,
Diffusion Language Models (DLMs) have recently achieved strong results in text generation. However, their multi-step sampling leads to slow inference, limiting practical use.
To address this, the authors extend Inverse Distillation, a technique originally developed to accelerate continuous diffusion models, to the discrete setting. However, this extension introduces both theoretical and practical challenges.
To overcome these challenges, the authors first provide a theoretical result demonstrating that their inverse formulation admits a unique solution, thereby ensuring valid optimization. They then introduce gradient-stable relaxations to support effective training.
As a result, experiments on multiple DLMs show that their method, Inverse-distilled Diffusion Language Models (IDLM), reduces the number of inference steps by 4×—64×, while preserving the teacher model’s entropy and generative perplexity.
This Monday, David Li and Nikita Gushchin will present their jointly led paper, which was recently accepted at ICML 2026.
Title: IDLM: Inverse-distilled Diffusion Language Models
Meeting Link: click here
Time: May 18 (Monday) 1pm ET / 10am PT / 7pm CET / 10:30pm IST
Paper: [2602.19066] IDLM: Inverse-distilled Diffusion Language Models
Prior knowledge:
Fundamentals of discrete diffusion (video by Sasha Rush)
The Diffusion Duality (video by our reading group)
Abstract: Diffusion Language Models (DLMs) have recently achieved strong results in text generation. However, their multi-step sampling leads to slow inference, limiting practical use. To address this, we extend Inverse Distillation, a technique originally developed to accelerate continuous diffusion models, to the discrete setting. Nonetheless, this extension introduces both theoretical and practical challenges. From a theoretical perspective, the inverse distillation objective lacks uniqueness guarantees, which may lead to suboptimal solutions. From a practical standpoint, backpropagation in the discrete space is non-trivial and often unstable. To overcome these challenges, we first provide a theoretical result demonstrating that our inverse formulation admits a unique solution, thereby ensuring valid optimization. We then introduce gradient-stable relaxations to support effective training. As a result, experiments on multiple DLMs show that our method, Inverse-distilled Diffusion Language Models (IDLM), reduces the number of inference steps by 4x-64x, while preserving the teacher model's entropy and generative perplexity.
Yours truly,
Subham, Justin, Zhihan
Website, Twitter, Discord, YouTube