Session 21: Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation
Diffusion LLM
[Diffusion LLM] Session 21: Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State ReformulationJun 15, 2026, 10:00am – Jun 15, 2026, 11:00am (GMT-07:00) Pacific Time - Los Angeles |
Hello folks,
Discrete diffusion models are often trained through clean-data prediction, but the prediction can be used in different ways to define the reverse dynamics. In Masked Diffusion Models (MDM) these choices largely coincide, whereas in Uniform Diffusion Models (UDM) they do not.
The authors show that the standard plug-in bridge parameterization for UDM is not optimized by the denoising posterior, but by a leave-one-out posterior that predicts each clean token without using its own noisy observation. This identifies a mismatch between the plug-in ELBO and the usual cross-entropy denoising objective.
The researchers characterize the leave-one-out target and derive exact conversions between the denoiser, the leave-one-out posterior, and the score. These conversions allow them to disentangle parameterization and the training objective.
Their results also lead to inference improvements without any additional training through an informed predictor-corrector sampler and improved temperature sampling based on the leave-one-out predictor.
The authors further introduce an absorbing-state reformulation of uniform diffusion that preserves the UDM joint law while decomposing it into masked-diffusion-like sampling operations, with simpler denoising posteriors, carry-over unmasking, and a natural remasking mechanism.
On language modeling, leave-one-out parameterizations consistently improve UDM generation, while the absorbing construction matches or surpasses masked diffusion. These results suggest that the empirical gap between masked and uniform diffusion is driven less by the choice of marginals themselves than by parameterization and sampling design.
This Monday, Samson Gourevitch, Yazid Janati, and Dario Shariatian will present their paper “Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation”.
Title: Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation
Meeting Link: click here
Time: June 15 (Monday) 1pm ET / 10am PT / 7pm CET / 10:30pm IST
Prior knowledge:
Fundamentals of discrete diffusion (video by Sasha Rush)
The Diffusion Duality (video by our reading group)
Abstract:
Discrete diffusion models are often trained through clean-data prediction, but the prediction can be used in different ways to define the reverse dynamics. In Masked Diffusion Models (MDM) these choices largely coincide, whereas in Uniform Diffusion Models (UDM) they do not. We show that the standard plug-in bridge parameterization for UDM is not optimized by the denoising posterior, but by a leave-one-out posterior that predicts each clean token without using its own noisy observation. This identifies a mismatch between the plug-in ELBO and the usual cross-entropy denoising objective. We characterize the leave-one-out target and derive exact conversions between the denoiser, the leave-one-out posterior, and the score. These conversions allow us to disentangle parameterization and training objective. Our results also lead to inference improvements without any additional training through an informed predictor-corrector sampler and improved temperature sampling based on the leave-one-out predictor. We further introduce an absorbing-state reformulation of uniform diffusion that preserves the UDM joint law while decomposing it into masked-diffusion-like sampling operations, with simpler denoising posteriors, carry-over unmasking, and a natural remasking mechanism. On language modeling, leave-one-out parameterizations consistently improve UDM generation, while the absorbing construction matches or surpasses masked diffusion. These results suggest that the empirical gap between masked and uniform diffusion is driven less by the choice of marginals themselves than by parameterization and sampling design. The code and models can be found at this https URL.
Yours truly,
Subham, Justin, Zhihan
Diffusion LLM
Gentle reminder: See you all at 1pm ET / 10am PT / 7pm CET / 10:30pm IST
Meeting Link: click here
Today's paper: [2605.22765] Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation