Session 26: Self-conditioned Flow Map Language Models via Fixed-point Flows
Diffusion LLM
Hello folks,
Self-conditioning is a core technique that enhances continuous flow-based language models by allowing the model to condition on its own denoising estimate. While empirically successful, maximizing its potential is difficult due to a number of reasons:
The underlying mechanisms driving its performance improvements are poorly understood.
It remains unclear how to leverage self-conditioning for few-step generators based on flow maps.
To address these issues, the authors propose fixed-point flows, a two-dimensional class of self-conditioned flows. They identify that flow models with self-conditioning actually solve a fixed-point iteration that bootstraps the learned denoiser's performance.
The framework distills these fixed-point flows by compressing both dimensions of the process: the fixed-point iterations (via fixed-point distillation) and the flow process (via flow map distillation). This ensures the resulting fixed-point flows define valid, highly efficient flow maps.
The resulting model, FMLM*, outperforms state-of-the-art self-conditioned and few-step models in one- and few-step generation on OpenWebText. The study demonstrates that properly compressing both the fixed-point and flow dimensions unlocks superior few-step generation capabilities.
This Monday, Jaehoon Yoo and Wonjung Kim will present their recent paper on the FMLM* framework.
Title: Self-conditioned Flow Map Language Models via Fixed-point Flows
Meeting Link: click here
Time: Aug 17 (Monday) 1pm ET / 10am PT / 7pm CET / 10:30pm IST
Paper: [2607.00714] Self-conditioned Flow Map Language Models via Fixed-point Flows
Prior knowledge:
Fundamentals of discrete diffusion (video by Sasha Rush)
Abstract:
Self-conditioning is a core technique that enhances continuous flow-based language models, where the model learns to denoise generated text by conditioning on its own denoising estimate. While empirically successful, its performance improvements are poorly understood. Moreover, there is growing interest in the use of few-step generators based on flow maps, for which how to leverage self-conditioning is unclear. Here, we show that flow language models with self-conditioning solve a fixed-point iteration that bootstraps the performance of the learned denoiser. We use this viewpoint to formulate fixed-point flows, a two-dimensional class of self-conditioned flows, where the first dimension represents the flow process and the second represents the fixed-point iteration. We show that fixed-point flows define valid flow maps, and show that they can be distilled from self-conditioned flow models by compressing both fixed-point iterations and flow process, the former with fixed-point distillation and the latter with flow map distillation. Our resulting flow map language models outperform state-of-the-art self-conditioned models in one- and few-step generation on OpenWebText under matched sampling budgets.
Yours truly,
Subham, Justin, Zhihan
Diffusion LLM
Meeting Link: click here
Time: Aug 17 (Monday) 1pm ET / 10am PT / 7pm CET / 10:30pm IST
Paper: [2607.00714] Self-conditioned Flow Map Language Models via Fixed-point Flows