Session 27: From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
Diffusion LLM
Hello folks,
Many discrete reasoning tasks, such as code generation, are inherently non-causal. Programmers naturally move between high-level structure and local details in a process called any-order inference. While masked diffusion models offer a native any-order prediction interface, maximizing their potential is difficult due to a core limitation:
This any-order interface does not automatically yield actual any-order inference
Fixed-canvas, token-level models suffer from "positional uncertainty"—they may know what semantic component should appear, but not where to place it
To address the interface-inference gap, the authors propose two complementary approaches designed to natively support any-order inference without relying on hand-designed mechanisms.
The framework tackles positional uncertainty through two methods: (1) Insertion-based masked diffusion (building on FlexMDM), which relaxes fixed-position commitments via insertions to enable generation across non-contiguous regions, and (2) Latent-space masked diffusion (LatentMDM), which shifts prediction to coarser semantic segments to enable search over latent generation orders.
The resulting models—a 7B FlexMDM trained for Python coding and a 125M LatentMDM for GSM8K—successfully induce distinct any-order inference behaviors. The study demonstrates that overcoming positional uncertainty through these approaches significantly improves downstream performance.
This Monday, the co-first authors Seunggeun Kim (@seunggeun_kr), Jaeyeon Kim (@Jaeyeon_Kim_0), Taekyun Lee (taekyunl.github.io), and Yuyuan Chen (yuyuanchen0.github.io) will present their recent paper on the FlexMDM and LatentMDM frameworks.
Title: From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
Meeting Link: click here
Time: Aug 31 (Monday) 1pm ET / 10am PT / 7pm CET / 10:30pm IST
Paper: [2607.26504] From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
Prior knowledge:
Fundamentals of discrete diffusion (video by Sasha Rush)
Abstract:
Many discrete reasoning tasks, such as code generation, are inherently non-causal: programmers move between high-level structure and local details, a process we call any-order inference. For autoregressive language models, which lack a native any-order interface, non-causal abilities such as infilling and next-edit prediction require hand-designed mechanisms. Can we instead design models that natively support any-order inference? Masked diffusion models have recently emerged as compelling candidates, as their any-order training objective naturally offers an any-order prediction interface. This interface, however, does not automatically yield any-order inference. We demonstrate that this interface-inference gap stems from positional uncertainty: fixed-canvas, token-level models may know what semantic component should appear without knowing where to place it. In light of this, we propose two complementary approaches: (1) Insertion-based masked diffusion, building on FlexMDM (Kim et al, 2025), relaxes fixed-position commitments via insertions, enabling generation across non-contiguous regions. (2) Latent-space masked diffusion shifts prediction to coarser semantic segments, enabling search over latent generation orders. Empirically, we train a 7B FlexMDM for Python coding and a 125M LatentMDM for GSM8K and show that both approaches induce distinct any-order inference behaviors and improve downstream performance.
Yours truly,
Subham, Justin, Zhihan