Daily TMLR digest for Nov 25, 2025
TMLR
New certifications
==================
Expert Certification: Node Embeddings via Neighbor Embeddings
Jan Niklas Böhm, Marius Keute, Alica Guzmán, Sebastian Damrich, Andrew Draganov, Dmitry Kobak
https://openreview.net/forum?id=8APIU9cauZ
---
Accepted papers
===============
Title: Say My Name: a Model's Bias Discovery Framework
Authors: Massimiliano Ciranni, Luca Molinaro, Carlo Alberto Barbano, Attilio Fiandrotti, Vittorio Murino, Vito Paolo Pastore, Enzo Tartaglione
Abstract: Due to the broad applicability of deep learning to downstream tasks and end-to-end training capabilities in the last few years, increasingly more concerns about potential biases to specific, non-representative patterns have been raised. Many works focusing on unsupervised debiasing leverage the tendency of deep models to learn “easier” samples, for example by clustering the latent space to obtain bias pseudo-labels. However, their interpretation is not trivial as it does not provide semantic information about the bias features. To address this issue, we introduce “Say My Name” (SaMyNa), a tool to identify semantic biases within deep models. Unlike existing methods, our approach focuses on biases learned by the model, enhancing explainability through a text-based pipeline. Applicable during either training or post-hoc validation, our method can disentangle task-related information and propose itself as a tool to analyze biases. Evaluation on typical benchmarks demonstrates its effectiveness in detecting biases and even disclaiming them. When sided with a traditional debiasing approach for bias mitigation, it can achieve state-of-the-art performance while having the advantage of associating a semantic meaning to the discovered bias.
URL: https://openreview.net/forum?id=EuUxiiDZ7d
---
Title: Node Embeddings via Neighbor Embeddings
Authors: Jan Niklas Böhm, Marius Keute, Alica Guzmán, Sebastian Damrich, Andrew Draganov, Dmitry Kobak
Abstract: Node embeddings are a paradigm in non-parametric graph representation learning, where graph nodes are embedded into a given vector space to enable downstream processing. State-of-the-art node-embedding algorithms, such as DeepWalk and node2vec, are based on random-walk notions of node similarity and on contrastive learning. In this work, we introduce the graph neighbor-embedding (graph NE) framework that directly pulls together embedding vectors of adjacent nodes without relying on any random walks. We show that graph NE strongly outperforms state-of-the-art node-embedding algorithms in terms of local structure preservation. Furthermore, we apply graph NE to the 2D node-embedding problem, obtaining graph t-SNE layouts that also outperform existing graph-layout algorithms.
URL: https://openreview.net/forum?id=8APIU9cauZ
---
New submissions
===============
Title: The Confusion is Real: GRAPHIC - A Network Science Approach to Confusion Matrices in Deep Learning
Abstract: Explainable artificial intelligence has emerged as a promising field of research to address reliability concerns in artificial intelligence. Despite significant progress in explainable artificial intelligence, few methods provide a systematic way to visualize and understand how classes are confused and how their relationships evolve as training progresses. In this work, we present GRAPHIC, an architecture-agnostic approach that analyzes neural networks on a class level. It leverages confusion matrices derived from intermediate layers using linear classifiers. We interpret these as adjacency matrices of directed graphs, allowing tools from network science to visualize and quantify learning dynamics across training epochs and intermediate layers. GRAPHIC provides insights into linear class separability, dataset issues, and architectural behavior, revealing, for example, similarities between flatfish and man and labeling ambiguities validated in a human study. In summary, by uncovering real confusions, GRAPHIC offers new perspectives on how neural networks learn.
URL: https://openreview.net/forum?id=UP9bx1WJwR
---
Title: Algorithmic Complexity Predicts when Path Information Im- proves Graph Neural Networks Performance on Molecular Graphs
Abstract: Graph Neural Networks (GNNs) are designed to process irregular relational data in rec-
ommendation systems, protein networks, social networks, and molecules. GNNs typically
rely on message passing and aggregation, with some architectures incorporating graph path
information in a bid to improve accuracy. However, it is unclear whether such incorporation
of path information truly improves GNN accuracy in all cases. As a first step, we herein
shed light on this issue for the case of molecular graphs. We evaluated Graphormer and
Mix-Hop models, with and without path information on 36 molecular datasets, derived from
six MoleculeNet benchmark datasets. Path information improved performance in some cases
but not in other cases. This finding is important, because these two models always incor-
porate path information in practice, whereas the finding shows this incorporation of path
information can actually be detrimental to the models’ accuracies. To more deeply probe
this observation, we developed a graph representation model called T-hop which allows us
to further highlight the use, versus non-use, of path information. On one hand, we formu-
late the Path Usefulness Measure (PUM) to quantify the benefit of path information. On
the other hand, we quantified the randomness of the different datasets via their algorithmic
complexities, using the Block Decomposition Method (BDM). We hypothesized, and con-
firmed our hypothesis, that: GNN models trained on molecular datasets with less random
structures (i.e. lower algorithmic complexity) should benefit from path information (i.e.
larger PUM), compared to datasets with more random structures. In summary, low algo-
rithmic complexity, which captures the presence of structure in molecular graphs, is useful
for predicting when path information improves accuracies in GNNs. A practical benefit of
this is that it leads to a more resource-efficient approach, wherein path information is only
incorporated for datasets with low algorithmic complexities.
URL: https://openreview.net/forum?id=8yPfME1opS
---
Title: Flow Matching for Tabular Data Synthesis
Abstract: Synthetic data generation is an important tool for privacy-preserving data sharing. While diffusion models have set recent benchmarks, flow matching (FM) offers a promising alternative. This paper presents different ways to implement flow matching for tabular data synthesis. We provide a comprehensive empirical study that compares flow matching (FM and variational FM) with a state-of-the-art diffusion method (TabDDPM and TabSyn) in tabular data synthesis. We evaluate both the standard Optimal Transport (OT) and the Variance Preserving (VP) probability paths, and also compare deterministic and stochastic samplers -- something possible when learning to generate using \textit{variational} flow matching -- characterising the empirical relationship between data utility and privacy risk. Our key findings reveal that flow matching, particularly TabbyFlow, outperforms diffusion baselines. Flow matching methods also achieves better performance with remarkably low function evaluations ($\leq$ 100 steps), offering a substantial computational advantage. The choice of probability path is also crucial, as using the OT path demonstrates superior performance, while VP has potential for producing synthetic data with lower disclosure risk. Lastly, our results show that making flows stochastic not only preserves marginal distributions but, in some instances, enables the generation of high utility synthetic data with reduced disclosure risk.
URL: https://openreview.net/forum?id=RdOjoAa66L
---
Title: BlockCert: Certified Blockwise Extraction of Transformer Mechanisms
Abstract: Mechanistic interpretability aspires to reverse-engineer neural networks into explicit algorithms, while model editing seeks to modify specific behaviours without retraining. Both areas are typically evaluated with informal evidence and ad-hoc experiments, with few explicit guarantees about how far an extracted or edited model can drift from the original on relevant inputs. We introduce BLOCKCERT, a framework for certified blockwise extraction of transformer mechanisms, and outline how a lightweight extension can support certified local edits. Given a pre-trained transformer and a prompt distribution, BLOCKCERT extracts structured surrogate implementations for residual blocks together with machine-checkable certificates that bound approximation error, record coverage metrics, and hash the underlying artifacts. We formalize a simple Lipschitz-based composition theorem in Lean 4 that lifts these local guarantees to a global deviation bound. Empirically, we apply the framework to GPT-2 small, TinyLlama-1.1B-Chat, and Llama-3.2-3B. Across these models we obtain high per-block coverage and small residual errors on the evaluated prompts, and in the TinyLlama setting we show that a fully stitched model matches the baseline perplexity within $\approx 6\times 10^{-5}$ on stress prompts. Our results suggest that blockwise extraction with explicit certificates is feasible for real transformer language models and offers a practical bridge between mechanistic interpretability and formal reasoning about model behaviour.
URL: https://openreview.net/forum?id=EIGlWWD83e
---
Title: Optimistic Online Learning in Symmetric Cone Games
Abstract: We introduce symmetric cone games (SCGs), a broad class of multi-player games where each player's strategy lies in a generalized simplex (the trace-one slice of a symmetric cone). This framework unifies a wide spectrum of settings, including normal-form games (simplex strategies), quantum games (density matrices), and continuous games with ball-constrained strategies. It also captures several structured machine learning and optimization problems, such as distance metric learning and Fermat–Weber facility location, as two-player zero-sum SCGs. To compute approximate Nash equilibria in two-player zero-sum SCGs, we propose a single online learning algorithm: Optimistic Symmetric Cone Multiplicative Weights Updates (OSCMWU). Unlike prior methods tailored to specific geometries, OSCMWU provides closed-form, projection-free updates over any symmetric cone and achieves a~$\tilde{\mathcal{O}}(1/\epsilon)$ iteration complexity for computing $\epsilon$-saddle points. Our analysis builds on the Optimistic Follow-the-Regularized-Leader framework and hinges on a key technical contribution: We prove that the symmetric cone negative entropy is strongly convex with respect to the trace-one norm. This result extends known results for the simplex and spectraplex to all symmetric cones, and may be of independent interest.
URL: https://openreview.net/forum?id=RQjvE2o3o6
---
Title: Data Compressibility Quantifies LLM Memorization
Abstract: Large Language Models (LLMs) are known to memorize portions of their training data, sometimes even reproduce content verbatim when prompted appropriately. Despite substantial interest, existing LLM memorization research has offered limited insight into how training data influences memorization and largely lacks quantitative characterization. In this work, we build upon the line of research that seeks to quantify memorization through data compressibility. We analyze why prior attempts fail to yield a reliable quantitative measure and show that a surprisingly simple shift from instance-level to set-level metrics uncovers a robust phenomenon, which we term the \textit{Entropy--Memorization (EM) Law}. This law states that a set-level data entropy estimator exhibits a linear correlation with memorization scores.
We validate the EM Law through extensive experiments across a wide range of open-source models and experimental configurations. We further investigate the role of the token space—an implicit yet pivotal factor in our method—and identify an additional variant of the EM Law. Besides, we made a side observation that EM Law enables a simple application to distinguish between LLM train data and test data.
URL: https://openreview.net/forum?id=6L4UXc7P3h
---
Title: The Role of Feature Interactions in Graph-based Tabular Deep Learning
Abstract: Accurate predictions on tabular data rely on capturing complex, dataset-specific feature interactions. Attention-based methods and graph neural networks, referred to as graph-based tabular deep learning (GTDL), aim to improve predictions by modeling these interactions as a graph. In this work, we analyze how these methods model the feature interactions. Current GTDL approaches primarily focus on optimizing predictive accuracy, often neglecting the accurate modeling of the underlying graph structure. Using synthetic datasets with known ground-truth graph structures, we find that current GTDL methods fail to recover meaningful feature interactions, as their edge recovery is close to random. This suggests that the attention mechanism and message-passing schemes used in GTDL do not effectively capture feature interactions. Furthermore, when we impose the true interaction structure, we find that the predictive accuracy improves. This highlights the need for GTDL methods to prioritize accurate modeling of the graph structure, as it leads to better predictions
URL: https://openreview.net/forum?id=olGaiwoZHZ
---
Title: No Data, No Optimization: A Lightweight Method To Disrupt Neural Networks With Sign-Flips
Abstract: Deep Neural Networks (DNNs) can be catastrophically disrupted by flipping only a handful of sign bits in their parameters. We introduce Deep Neural Lesion (DNL), a data-free, lightweight method that locates these critical parameters and triggers massive accuracy drops. We validate its efficacy on a wide variety of computer vision models and datasets. The method requires no training data or optimization and can be carried out via common exploits software, firmware or hardware based attack vectors. An enhanced variant that uses a single forward and backward pass further amplifies the damage beyond DNL's zero-pass approach. Flipping just two sign bits in ResNet50 on ImageNet reduces accuracy by 99.8%. We also show that selectively protecting a small fraction of vulnerable sign bits provides a practical defense against such attacks.
URL: https://openreview.net/forum?id=kN1s53X3zl
---
Title: Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuning
Abstract: Supervised fine-tuning (SFT) plays a critical role for pretrained large language models (LLMs), notably enhancing their capacity to acquire domain-specific knowledge while preserving or potentially augmenting their general-purpose capabilities. However, the efficacy of SFT hinges on data quality as well as data volume, otherwise it may result in limited performance gains or even degradation relative to the associated baselines. To mitigate such reliance, we suggest categorizing tokens within each corpus into two parts---positive and negative tokens---based on whether they are useful to improve model performance. Positive tokens can be trained in common ways, whereas negative tokens, which may lack essential semantics or be misleading, should be explicitly forgotten. Overall, the token categorization facilitate the model to learn less informative message, and the forgetting process shapes a knowledge boundary to guide the model on what information to learn more precisely. We conduct experiments across diverse and well-established benchmarks using various model architectures, demonstrating that this forgetting mechanism enhances model performance.
URL: https://openreview.net/forum?id=s36smEoUoX
---
Title: Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
Abstract: Recent impressive results from large reasoning models have been interpreted as a triumph of Chain of Thought (CoT), and especially of the process of training on CoTs sampled from base LLMs in order to help find new reasoning patterns. While these traces certainly seem to help the model performance, it is not clear how they actually influence model performance, with some works ascribing semantics to them and others cautioning against relying on them as transparent and faithful proxies of the model's internal computational process. To systematically investigate the role of end-user semantics of derivational traces, we set up a controlled study where we train transformer models from scratch on formally verifiable reasoning traces and the solutions they lead to, constraining both intermediate steps and final outputs to align with those of a formal solver. We notice that, despite significant improvements over the solution-only baseline, models trained on entirely correct traces can still produce invalid reasoning traces even when arriving at correct solutions. More interestingly, our experiments also show that models trained on corrupted traces, whose intermediate reasoning steps bear no relation to the problem they accompany, achieve performance largely comparable to those trained on correct traces. In fact, our corrupted models generalize better on out-of-distribution tasks. We also study the effect of GRPO-based RL post-training on trace validity, noting that while solution accuracy increase, this is not accompanied by any improvements in trace validity. Finally, we examine whether reasoning-trace length reflects inference-time scaling and find that trace length is largely agnostic to the underlying computational complexity of the problem being solved. These results challenge the assumption that intermediate tokens or "Chains of Thought" reflect or induce predictable reasoning behaviors and caution against anthropomorphizing such outputs or over-interpreting them (despite their mostly seemingly forms) as evidence of human-like or algorithmic behaviors in language models.
URL: https://openreview.net/forum?id=gDE7YcRC3F
---
Title: Inference, Fast and Slow: Reinterpreting VAEs for OOD Detection
Abstract: Unsupervised out-of-distribution (OOD) detection is critical for the safe deployment of machine learning systems, yet standard likelihood-based methods using deep generative models (DGMs) often fail, assigning deceptively high likelihoods to anomalous data. We attribute this failure, particularly within Variational Autoencoders (VAEs), to a phenomenon we term likelihood cancellation: informative signals from the model’s encoder and decoder can neutralize each other within the final scalar likelihood. To overcome this, we introduce the Likelihood Path (LPath) Principle, a new framework that extracts a robust OOD signal from the entire computational path of a VAE. We operationalize this principle by reinterpreting VAEs through the lens of fast and slow weights, enabling online, instance-wise inference without costly retraining. Our method extracts minimal sufficient statistics from the VAE’s inference path and feeds them into a classical density estimator. On standard benchmarks (CIFAR-10, SVHN, CIFAR-100), our LPath method achieves state-of-the-art OOD detection, outperforming models with over 10x the parameters. Our lightweight 3M-parameter VAE provides a highly efficient and principled solution for real-world, streaming OOD detection.
URL: https://openreview.net/forum?id=JqRRr90Fdb
---
Title: A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms
Abstract: The development of high-level autonomous driving (AD) is shifting from perception-centric limitations to a more fundamental bottleneck, i.e., a deficit in robust and generalizable reasoning. Although current systems manage structured environments, they consistently falter in long-tail scenarios and complex social interactions that require human-like judgment. Meanwhile, the advent of large language and multimodal models (LLMs and MLLMs) presents a transformative opportunity to integrate a powerful cognitive engine into AD systems, moving beyond pattern matching toward genuine comprehension. However, a systematic framework to guide this integration is critically lacking. To bridge this gap, we provide a comprehensive review of this emerging field, arguing that reasoning must be elevated from a modular component to the central cognitive core. Specifically, we first propose a novel Cognitive Hierarchy to deconstruct the monolithic driving task based on its cognitive and interactive complexity. Based on that, we further derive and systematize seven core reasoning challenges, such as the responsiveness-reasoning trade-off and social game. Furthermore, we conduct a dual-perspective review of the state-of-the-art, analyzing both system-centric approaches to architecting intelligent agents and evaluation-centric practices for their validation. Our analysis reveals a clear trend toward holistic and interpretable "glass-box" agents. In conclusion, we identify a fundamental and unresolved tension between the high-latency, deliberative nature of LLM-based reasoning and the millisecond-scale, safety-critical demands of vehicle control. For future work, the primary objective is to bridge the symbolic-to-physical gap, including verifiable neuro-symbolic architectures, robust reasoning under uncertainty, and scalable models for implicit social negotiation.
URL: https://openreview.net/forum?id=XwQ7dc4bqn
---
Title: A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection
Abstract: The log-linear model has received a significant amount of theoretical attention in previous decades and remains the fundamental tool used for learning probability distributions over discrete variables. Despite its large popularity in statistical mechanics and high-dimensional statistics, the vast majority of related energy-based models only focus on the two-variable relationships, such as Boltzmann machines and Markov graphical models. Although these approaches have easier-to-solve structure learning problems and easier-to-optimize parametric distributions, they often ignore the rich structure which exists in the higher- order interactions between different variables. Using more recent tools from the field of information geometry, we revisit the classical formulation of the log-linear model with a focus on higher-order mode interactions, going beyond the 1-body modes of independent distributions and the 2-body modes of Boltzmann distributions. This perspective allows us to define a complete decomposition of the KL error. This then motivates the formulation of a sparse selection problem over the set of possible mode interactions. In the same way as sparse graph selection allows for better generalization, we find that our learned distributions are able to more efficiently use the finite amount of data which is available in practice. We develop an algorithm called MAHGenTa which leverages a novel Monte-Carlo sampling technique for energy-based models alongside a greedy heuristic for incorporating statistical robustness. On both synthetic and real-world datasets, we demonstrate our algorithm’s effectiveness in maximizing the log-likelihood for the generative task and also the ease of adaptability to the discriminative task of classification.
URL: https://openreview.net/forum?id=DxUXI15C38
---
Title: Generalizing Coverage Plots for Simulation-based Inference
Abstract: Simulation-based inference (SBI) aims to find the probabilistic inverse of a non-linear function
by fitting the posterior with a generative model on samples. Applications demand accurate
uncertainty quantification, which can be difficult to achieve and verify. Since the ground
truth model is implicitly defined in SBI, we cannot compute likelihood values nor draw
samples from the posterior. This renders two-sample testing against the posterior impossible
for any practical use and calls for proxy verification methods such as expected coverage
testing. We introduce a differentiable objective that encourages coverage in the generative
model by parameterizing the dual form of the total variation norm with neural networks.
However, we find that coverage tests can easily report a good fit when the approximant
deviates significantly from the target distribution and give strong empirical evidence and
theoretical arguments why the expected coverage plot is, in general, not a reliable indicator
of posterior fit. To address this matter, we introduce a new ratio coverage plot as a better
alternative to coverage, which is not susceptible to the same blind spots. It comes at the
price of estimating a ratio between our model and the ground truth posterior, which can be
done using standard algorithms. We provide experimental results that back up this claim,
and provide multiple algorithms for estimating ratio coverage.
URL: https://openreview.net/forum?id=eqYapbG2jO
---