Weekly TMLR digest for Jul 06, 2025
TMLR
Accepted papers
===============
Title: What Matters for Model Merging at Scale?
Authors: Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, Tsendsuren Munkhdalai
Abstract: Model merging aims to combine multiple expert models into a more capable single model, offering benefits such as reduced storage and serving costs, improved generalization, and support for decentralized model development. Despite its promise, previous studies have primarily focused on merging a few small models. This leaves many unanswered questions about the effect of scaling model size and how it interplays with other key factors—like the base model quality and number of expert models— to affect the merged model’s performance. This work systematically evaluates the utility of model merging at scale for transformer based models to examine the impact of these different factors. We experiment with merging fully fine-tuned models using four popular merging methods—Averaging, Task Arithmetic, Dare-TIES, and TIES-Merging—across model sizes ranging from 1B to 64B parameters and merging up to 8 different expert models. We evaluate the merged models on both held-in tasks, i.e., the expert’s training tasks, and zero-shot generalization to unseen held-out tasks. Our wide range of experiments provide several new insights about merging transformer based language models at scale and the interplay between different factors. First, we find that merging is more effective when experts are created from strong base models, i.e., models with good zero-shot performance, compared to pre-trained ones. Second, larger models perform better when merged. Third merging consistently improves generalization capabilities. Notably, when merging eight large expert models, the merged models often generalize better compared to the multitask trained models. Fourth, we can better merge more expert models when working with larger models. Fifth, different merging methods behave very similarly at larger scales. Overall, our findings shed light on some interesting properties of model merging while also highlighting some limitations.
URL: https://openreview.net/forum?id=9sbetmvNpW
---
Title: SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks
Authors: Kim-Celine Kahl, Selen Erkan, Jeremias Traub, Carsten T. Lüth, Klaus Maier-Hein, Lena Maier-hein, Paul F Jaeger
Abstract: Vision-Language Models (VLMs) have great potential in medical tasks, like Visual Question Answering (VQA), where they could act as interactive assistants for both patients and clinicians. Yet their robustness to distribution shifts on unseen data remains a key concern for safe deployment. Evaluating such robustness requires a controlled experimental setup that allows for systematic insights into the model's behavior. However, we demonstrate that current setups fail to offer sufficiently thorough evaluations.
To address this gap, we introduce a novel framework, called SURE-VQA, centered around three key requirements to overcome current pitfalls and systematically analyze VLM robustness:
1) Since robustness on synthetic shifts does not necessarily translate to real-world shifts, it should be measured on real-world shifts that are inherent to the VQA data; 2) Traditional token-matching metrics often fail to capture underlying semantics, necessitating the use of large language models (LLMs) for more accurate semantic evaluation; 3) Model performance often lacks interpretability due to missing sanity baselines, thus meaningful baselines should be reported that allow assessing the multimodal impact on the VLM.
To demonstrate the relevance of this framework, we conduct a study on the robustness of various Fine-Tuning (FT) methods across three medical datasets with four types of distribution shifts.
Our study highlights key insights into robustness: 1) No FT method consistently outperforms others in robustness, and 2) robustness trends are more stable across FT methods than across distribution shifts. Additionally, we find that simple sanity baselines that do not use the image data can perform surprisingly well and confirm LoRA as the best-performing FT method on in-distribution data.
Code is provided at https://github.com/IML-DKFZ/sure-vqa.
URL: https://openreview.net/forum?id=qjNdGpgpV8
---
Title: Variance Dichotomy in Feature Spaces of Facial Recognition Systems is a Weak Defense against Simple Weight Manipulation Attacks
Authors: Matthew Bowditch, Mike Paterson, Matthias Englert, Ranko Lazic
Abstract: We show that several leading pretrained facial recognition systems exhibit a variance dichotomy in their feature space. In other words, the feature vectors approximately lie in a lower dimensional linear subspace. We demonstrate that this variance dichotomy degrades the performance of an otherwise powerful scheme for anonymity/unlinkability and confusion attacks on facial recognition system devised by Zehavi et al. (2024), which is based on simple weight manipulations in only the last hidden layer. Lastly, we propose a method for the attacker to overcome this intrinsic defense of these pretrained facial recognition systems.
URL: https://openreview.net/forum?id=Q1Cf07flwD
---
Title: Learned-Database Systems Security
Authors: Roei Schuster, Jin Peng Zhou, Thorsten Eisenhofer, Paul Grubbs, Nicolas Papernot
Abstract: A learned database system uses machine learning (ML) internally to improve performance. We can expect such systems to be vulnerable to some adversarial-ML attacks. Often, the learned component is shared between mutually-distrusting users or processes, much like microarchitectural resources such as caches, potentially giving rise to highly-realistic attacker models. However, compared to attacks on other ML-based systems, attackers face a level of indirection as they cannot interact directly with the learned model. Additionally, the difference between the attack surface of learned and non-learned versions of the same system is often subtle. These factors obfuscate the de-facto risks that the incorporation of ML carries. We analyze the root causes of potentially-increased attack surface in learned database systems and develop a framework for identifying vulnerabilities that stem from the use of ML. We apply our framework to a broad set of learned components currently being explored in the database community. To empirically validate the vulnerabilities surfaced by our framework, we choose 3 of them and implement and evaluate exploits against these. We show that the use of ML cause leakage of past queries in a database, enable a poisoning attack that causes exponential memory blowup in an index structure and crashes it in seconds, and enable index users to snoop on each others' key distributions by timing queries over their own keys. We find that adversarial ML is an universal threat against learned components in database systems, point to open research gaps in our understanding of learned-systems security, and conclude by discussing mitigations, while noting that data leakage is inherent in systems whose learned component is shared between multiple parties.
URL: https://openreview.net/forum?id=XNVBSbtcKB
---
Title: Controlled Model Debiasing through Minimal and Interpretable Updates
Authors: Federico Di Gennaro, Thibault Laugel, Vincent Grari, Marcin Detyniecki
Abstract: Traditional approaches to learning fair machine learning models often require rebuilding models from scratch, typically without considering potentially existing models. In a context where models need to be retrained frequently, this can lead to inconsistent model updates, as well as redundant and costly validation testing. To address this limitation, we introduce the notion of controlled model debiasing, a novel supervised learning task relying on two desiderata: that the differences between the new fair model and the existing one should be (i) minimal and (ii) interpretable. After providing theoretical guarantees to this new problem, we introduce a novel algorithm for algorithmic fairness, COMMOD, that is both model-agnostic and does not require the sensitive attribute at test time. In addition, our algorithm is explicitly designed to enforce minimal and interpretable changes between biased and debiased predictions in a binary classification task—a property that, while highly desirable in high-stakes applications, is rarely prioritized as an explicit objective in fairness literature. Our approach combines a concept-based architecture and adversarial learning and we demonstrate through empirical results that it achieves comparable performance to state-of-the-art debiasing methods while performing minimal and interpretable prediction changes.
URL: https://openreview.net/forum?id=B9fdU4qjpD
---
Title: Fairness and Disentanglement: A Critical Review of Predominant Bias in Neural Networks
Authors: Jiazhi Li, Mahyar Khayatkhoei, Jiageng Zhu, Hanchen Xie, Mohamed E. Hussein, Wael AbdAlmageed
Abstract: Bias issues of neural networks garner significant attention along with their promising advancement.
Among various bias issues, mitigating two predominant biases is crucial in advancing fair and trustworthy AI: (1) ensuring neural networks yield even performance across demographic groups, and (2) ensuring algorithmic decision-making does not rely on protected attributes.
However, upon the investigation of 415 papers in the relevant literature, we find that there exists a persistent, extensive but under-explored confusion regarding these two types of biases.
Furthermore, the confusion has already significantly hampered the clarity of the community and the subsequent development of debiasing methodologies.
Thus, in this work, we aim to restore clarity by providing two mathematical definitions for these two predominant biases and leveraging these definitions to unify a comprehensive list of papers.
Next, we highlight the common phenomena and the possible reasons for the existing confusion.
To alleviate the confusion, we provide extensive experiments on synthetic, census, and image datasets to validate the distinct nature of these biases, distinguish their different real-world manifestations, and evaluate the effectiveness of a comprehensive list of bias assessment metrics in assessing the mitigation of these biases.
Further, we compare these two types of biases from multiple dimensions, including the underlying causes, debiasing methods, evaluation protocol, prevalent datasets, and future directions.
Last, we provide several suggestions aiming to guide researchers engaged in bias-related work to avoid confusion and further enhance clarity in the community.
URL: https://openreview.net/forum?id=LLiJ1WsL2e
---
Title: Predictive Control and Regret Analysis of Non-Stationary MDP with Look-ahead Information
Authors: Ziyi Zhang, yorie nakahira, Guannan Qu
Abstract: Policy design in non-stationary Markov Decision Processes (MDPs) is inherently challenging due to the complexities introduced by time-varying system transition and reward, which make it difficult for learners to determine the optimal actions for maximizing cumulative future rewards. Fortunately, in many practical applications, such as energy systems, look-ahead predictions are available, including forecasts for renewable energy generation and demand. In this paper, we leverage these look-ahead predictions and propose an algorithm designed to achieve low regret in non-stationary MDPs by incorporating such predictions. Our theoretical analysis demonstrates that, under certain assumptions, the regret decreases exponentially as the look-ahead window expands. When the system prediction is subject to error, the regret does not explode even if the prediction error grows sub-exponentially as a function of the prediction horizon. We validate our approach through simulations and confirm its efficacy in non-stationary environments.
URL: https://openreview.net/forum?id=uObs1YwXjQ
---
Title: On Efficient Bayesian Exploration in Model-Based Reinforcement Learning
Authors: Alberto Caron, Vasilios Mavroudis, Chris Hicks
Abstract: In this work, we address the challenge of data-efficient exploration in reinforcement learning by examining existing principled, information-theoretic approaches to intrinsic motivation. Specifically, we focus on a class of exploration bonuses that targets epistemic uncertainty rather than the aleatoric noise inherent in the environment. We prove that these bonuses naturally signal epistemic information gains and converge to zero once the agent becomes sufficiently certain about the environment’s dynamics and rewards, thereby aligning exploration with genuine knowledge gaps. Our analysis provides formal guarantees for IG-based approaches, which previously lacked theoretical grounding. To enable practical use, we also discuss tractable approximations via sparse variational Gaussian Processes, Deep Kernels and Deep Ensemble models. We then outline a general framework — Predictive Trajectory Sampling with Bayesian Exploration (PTS-BE) — which integrates model-based planning with information-theoretic bonuses to achieve sample-efficient deep exploration. We empirically demonstrate that PTS-BE substantially outperforms other baselines across a variety of environments characterized by sparse rewards and/or purely exploratory tasks.
URL: https://openreview.net/forum?id=Na02hDWqkF
---
Title: Batch Training for Streaming Time Series: A Transferable Augmentation Framework to Combat Distribution Shifts
Authors: Weiyang Zhang, Xinyang Chen, Yu Sun, Weili Guan, Liqiang Nie
Abstract: Multivariate time series forecasting, which predicts future dynamics by analyzing historical data, has become an essential tool in modern data analysis. With the development of deep models, batch-training based time series forecasting has made significant progress. However, in real-world applications, time series data is often collected incrementally in a streaming manner, with only a portion of the data available at each time step. As time progresses, distribution shifts in the data can occur, leading to a drastic decline in model performance. To address this challenge, online test-time adaptation and online time series forecasting have emerged as a promising solution. However, for the former, most online test-time adaptation methods are primarily designed for images and do not consider the specific characteristics of time series. As for the latter, online time series forecasting typically relies on updating the model with each newly collected sample individually, which may be problematic when the sample deviates significantly from the historical data distribution and contains noise, which may lead to a worse generalization performance.
In this paper, we propose Batch Training with Transferable Online Augmentation (BTOA), which enhances model performance through three key ideas while enabling batch training. First, to fully leverage historical information, Transferable Historical Sample Selection (THSS) is proposed with theoretical guarantees to select historical samples that are most similar to the test-time distribution. Then, to mitigate the negative impact of distribution shifts through batch training and take advantage of the unique characteristics of time series, Transferable Online Augmentation (TOA) is proposed to augment the selected historical samples from the perspective of amplitude and phase in the frequency domain in a two-stream manner. Finally, a prediction module that utilizes a series decomposition module and a two-stream forecaster is employed to extract the complex patterns in time series, boosting the prediction performance. Moreover, BTOA is a general approach that is readily pluggable into any existing batch-training based deep models.Comprehensive experiments under both ideal and practice experimental settings demonstrate that the proposed method exhibits superior performance across all seven benchmark datasets. Compared to state-of-the-art approaches, our method reduces the Mean Squared Error (MSE) by up to 13.7\%.
URL: https://openreview.net/forum?id=Ht7rlkRCHq
---
Title: Test-Time Fairness and Robustness in Large Language Models
Authors: Leonardo Cotta, Chris J. Maddison
Abstract: Frontier Large Language Models (LLMs) can be socially discriminatory or sensitive to spurious features of their inputs. Because only well-resourced corporations can train frontier LLMs, we need robust test-time strategies to control such biases. Existing solutions, which instruct the LLM to be fair or robust, rely on the model’s implicit understanding of bias. Causality provides a rich formalism through which we can be explicit about our debiasing requirements. Yet, as we show, a naive application of the standard causal debiasing strategy, counterfactual data augmentation, fails to fulfill individual-level debiasing requirements at test time. To address this, we develop stratified invariance, a flexible debiasing notion that can capture a range of debiasing requirements, from population level to individual level, through an additional measurement that stratifies the predictions. We developed a complete test for this new approach and introduced a data augmentation strategy that guarantees stratified invariance at test time under suitable assumptions, together with a prompting strategy that encourages stratified invariance in LLMs. We show that our prompting strategy, unlike implicit instructions, consistently reduces the bias of frontier LLMs across a suite of synthetic and real-world benchmarks without requiring additional data, finetuning or pre-training.
URL: https://openreview.net/forum?id=1fML4VF5FG
---
Title: Collaboration with Dynamic Open Ad Hoc Team via Team State Modelling
Authors: Jing Sun, Cong Zhang, Zhiguang Cao
Abstract: Open ad hoc teamwork presents the challenging problem of designing an autonomous agent that can rapidly adapt to collaborate with teammates without prior coordination in an open environment. Existing methods primarily rely on fixed, predefined teammate types, overlooking the fact that teammates may change dynamically. To address this limitation, we propose a novel reinforcement learning approach, the Open Online Teammate Adaptation Framework (Open-OTAF), which enables a controlled agent to collaborate with dynamic teammates in open ad hoc environments. To achieve this, the controlled agent employs a dual teamwork situation inference model to capture the current teamwork state, facilitating decision-making under partial observability. To handle the dynamic nature of teammate types, we first introduce a Chinese Restaurant Process-based model to categorize diverse teammate policies into distinct clusters, improving the efficiency of identifying teamwork situations. Next, to model heterogeneous agent relationships and accommodate a variable number of teammates, we represent the team as a heterogeneous graph and leverage heterogeneous graph attention neural networks to learn the representation of the teamwork situation. Extensive experiments across four challenging multi-agent benchmark tasks—Level-Based Foraging, Wolf-Pack, Cooperative Navigation, and FortAttack—demonstrate that our method successfully enables dynamic teamwork in open ad hoc settings. Open-OTAF outperforms state-of-the-art methods, achieving superior performance with faster convergence.
URL: https://openreview.net/forum?id=BukMU42P3G
---
Title: Do Concept Bottleneck Models Respect Localities?
Authors: Naveen Janaki Raman, Mateo Espinosa Zarlenga, Juyeon Heo, Mateja Jamnik
Abstract: Concept-based explainability methods use human-understandable intermediaries to produce explanations for machine learning models. These methods assume concept predictions can help understand a model's internal reasoning. In this work, we assess the degree to which such an assumption is true by analyzing whether concept predictors leverage "relevant" features to make predictions, a term we call locality. Concept-based models that fail to respect localities also fail to be explainable because concept predictions are based on spurious features, making the interpretation of the concept predictions vacuous. To assess whether concept-based models respect localities, we construct and use three metrics to characterize when models respect localities, complementing our analysis with theoretical results. Each of our metrics captures a different notion of perturbation and assess whether perturbing "irrelevant" features impacts the predictions made by a concept predictors. We find that many concept-based models used in practice fail to respect localities because concept predictors cannot always clearly distinguish distinct concepts. Based on these findings, we propose suggestions for alleviating this issue.
URL: https://openreview.net/forum?id=4mCkRbUXOf
---
Title: LC-PLM: Long-context Protein Language Modeling Using Bidirectional Mamba with Shared Projection Layers
Authors: Yingheng Wang, Zichen Wang, Gil Sadeh, Luca Zancato, Alessandro Achille, George Karypis, Huzefa Rangwala
Abstract: Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. Most protein LMs are based on the Transformer architecture trained on individual proteins with short context lengths. Such protein LMs cannot extrapolate to longer proteins and protein complexes well. They also fail to account for the underlying biological mechanisms carried out by biomolecular interactions and dynamics i.e., proteins often interact with other proteins, molecules, and pathways in complex biological systems. In this work, we propose LC-PLM based on an alternative protein LM architecture, BiMamba-S, built upon selective structured state-space models, to learn high-quality universal protein representations at the amino acid token level using masked language modeling. We also introduce its graph-contextual variant, LC-PLM, which contextualizes protein-protein interaction (PPI) graphs for a second stage of training. LC-PLM demonstrates favorable neural scaling laws, better length extrapolation capability, and up to 30% and 16% improvements on protein downstream tasks compared to Transformer-based ESM-2 when trained with 100B and 1T tokens, respectively. LC-PLM-G further trained within the context of PPI graphs shows promising results on protein structure and function prediction tasks. Our study demonstrates the benefit of increasing the context size with computationally efficient LM architecture (e.g., structured state space models) in learning universal protein representations and incorporating molecular interaction contexts contained in biological graphs. Model is available at github.com/amazon-science/LC-PLM.
URL: https://openreview.net/forum?id=dWvztQzfy4
---
Title: DisDet: Exploring Detectability of Backdoor Attack on Diffusion Models
Authors: Yang Sui, Huy Phan, Jinqi Xiao, Tianfang Zhang, Zijie Tang, Cong Shi, Yan Wang, Yingying Chen, Bo Yuan
Abstract: In the exciting generative AI era, the diffusion model has emerged as a very powerful and widely adopted content-generation tool. Very recently, some pioneering works have shown the vulnerability of the diffusion model against backdoor attacks, calling for in-depth analysis and investigation of the security challenges. In this paper, we explore the detectability of the poisoned noise input for the backdoored diffusion models, an important performance metric yet little explored in the existing works. Starting from the perspective of a defender, we first analyze the distribution discrepancy of the trigger pattern in the existing diffusion backdoor attacks. Based on this finding, we propose a trigger detection mechanism that can effectively identify the poisoned input noise. Then, from the attack side, we propose a backdoor attack strategy that can learn the unnoticeable trigger to evade our proposed detection scheme. Our empirical evaluations across various diffusion models and datasets demonstrate the effectiveness of the proposed trigger detection and detection-evading attack strategy. For trigger detection, our distribution discrepancy-based solution can achieve a 100% detection rate for the Trojan triggers used in the existing works. For evading trigger detection, our proposed stealthy trigger design approach performs end-to-end learning to make the distribution of poisoned noise input approach that of benign noise, enabling nearly 100% detection pass rate with very high attack and benign performance for the backdoored diffusion models.
URL: https://openreview.net/forum?id=SfqCaAOF1S
---
Title: Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentum
Authors: Riccardo Zaccone, Sai Praneeth Karimireddy, Carlo Masone, Marco Ciccone
Abstract: Federated Learning (FL) has emerged as the state-of-the-art approach for learning from decentralized data in privacy-constrained scenarios. However, system and statistical challenges hinder its real-world applicability, requiring efficient learning from edge devices and robustness to data heterogeneity. Despite significant research efforts, existing approaches often degrade severely due to the joint effect of heterogeneity and partial client participation. In particular, while momentum appears as a promising approach for overcoming statistical heterogeneity, in current approaches its update is biased towards the most recently sampled clients. As we show in this work, this is the reason why it fails to outperform FedAvg, preventing its effective use in real-world large-scale scenarios.
In this work, we propose a novel Generalized Heavy-Ball Momentum (GHBM) and theoretically prove it enables convergence under unbounded data heterogeneity in cyclic partial participation, thereby advancing the understanding of momentum's effectiveness in FL.
We then introduce adaptive and communication-efficient variants of GHBM that match the communication complexity of FedAvg in settings where clients can be stateful.
Extensive experiments on vision and language tasks confirm our theoretical findings, demonstrating that GHBM substantially improves state-of-the-art performance under random uniform client sampling, particularly in large-scale settings with high data heterogeneity and low client participation.
URL: https://openreview.net/forum?id=LNoFjcLywb
---
Title: Are Convex Optimization Curves Convex?
Authors: Guy Barzilai, Ohad Shamir, Moslem Zamani
Abstract: In this paper, we study when we might expect the optimization curve induced by gradient descent to be \emph{convex} -- precluding, for example, an initial plateau followed by a sharp decrease, making it difficult to decide when optimization should stop. Although such undesirable behavior can certainly occur when optimizing general functions, might it also occur in the benign and well-studied case of smooth convex functions? As far as we know, this question has not been tackled in previous work. We show, perhaps surprisingly, that the answer crucially depends on the choice of the step size. In particular, for the range of step sizes which are known to result in monotonic convergence to an optimal value, we characterize a regime where the optimization curve will be provably convex, and a regime where the curve can be non-convex. We also extend our results to gradient flow, and to the closely-related but different question of whether the gradient norm decreases monotonically.
URL: https://openreview.net/forum?id=TZtpxselK2
---
Title: Evolution guided generative flow networks
Authors: Zarif Ikram, Ling Pan, Dianbo Liu
Abstract: Generative Flow Networks (GFlowNets) are a family of probabilistic generative models recently invented that learn to sample compositional objects proportional to their rewards. One big challenge of GFlowNets is training them effectively when dealing with long time horizons and sparse rewards. To address this, we propose Evolution guided generative flow networks (EGFN), a simple but powerful augmentation to the GFlowNets training using Evolutionary algorithms (EA). Our method can work on top of any GFlowNets training objective, by training a set of agent parameters using EA, storing the resulting trajectories in the prioritized replay buffer, and training the GFlowNets agent using the stored trajectories. We present a thorough investigation over a wide range of toy and real-world benchmark tasks showing the effectiveness of our method in handling long trajectories and sparse rewards. We release the code at http://github.com/zarifikram/egfn.
URL: https://openreview.net/forum?id=UgZIR6TF5N
---
Title: Graph Fourier Neural ODEs: Modeling Spatial-temporal Multi-scales in Molecular Dynamics
Authors: Fang Sun, Zijie Huang, Haixin Wang, Huacong Tang, Xiao Luo, Wei Wang, Yizhou Sun
Abstract: Accurately predicting long-horizon molecular dynamics (MD) trajectories remains a significant challenge, as existing deep learning methods often struggle to retain fidelity over extended simulations. We hypothesize that one key factor limiting accuracy is the difficulty of capturing interactions that span distinct spatial and temporal scales—ranging from high-frequency local vibrations to low-frequency global conformational changes. To address these limitations, we propose **Graph Fourier Neural ODEs (GF-NODE)**, integrating a graph Fourier transform for spatial frequency decomposition with a Neural ODE framework for continuous-time evolution. Specifically, GF-NODE first decomposes molecular configurations into multiple spatial frequency modes using the graph Laplacian, then evolves the frequency components in time via a learnable Neural ODE module that captures both local and global dynamics, and finally reconstructs the updated molecular geometry through an inverse graph Fourier transform. By explicitly modeling high- and low-frequency phenomena in this unified pipeline, GF-NODE more effectively captures long-range correlations and local fluctuations alike. We provide theoretical insight through heat equation analysis on a simplified diffusion model, demonstrating how graph Laplacian eigenvalues can determine temporal dynamics scales, and crucially validate this correspondence through comprehensive empirical analysis on real molecular dynamics trajectories showing quantitative spatial-temporal correlations across diverse molecular systems. Experimental results on challenging MD benchmarks, including MD17 and alanine dipeptide, demonstrate that GF-NODE achieves state-of-the-art accuracy while preserving essential geometrical features over extended simulations. These findings highlight the promise of bridging spectral decomposition with continuous-time modeling to improve the robustness and predictive power of MD simulations. Our implementation is publicly available at https://github.com/FrancoTSolis/GF-NODE-code .
URL: https://openreview.net/forum?id=XK7cIdj6Fz
---
Title: A Stochastic Polynomial Expansion for Uncertainty Propagation through Networks
Authors: Songhan Zhang, ShiNung Ching
Abstract: Network-based machine learning constructs are becoming more prevalent in sensing and decision-making systems. As these systems are implemented in safety-critical environments such as pedestrian detection and power management, it is crucial to evaluate confidence in their decisions. At the heart of this problem is a need to understand and characterize how errors at the input of networks become progressively expanded or contracted as signals move through layers, especially in light of the non-trivial nonlinearities manifest throughout modern machine learning architectures. When sampling methods become expensive due to network size or complexity, approximation is needed and popular methods include Jacobian (first order Taylor) linearization and stochastic linearization. However, despite computational tractability, the accuracy of these methods can break down in situations with moderate to high input uncertainty.
Here, we present a generalized method of propagating variational multivariate Gaussian distributions through neural networks. We propose a modified Taylor expansion function for nonlinear transformation of Gaussian distributions, with an additional approximation in which the polynomial terms act on independent Gaussian random variables (which are identically distributed). With these approximated higher order terms (HOTs), we obtain significantly more accurate estimation of layer-wise distributions. Despite the introduction of the HOTs, this method can propagate a full covariance matrix with a complexity of $\boldsymbol{O}(n^2)$ (and $\boldsymbol{O}(n)$ if only propagating marginal variance), comparable to Jacobian linearization. Thus, our method finds a balance between efficiency and accuracy. We derived the closed form solutions for this approximate Stochastic Taylor expansion for seven commonly used nonlinearities and verified the effectiveness of our method in deep residual neural networks, Bayesian neural networks, and variational autoencoders. This general method can be integrated into use-cases such as Kalman filtering, adversarial training, and variational learning.
URL: https://openreview.net/forum?id=lyDRBhUjhv
---
Title: Fair Online Influence Maximization
Authors: Xiangqi Wang, Shaokun Zhang, Jose Efraim Aguilar Escamilla, Qingyun Wu, Xiangliang Zhang, Jian Kang, Huazheng Wang
Abstract: Fair influence maximization in networks has been actively studied to ensure equity in fields like viral marketing and public health. Existing studies often assume an offline setting, meaning that the learner identifies a set of seed nodes with known per-edge activation probabilities. In this paper, we study the problem of fair online influence maximization, i.e., without knowing the ground-truth activation probabilities. The learner in this problem aims to maximally propagate the information among demographic groups, while interactively selecting seed nodes and observing the activation feedback on the fly. We propose Fair Online Influence Maximization (FOIM) framework that can solve the online influence maximization problem under a wide range of fairness notions. Given a fairness notion, FOIM solves the problem with a combinatorial multi-armed bandit algorithm for balancing exploration-exploitation and an offline fair influence maximization oracle for seed nodes selection. FOIM enjoys sublinear regret when the fairness notion satisfies two mild conditions, i.e., monotonicity and bounded smoothness. Our analyses show that common fairness notions, including maximin fairness, diversity fairness, and welfare function, all satisfy the condition, and we prove the corresponding regret upper bounds under these notions. Extensive empirical evaluations on three real-world networks demonstrate the efficacy of our proposed framework.
URL: https://openreview.net/forum?id=T1NjRBI5xs
---
New submissions
===============
Title: Empirical Comparison of Membership Inference Attacks in Deep Transfer Learning
Abstract: With the emergence of powerful large-scale foundation models, the training paradigm is increasingly shifting from from-scratch training to transfer learning. This enables high utility training with small, domain-specific datasets typical in sensitive applications.
Membership inference attacks (MIAs) provide an empirical estimate of the privacy leakage by machine learning models. Yet, prior assessments of MIAs against models fine-tuned with transfer learning rely on a small subset of possible attacks. We address this by comparing performance of diverse MIAs in transfer learning settings to help practitioners identify the most efficient attacks for privacy risk evaluation. We find that attack efficacy decreases with the increase in training data for score-based MIAs. We find that there is no one MIA which captures all privacy risks in models trained with transfer learning. While the Likelihood Ratio Attack (LiRA) demonstrates superior performance across most experimental scenarios, the Inverse Hessian Attack (IHA) proves to be more effective against models fine-tuned on PatchCamelyon dataset in high data regime.
URL: https://openreview.net/forum?id=UligTUCgdt
---
Title: On the Importance of Pretraining Data Alignment for Atomic Property Prediction
Abstract: This paper challenges the recent paradigm in atomic property prediction that links progress to growing dataset sizes and computational resources. We show that pretraining on a carefully selected task-aligned dataset can match or even surpass large-scale pretraining, while using only 1/24th of the computational cost. We introduce the Chemical Similarity Index (CSI), a simple metric for molecular graphs inspired by the Fréchet Inception Distance in computer vision, which quantifies the alignment between upstream pretraining datasets and downstream tasks. By selecting the most aligned dataset with minimal CSI distance, we show that models pretrained on a smaller, focused dataset consistently achieve better performance on downstream tasks than those pretrained on massive, mixed datasets such as JMP. This holds even when the mixed dataset includes the upstream dataset most aligned with the downstream task. Counterintuitively, we also find that indiscriminately adding more data can degrade model performance when the additional data is poorly aligned with the target task. Our findings highlight that quality often outperforms quantity in pretraining for atomic property prediction.
URL: https://openreview.net/forum?id=jfD9BsrDTb
---
Title: FORTRESS: Fast, Tuning-Free Retrieval Ensemble for Scalable LLM Safety
Abstract: The rapid adoption of Large Language Models in user-facing applications has magnified security risks, as adversarial prompts continue to circumvent built-in safeguards with increasing sophistication. Current external safety classifiers predominantly rely on supervised fine-tuning—a computationally expensive approach that proves brittle against novel attacks and demands constant retraining cycles. We present FORTRESS, a Fast, Orchestrated Tuning-free Retrieval Ensemble for Scalable Safety that eliminates the need for costly, gradient-based fine-tuning. Our framework unifies semantic retrieval and dynamic perplexity analysis with a single instruction-tuned LLM, creating an efficient pipeline that adapts to emerging threats through simple data ingestion rather than model retraining. FORTRESS employs a novel dynamic ensemble strategy that intelligently weighs complementary signals: semantic similarity for known threat patterns and statistical anomaly detection for zero-day attacks. Extensive evaluation across nine safety benchmarks demonstrates that FORTRESS achieves state-of-the-art performance with an F1 score of 91.6\%, while operating over five times faster than leading fine-tuned classifiers. Its data-centric design enables rapid adaptation to new threats through simple data ingestion—a process we show improves performance without a latency trade-off—offering a practical, scalable, and robust approach to LLM safety.
URL: https://openreview.net/forum?id=lCn7RT9DGq
---
Title: Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computation
Abstract: Many domains, from deep learning to finance, require compounding real numbers over long sequences, often leading to catastrophic numerical underflow or overflow. We introduce generalized orders of magnitude (GOOMs), a principled extension of traditional orders of magnitude that enables stable computation over significantly larger dynamic ranges of real numbers than previously possible. We implement GOOMs in PyTorch, along with an efficient custom parallel prefix scan, to support native execution on parallel hardware such as GPUs. We demonstrate that GOOMs outperform traditional approaches with three representative experiments: (1) compounding up to one million real matrix products beyond standard floating-point limits; (2) estimating spectra of Lyapunov exponents in parallel, using a novel selective-resetting method to prevent state colinearity; and (3) training deep recurrent neural networks that maintain long-range dependencies without numerical degradation, allowing non-diagonal recurrent states to fluctuate freely as we compute them via a parallel scan over all time steps. Our results show that GOOMs, combined with efficient parallel scanning, offer a scalable and numerically robust alternative to conventional floating-point arithmetic for high-dynamic-range applications.
URL: https://openreview.net/forum?id=SUuzb0SOGu
---
Title: SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
Abstract: We introduce SpidR, a self-supervised speech representation model that efficiently learns strong representations for spoken language modeling. It is trained on unlabelled speech using a masked prediction objective combined with self-distillation and online clustering. The intermediate layers of the student model learn to predict assignments derived from the teacher intermediate layers. This learning objective stabilizes the online clustering procedure compared to previous approaches, resulting in higher-quality codebooks. SpidR outperforms previous state-of-the-art methods on downstream language modeling metrics while significantly reducing pretraining time, requiring only a day to pretrain with 16 GPUs instead of a week. We will open-source the training code and model checkpoints upon acceptance.
URL: https://openreview.net/forum?id=E7XAFBpfZs
---
Title: In-Context Learning with Hypothesis-Class Guidance
Abstract: Recent research has investigated the underlying mechanisms of in-context learning (ICL) both theoretically and empirically, often using data generated from simple function classes. However, the existing work often focuses on the sequence consisting solely of labeled examples, while in practice, labeled examples are typically accompanied by an instruction, providing some side information about the task. In this work, we propose ICL with hypothesis-class guidance (ICL-HCG), a novel synthetic data model for ICL where the input context consists of the literal description of a (finite) hypothesis class $\mathcal{H}$ and $(x,y)$ pairs from a hypothesis chosen from $\mathcal{H}$. Under our framework ICL-HCG, we conduct extensive experiments to explore: (i) a variety of generalization abilities to new hypothesis classes; (ii) different model architectures; (iii) sample complexity; (iv) in-context data imbalance; (v) the role of instruction; and (vi) the effect of pretraining hypothesis diversity. As a result, we show that (a) Transformers can successfully learn ICL-HCG and generalize to unseen hypotheses and unseen hypothesis classes, and (b) compared with ICL without instruction, ICL-HCG achieves significantly higher accuracy, demonstrating the role of instructions.
URL: https://openreview.net/forum?id=VjIBgvOeLg
---
Title: Why Settle for Mid: A Probabilistic Viewpoint to Spatial Relationship Alignment in Text-to-image Models
Abstract: Despite the ability of text-to-image models to generate high-quality, realistic, and diverse images, they face challenges in compositional generation, often struggling to accurately represent details specified in the input prompt. A prevalent issue in compositional generation is the misalignment of spatial relationships, as models often fail to faithfully generate images that reflect the spatial configurations specified between objects in the input prompts.
To address this challenge, we propose a novel probabilistic framework for modeling the relative spatial positioning of objects in a scene, leveraging the concept of \textit{Probability of Superiority (PoS)}. Building on this insight, we make two key contributions. First, we introduce a novel evaluation metric, \textit{PoS-based Evaluation (PSE)}, designed to assess the alignment of 2D and 3D spatial relationships between text and image, with improved adherence to human judgment. Second, we propose \textit{PoS-based Generation (PSG)}, an inference-time method that improves the alignment of 2D and 3D spatial relationships in T2I models without requiring fine-tuning. \emph{PSG} employs a Part-of-Speech PoS-based reward function that can be utilized in two distinct ways: (1) as a \textit{gradient-based} guidance mechanism applied to the cross-attention maps during the denoising steps, or (2) as a \textit{search-based} strategy that evaluates a set of initial noise vectors to select the best one. Extensive experiments demonstrate that the \emph{PSE} metric exhibits stronger alignment with human judgment compared to traditional center-based metrics, providing a more nuanced and reliable measure of complex spatial relationship accuracy in text-image alignment. Furthermore, \emph{PSG} significantly enhances the ability of text-to-image models to generate images with specified spatial configurations, outperforming state-of-the-art methods across multiple evaluation metrics and benchmarks.
URL: https://openreview.net/forum?id=mFlanJKVFD
---
Title: Random Fourier Feature Shared Latent Variable Models for User Preference Visualization and Analysis
Abstract: Understanding user preferences plays a crucial role in domains where strategies are designed by domain experts, such as personalized recommendations, targeted marketing, and human-centered interface design.
However, many existing methods prioritize predictive accuracy over model transparency, limiting their use in settings that require interpretability.
To address this issue, we propose the \textbf{Random Fourier Feature Shared Latent Variable Model (RFSLVM)}, a probabilistic generative model based on Gaussian processes that enables interpretable analyses of user preferences.
RFSLVM jointly models two data modalities: real-valued item features and binary user ratings.
It learns a \textit{two-dimensional} \textbf{visualization space} that captures relationships among items and user ratings.
Additionally, it infers \textit{user-specific} \textbf{preference vectors} that are compact and continuous representations of generally nonlinear preference functions.
These vectors support tasks such as measuring user similarity and performing preference-based clustering, thereby facilitating downstream analysis and decision-making.
We evaluate RFSLVM on multiple real-world datasets and find that it performs competitively against baseline models, while maintaining interpretability.
In addition, we demonstrate the utility of the learned representations through qualitative analyses, including hierarchical clustering and the identification of latent preference patterns.
These findings suggest that RFSLVM offers a practical and interpretable approach to modeling user preferences in real-world applications.
URL: https://openreview.net/forum?id=35HRXmXPz7
---
Title: Large Language Models are Locally Linear Mappings
Abstract: Despite significant progress in transformer interpretability, an understanding of the computational mechanisms of large language models (LLMs) remains a fundamental challenge. We demonstrate that the inference operation of LLMs can be mapped to an equivalent linear system that nearly exactly reconstructs the predicted output embedding for a given input sequence. Extending techniques from image diffusion models that exhibit local or piecewise linearity, we strategically detach components of the gradient computation with respect to an input sequence for a next-token prediction such that the Jacobian of the model reproduces the output with one linear operation per input token. We demonstrate this approach across models, including Qwen 3, Gemma 3, Llama 3, Phi 4, Mistral Ministral and OLMo 2, up to Llama 3.3 70B Q4. With the singular value decomposition of the detached Jacobian, we show that these LLMs operate in extremely low-dimensional subspaces where the largest singular vectors decode to distinct concepts related to possible output tokens. We examine the equivalent linear operation of each successive layer (and its attention and multilayer perceptron components) and observe the emergence of semantic concepts. We demonstrate that the detached Jacobian of middle layer representations can be used as a steering operator to insert semantic concepts into unrelated text, which could be useful for improving safety and decreasing bias. Despite their expressive power and global nonlinearity, modern LLMs can be interpreted through locally linear decompositions that provide insights into their internal representations and reveal interpretable semantic structures in the next-token prediction process.
URL: https://openreview.net/forum?id=oDWbJsIuEp
---
Title: SaFARi: State-Space Models for Frame-Agnostic Representation
Abstract: State-Space Models (SSMs) have re-emerged as a powerful tool for online function approximation, and as the backbone of machine learning models for long-range dependent data.
However, to date, only a few polynomial bases have been explored for this purpose, and the state-of-the-art implementations were built upon the best of a few limited options. In this paper, we present a generalized method for building an SSM with any frame or basis, rather than being restricted to polynomials.
This framework encompasses the approach known as HiPPO, but also permits an infinite diversity of other possible "species" within the SSM architecture. We dub this approach SaFARi: SSMs for Frame-Agnostic Representation.
URL: https://openreview.net/forum?id=UAgxU8gBtv
---
Title: Teaching Diffusion Models to Ground Alpha Matte
Abstract: The power of visual language models is showcased in visual understanding tasks, where language-guided models achieve impressive flexibility and precision. In this paper, we extend this capability to the challenging domain of image matting by framing it as a soft grounding problem, enabling a single diffusion model to handle diverse objects, textures, and transparencies, all directed by descriptive text prompts. Our method teaches the diffusion model to ground alpha mattes by guiding it through a process of instance-level localization and transparency estimation. First, we introduce an intermediate objective that trains the model to accurately localize semantic components of the matte based on natural language cues, establishing a robust spatial foundation. Building on this, the model progressively refines its transparency estimation abilities, using the learned semantic structure as a prior to enhance the precision of alpha matte predictions. By treating spatial localization and transparency estimation as distinct learning objectives, our approach allows the model to fully leverage the semantic depth of diffusion models, removing the need for rigid visual priors. Extensive experiments highlight our model’s adaptability, precision, and computational efficiency, setting a new benchmark for flexible, text-driven image matting solutions.
URL: https://openreview.net/forum?id=2gNy9Yeg8J
---
Title: Bias Spillover in Language Models: A Review of Political Alignment, Regional Fragility, and Multi-Axis Risks
Abstract: Efforts to mitigate social bias in large language models (LLMs) often address individual dimensions such as gender or political ideology in isolation. However, interventions targeting one axis can unintentionally influence others, a phenomenon we term bias spillover. This paper presents a structured review of over 80 studies, synthesizing empirical and theoretical evidence of cross-axis interference in model behavior. We identify four core mechanisms that drive spillover, including representational entanglement and conflicts introduced during fine-tuning, and we clarify the distinction between co-occurring biases and causal spillover. Our analysis reveals major limitations in current auditing practices, including the lack of standardized tools for measuring intersectional effects and limited coverage of non-Western and multilingual contexts. In response, we introduce a typology of auditing frameworks and recommend mitigation strategies that account for entangled social representations. These findings underscore the need for spillover-aware evaluation and debiasing approaches that move beyond isolated fairness metrics and reflect the complexity of real-world sociopolitical contexts.
URL: https://openreview.net/forum?id=hv82NTjEhs
---
Title: The kernel of graph indices for vector search
Abstract: The most popular graph indices for vector search use principles from computational geometry to build the graph. Hence, their formal graph navigability guarantees are only valid in Euclidean space. In this work, we show that machine learning can be used to build graph indices for vector search in metric and non-metric vector spaces (e.g., for inner product similarity). From this novel perspective, we introduce the Support Vector Graph (SVG), a new type of graph index that leverages kernel methods to establish the graph connectivity and that comes with formal navigability guarantees valid in metric and non-metric vector spaces. In addition, we interpret the most popular graph indices, including HNSW and DiskANN, as particular specializations of SVG and show that new indices can be derived from the principles behind this specialization. Finally, we propose SVG-L0 that incorporates an $\ell_0$ sparsity constraint into the SVG kernel method to build graphs with a bounded out-degree. This yields a principled way of implementing this practical requirement, in contrast to the traditional heuristic of simply truncating the out edges of each node. Additionally, we show that SVG-L0 has a self-tuning property that avoids the heuristic of using a set of candidates to find the out-edges of each node and that keeps its computational complexity in check.
URL: https://openreview.net/forum?id=28LzHsSjef
---
Title: Say My Name: a Model's Bias Discovery Framework
Abstract: Due to the broad applicability of deep learning to downstream tasks and end-to-end training capabilities in the last few years, increasingly more concerns about potential biases to specific, non-representative patterns have been raised. Many works focusing on unsupervised debiasing leverage the tendency of deep models to learn “easier” samples, for example by clustering the latent space to obtain bias pseudo-labels. However, their interpretation is not trivial as it does not provide semantic information about the bias features. To address this issue, we introduce “Say My Name” (SaMyNa), a tool to identify semantic biases within deep models. Unlike existing methods, our approach focuses on biases learned by the model, enhancing explainability through a text-based pipeline. Applicable during either training or post-hoc validation, our method can disentangle task-related information and propose itself as a tool to analyze biases. Evaluation on typical benchmarks demonstrates its effectiveness in detecting biases and even disclaiming them. When sided with a traditional debiasing approach for bias mitigation, it can achieve state-of-the-art performance while having the advantage of associating a semantic meaning to the discovered bias.
URL: https://openreview.net/forum?id=EuUxiiDZ7d
---
Title: Cluster Agnostic Network Lasso Bandits
Abstract: We consider a multi-task contextual bandit setting, where the learner is given a graph encoding relations between the bandit tasks. The tasks' preference vectors are assumed to be piecewise constant over the graph, forming clusters. At every round, we estimate the preference vectors by solving an online network lasso problem with a suitably chosen, time-dependent regularization parameter. We establish a novel oracle inequality relying on a convenient restricted eigenvalue assumption. Our theoretical findings highlight the importance of dense intra-cluster connections and sparse inter-cluster ones. That results in a sublinear regret bound significantly lower than its counterpart in the independent task learning setting. Finally, we support our theoretical findings by experimental evaluation against graph bandit multi-task learning and online clustering of bandits algorithms.
URL: https://openreview.net/forum?id=QjAyoMP1DD
---
Title: Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
Abstract: This work presents an analytical framework for the design and analysis of LLM-based algorithms, i.e., algorithms that contain one or multiple calls of large language models (LLMs) as sub-routines and critically rely on the capabilities of LLMs. While such algorithms, ranging from basic LLM calls with prompt engineering to complicated LLM-powered agentic workflows and compound AI systems, have achieved remarkable empirical success, their design and optimization oftentimes require extensive trial-and-errors and case-by-case analysis. Our proposed framework serves as an attempt to mitigate such headaches, offering a formal and systematic approach for analyzing how the accuracy and efficiency of an LLM-based algorithm will be impacted by critical design choices, such as the pattern and granularity of task decomposition, or the prompt for each LLM call. Through a wide range of case studies covering diverse algorithm patterns (including parallel/hierarchical/recursive task decomposition and generic directed acyclic graphs), we demonstrate the proposed framework in action and derive interesting insights that generalize across scenarios, accompanied by empirical validation.
URL: https://openreview.net/forum?id=nJ7RECkxQr
---
Title: Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model
Abstract: Reinforcement learning from human feedback (RLHF) has been widely adopted to align language models (LMs) with human preference. Previous RLHF works typically take a bandit formulation, which, though intuitive, ignores the sequential nature of LM generation and can suffer from the sparse reward issue. While recent works propose dense token-level RLHF, treating each token as an action may be oversubtle to proper reward assignment. In this paper, we seek to get the best of both by training and utilizing a segment-level reward model, which assigns a reward to each semantically complete text segment that spans over a short sequence of tokens. For reward learning, our method allows dynamic text segmentation and compatibility with standard sequence-preference datasets. For effective RL-based LM training against segment reward, we generalize the classical scalar bandit reward normalizers into location-aware normalizer functions and interpolate the segment reward for further densification. Our method performs competitively on three popular RLHF benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation studies are conducted to further demonstrate our method.
URL: https://openreview.net/forum?id=YhLlqD0UNi
---
Title: Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysis
Abstract: Large Language Models (LLMs), already shown to ace various unstructured text comprehension tasks, have also remarkably been shown to tackle table (structured) comprehension tasks without specific training. Building on earlier studies of LLMs for tabular tasks, we probe how in-context learning (ICL), model scale, instruction tuning, and domain bias affect Tabular QA (TQA) robustness by testing LLMs, under diverse augmentations and perturbations, on diverse domains: Wikipedia-based $\textbf{WTQ}$, financial $\textbf{TAT-QA}$, and scientific $\textbf{SCITAB}$. Although instruction tuning and larger, newer LLMs deliver stronger, more robust TQA performance, data contamination and reliability issues, especially on $\textbf{WTQ}$, remain unresolved. Through an in-depth attention analysis, we reveal a strong correlation between perturbation-induced shifts in attention dispersion and the drops in performance, with sensitivity peaking in the model's middle layers. We highlight the need for improved interpretable methodologies to develop more reliable LLMs for table comprehension. Through an in-depth attention analysis, we reveal a strong correlation between perturbation-induced shifts in attention dispersion and performance drops, with sensitivity peaking in the model's middle layers. Based on these findings, we argue for the development of structure-aware self-attention mechanisms and domain-adaptive processing techniques to improve the transparency, generalization, and real-world reliability of LLMs on tabular data.
URL: https://openreview.net/forum?id=PYHIDN9Wuq
---
Title: Avoiding Structural Pitfalls: Self-Supervised Low-Rank Feature Tuning for Graph Test-Time Adaptation
Abstract: Pre-trained graph neural networks (GNNs) have demonstrated significant success in leveraging large-scale graph data to learn transferable representations. However, their performance often degrades under distribution shifts, particularly in real-world scenarios where test labels are unavailable. To address this challenge, we propose Graph Optimization via Augmented Transformations (GOAT), a novel self-supervised test-time tuning paradigm that adapts pre-trained GNNs to distribution-shifted test data by focusing exclusively on node feature transformations. By avoiding complex and often suboptimal graph structure transformations, GOAT overcomes the limitations of existing data-centric methods.
To further address the issue of transformation collapse, where feature transformations converge to trivial solutions, we introduce a parameter-efficient low-rank adapter that generates diverse transformations tailored to individual input graphs. This design not only enhances adaptation performance but also improves interpretability by avoiding modifications to the graph structure. Through extensive experiments on six real-world datasets with diverse distribution shifts, we demonstrate that GOAT achieves consistent performance improvements across different pre-trained GNN backbones, outperforming state-of-the-art test-time adaptation methods.
URL: https://openreview.net/forum?id=yiS6q42LLt
---
Title: Unlocking the Power of GANs in Non-Autoregressive Text Generation under Weak Conditions
Abstract: Non-autoregressive (NAR) models once received great attention from the community, but obtain much less attention in the quest for general artificial intelligence. Our analyses reveal that the convergence problem in existing NAR models trained under Maximum Likelihood Estimation (MLE) is more severe in tasks where input does not provide the definite semantic meaning of the output. These input conditions, which we denote as weak conditions, cover most ``creative'' tasks, so existing NAR models struggle to obtain satisfactory performance in these tasks and are only developed in limited scenarios. This causes existing NAR models to struggle to keep pace with the rapidly evolving demands of diverse and challenging tasks. Different with MLE, which is incompatible with NAR models, Generative Adversarial Networks (GANs) are more suitable for NAR models in terms of theoretical convergence and inference manners. We thus propose an Adversarial Non-autoregressive Transformer (ANT) based on GANs for weak condition tasks. ANT supports two features: 1) Position-Aware Self-Modulation to provide more effective input signals, and 2) Dependency Feed Forward Network to strengthen its capacity in dependency modeling. The experimental results demonstrate that ANT achieves comparable performance with mainstream models in much higher efficiency and has great potential in various applications like latent interpolation and semi-supervised learning.
URL: https://openreview.net/forum?id=cBOwedlMTQ
---
Title: Unifying VXAI: A Systematic Review and Framework for the Evaluation of Explainable AI
Abstract: Modern AI systems frequently rely on opaque black-box models, most notably Deep Neural Networks, whose performance stems from complex architectures with millions of learned parameters. While powerful, their complexity poses a major challenge to trustworthiness, particularly due to a lack of transparency. Explainable AI (XAI) addresses this issue by providing human-understandable explanations of model behavior. However, to ensure their usefulness and trustworthiness, such explanations must be rigorously evaluated. Despite the growing number of XAI methods, the field lacks standardized evaluation protocols and consensus on appropriate metrics. To address this gap, we conduct a systematic literature review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and introduce a unified framework for the eValuation of XAI (VXAI). We identify 362 relevant publications and aggregate their contributions into 41 functionally similar metric groups. In addition, we propose a three-dimensional categorization scheme spanning explanation type, evaluation contextuality, and explanation quality desiderata. Our framework provides the most comprehensive and structured overview of VXAI to date. It supports systematic metric selection, promotes comparability across methods, and offers a flexible foundation for future extensions.
URL: https://openreview.net/forum?id=wAvFLe7o0E
---
Title: EnsDiff: Ensemble Precipitation Nowcasting with Diffusion
Abstract: Operational Numerical Weather Prediction (NWP) precipitation nowcasting usually considers forecast reliability by utilizing an ensemble of model forecasts. Existing data-driven methods often optimize MSE deterministically or resort to probabilistic forecasting with generative models. However, they only emphasize the optimization of the point forecast metrics, which makes it challenging to balance the trade-off between the optimization of accuracy and uncertainty. Human experts can hardly make an appropriate decision with an ensemble forecast when forecast calibration and sharpness are not considered. In this paper, we propose EnsDiff, which models the probability distribution to produce ensemble diffusion predictions. Not only does it outperform the existing models on a proper scoring rule, Continuous Ranked Probability Score (CRPS), but it also outperforms others on the deterministic metrics. Extensive experiments show that EnsDiff can enhance probabilistic, deterministic skills, and perceptual quality, outperforming state-of-the-art models.
URL: https://openreview.net/forum?id=oI705TpPlZ
---
Title: Mesh-Informed Neural Operator : A Transformer Generative Approach
Abstract: Generative models in function spaces, situated at the intersection of generative modeling and operator learning, are attracting increasing attention due to their immense potential in diverse scientific and engineering applications. While functional generative models are theoretically domain- and discretization-agnostic, current implementations heavily rely on the Fourier Neural Operator (FNO), limiting their applicability to regular grids and rectangular domains. To overcome these critical limitations, we introduce the Mesh-Informed Neural Operator (MINO). By leveraging graph neural operators and cross-attention mechanisms, MINO offers a principled, domain- and discretization-agnostic backbone for generative modeling in function spaces. This advancement significantly expands the scope of such models to more diverse applications in generative, inverse, and regression tasks. Furthermore, MINO provides a unified perspective on integrating neural operators with general advanced deep learning architectures. Finally, we introduce a suite of standardized evaluation metrics that enable objective comparison of functional generative models, addressing another critical gap in the field.
URL: https://openreview.net/forum?id=K8qAuRfv0G
---
Title: Trustworthiness in Generative Foundation Models Is Still Poorly Understood
Abstract: Generative Foundation Models (GenFMs) have seen extensive deployment across diverse domains, significantly impacting society yet simultaneously raising critical concerns about their trustworthiness, including misinformation, safety risks, fairness, and privacy violations. Recognizing the complex nature of these issues, to bridge the gap between abstract principles and operational actions throughout the GenFM lifecycle, we propose a flexible and multidimensional set of trustworthiness guidelines. These guidelines incorporate ethical principles, legal standards, and operational needs, addressing key dimensions such as fairness, transparency, human oversight, accountability, robustness, harmlessness, truthfulness, and privacy. Our guidelines serve as adaptable tools to bridge abstract principles and practical implementations across varied scenarios. Building upon these guidelines, we identify several core challenges currently unresolved in both theory and practice. Specifically, we examine the dynamic tension between adaptability and consistent safety, the ambiguities in defining and detecting harmful content, and the balancing of trustworthiness with model utility. Through our analysis, we reveal that the trustworthiness of GenFMs remains inadequately understood, highlighting the necessity for continuous, context-sensitive evaluation approaches. Consequently, we propose potential solutions and methodological directions, emphasizing integrated strategies that combine internal alignment mechanisms with external safeguards. Our findings underscore that trustworthiness is not static but rather demands ongoing refinement to ensure the responsible, fair, and safe deployment of GenFMs across various application domains.
URL: https://openreview.net/forum?id=myH1uyrss3
---
Title: Energy-Efficient Deep Learning via Update Sampling from a Generalized Gaussian Distribution: An Empirical Study
Abstract: The computation of loss gradients via backpropagation constitutes a significant portion of the energy consumption in the training of deep learning (DL) models. This paper introduces a simple yet effective method to reduce energy usage during training by leveraging the overparameterization of DL models. Under this assumption, the loss landscape is smooth, and we hypothesize that gradient elements follow a Generalized Gaussian Distribution (GGD). Based on this hypothesis, energy savings are achieved by skipping entire training epochs and estimating gradients by sampling from a GGD. Specifically, parameter updates during skipped epochs are performed by adding GGD-based samples of gradient components to the model parameters from the previous epoch. Furthermore, we present a theorem that provides an upper bound on the expected loss behavior, along with the corresponding convergence rate. We provide extensive empirical validation of our GGD hypothesis across various tasks—image classification, object detection, and image segmentation—using widely adopted DL models. Results show substantial reductions in energy consumption without compromising model performance. Additionally, we evaluate our method on Domain Adaptation (DA), Domain Generalization (DG), and Federated Learning (FL) tasks, observing similar energy savings. To further validate the adaptability of our sampling strategy, we also test it in large language model (LLM) pre-training, demonstrating its effectiveness across diverse settings.
URL: https://openreview.net/forum?id=fbUkDepZb2
---
Title: Causal Ordering for Structure Learning From Time Series
Abstract: Predicting causal structure from time series data is crucial for understanding complex phenomena in physiology, brain connectivity, climate dynamics, and socio-economic behaviour. Causal discovery in time series is hindered by the combinatorial complexity of identifying true causal relationships, especially as the number of variables and time points grow. A common approach to simplify the task is the so-called ordering-based methods. Traditional ordering methods inherently limit the representational capacity of the resulting model. In this work, we fix this issue by leveraging multiple valid causal orderings, instead of a single one as standard practice. We propose DOTS (**D**iffusion **O**rdered **T**emporal **S**tructure), using diffusion-based causal discovery for temporal data. By integrating multiple orderings, DOTS effectively recovers the transitive closure of the underlying directed acyclic graph (DAG), mitigating spurious artifacts inherent in single-ordering approaches. We formalise the problem under standard assumptions such as stationarity and the additive noise model, and leverage score matching with diffusion processes to enable efficient Hessian estimation. Extensive experiments validate the approach. Empirical evaluations on synthetic and real-world datasets demonstrate that DOTS outperforms state-of-the-art baselines, offering a scalable and robust approach to temporal causal discovery. On synthetic benchmarks spanning $d{=}3{-}6$ variables, $T{=}200{-}5{,}000$ samples and up to three lags, DOTS improves mean window‑graph $F1$ from $0.63$ (best baseline) to $0.81$. On the CausalTime real‑world benchmark (*Medical*, *AQI*, *Traffic*; $d{=}20{-}36$), it attains the highest average summary‑graph $F1$ while halving runtime relative to graph‑optimisation methods. These results establish DOTS as a scalable and accurate solution for temporal causal discovery. Code is available at https://anonymous.4open.science/r/dots-TMLR.
URL: https://openreview.net/forum?id=hWuTzqggSd
---
Title: Explaining Bayesian Neural Networks
Abstract: To advance the transparency of learning machines such as Deep Neural Networks (DNNs), the field of explainable AI (XAI) was established to provide interpretations of DNNs' predictions. While different explanation techniques exist, a popular approach is given in the form of attribution maps, which illustrate, given a particular data point, the relevant patterns the model has used for making its prediction. Although Bayesian models such as Bayesian Neural Networks (BNNs) have a limited form of transparency built-in through their prior weight distribution, they lack explanations of their predictions for given instances. In this work, we bring together these two perspectives of transparency into a holistic explanation framework for explaining BNNs. Within the Bayesian framework, the network weights follow a probability distribution. Hence, the standard (deterministic) prediction strategy of DNNs extends in BNNs to a predictive distribution, and thus the standard explanation extends to an explanation distribution. Exploiting this view, we uncover that BNNs implicitly employ multiple heterogeneous prediction strategies. While some of these are inherited from standard DNNs, others are revealed to us by considering the inherent uncertainty in BNNs. Our quantitative and qualitative experiments on toy/benchmark data and real-world data from pathology show that the proposed approach of explaining BNNs can lead to more effective and insightful explanations.
URL: https://openreview.net/forum?id=ZxsR4t3wJd
---
Title: MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. Typically, LLMs are first pre-trained on large corpora and subsequently fine-tuned on task-specific datasets. However, during fine-tuning, LLMs may forget some knowledge acquired in the pre-training stage, leading to a decline in general capabilities. Existing approaches to mitigate forgetting often rely on access to pre-training data, which may be unavailable in many real-world scenarios—such as fine-tuning checkpoint-only open-source LLMs. To address this challenge, we propose a new fine-tuning algorithm termed Momentum-Filtered Optimizer (MoFO).
MoFO is an extension of greedy block coordinate descent (BCD) methods: in each iteration, MoFO only updates the model parameters with the largest momentum magnitudes, while keeping all other parameters fixed. MoFO achieves similar fine-tuning performance to the default fine-tuning algorithm while effectively mitigating knowledge forgetting. We validate MoFO through rigorous convergence analysis and extensive experiments, demonstrating its effectiveness in mitigating forgetting without pre-training data.
URL: https://openreview.net/forum?id=T1qXIDn9my
---
Title: A Survey of Automatic Hallucination Evaluation on Natural Language Generation
Abstract: The proliferation of Large Language Models (LLMs) has introduced a critical challenge: accurate hallucination evaluation that ensures model reliability. While Automatic Hallucination Evaluation (AHE) has emerged as essential, the field suffers from methodological fragmentation, hindering both theoretical understanding and practical advancement. This survey addresses this critical gap through a comprehensive analysis of 74 evaluation methods, revealing that 74% specifically target LLMs, a paradigm shift that demands new evaluation frameworks. We formulate a unified evaluation pipeline encompassing datasets and benchmarks, evidence collection strategies, and comparison mechanisms, systematically documenting the evolution from pre-LLM to post-LLM methodologies. Beyond taxonomical organization, we identify fundamental limitations in current approaches and their implications for real-world deployment. To guide future research, we delineate key challenges and propose strategic directions, including enhanced interpretability mechanisms and integration of application-specific evaluation criteria, ultimately providing a roadmap for developing more robust and practical hallucination evaluation systems.
URL: https://openreview.net/forum?id=qnTzKAsNND
---
Title: Gradients protection in federated learning for Biometric authentication model
Abstract: Privacy-Preserving Machine Learning, Gradient Inversion Attacks, Deep, Gradient Leakage (DGL), Biometric Authentication, Client-Side Obfuscation, Centralized Authentication Model, Multi Biometric Systems, Face and Fingerprint Integration}
In federated learning (FL) environments, biometric authentication systems encounter a distinct challenge: safeguarding user privacy without sacrificing the precision necessary for identity confirmation. Although previous FL privacy research has primarily addressed broad-spectrum protections, this paper concentrates on the particular weaknesses of biometric authentication models, especially those susceptible to gradient inversion and deep gradient leakage (DGL) attacks.
We introduce an innovative privacy-preserving framework specifically designed for federated biometric authentication. Our approach employs a dual strategy: (1) an authentication model that is trained on both original and modified biometric samples to maintain resilience against input perturbations, and (2) a client-side obfuscation technique that alters biometric data prior to gradient computation, efficiently preventing reconstruction attempts. The obfuscation is adaptive and privacy-aware, selectively preserving critical biometric features necessary for authentication while discarding nonessential components to reduce input size and improve accuracy. Simultaneously, this process increases the gradient distance between the original and shared data, enhancing protection against reconstruction. Additionally, block-wise shuffling is employed to disrupt the semantic structure, ensuring that any reconstructed image lacks meaningful visual content.
To validate its practical use, our framework is tested in a multibiometric context using facial and fingerprint information. The blockwise transformation strategy ensures superior authentication efficiency while reducing privacy risks. Experiments conducted in various adversarial FL settings reveal that our method significantly enhances defenses against reconstruction attacks, outperforming traditional measures.
URL: https://openreview.net/forum?id=rwdhVUZPts
---
Title: Enhancing Physics-Informed Neural Networks Through Fea- ture Engineering
Abstract: Physics-Informed Neural Networks (PINNs) seek to solve partial differential equations (PDEs) with deep learning. Mainstream approaches that deploy fully-connected multi-layer deep learning architectures require prolonged training to achieve moderate accuracy, while recent work on feature engineering allows higher accuracy and faster convergence. This paper introduces SAFE-NET, a Single-layered Adaptive Feature Engineering NETwork that improves errors with far fewer parameters than baseline feature engineering methods. SAFE-NET returns to basic ideas in machine learning, using Fourier features,
a simplified single hidden layer network architecture, and an effective optimizer that improves the conditioning of the PINN optimization problem. Numerical results show that SAFE-NET converges faster and typically outperforms deeper networks and more complex architectures. It consistently uses fewer parameters --- on average, 53% fewer than the competing feature engineering methods and 70-100$\boldsymbol{\times}$ fewer than state-of-the-art large-scale architectures --- while achieving comparable accuracy in less than 30% of the training epochs. Moreover, each SAFE-NET epoch is 95% faster than those of competing feature engineering approaches. These findings challenge the prevailing belief that modern PINNs effectively learn relevant features and highlight the efficiency gains possible through feature engineering.
URL: https://openreview.net/forum?id=J25OqR4pBB
---
Title: Decomposable Neuro Symbolic Regression
Abstract: Symbolic regression (SR) models complex systems by discovering mathematical expressions that capture underlying relationships in observed data. However, most SR methods prioritize minimizing prediction error over identifying the governing equations, often producing overly complex or inaccurate expressions. To address this, we present a decomposable SR method that generates interpretable multivariate expressions leveraging transformer models, genetic algorithms (GAs), and genetic programming (GP). In particular, our explainable SR method distills a trained "opaque'' regression model into mathematical expressions that serve as explanations of its computed function. Our method employs a Multi-Set Transformer to generate multiple univariate symbolic skeletons that characterize how each variable influences the opaque model's response. We then evaluate the generated skeletons' performance using a GA-based approach to select a subset of high-quality candidates before incrementally merging them via a GP-based cascade procedure that preserves their original skeleton structure. The final multivariate skeletons undergo coefficient optimization via a GA. We evaluated our method on problems with controlled and varying degrees of noise, demonstrating lower or comparable interpolation and extrapolation errors compared to two GP-based and three neural SR methods. Unlike these methods, our approach consistently learned expressions that matched the original mathematical structure.
URL: https://openreview.net/forum?id=jtUTfCOksJ
---
Title: Flow-Attentional Graph Neural Networks
Abstract: Graph Neural Networks (GNNs) have become essential for learning from graph-structured data. However, existing GNNs do not consider the conservation law inherent in graphs associated with a flow of physical resources, such as electrical current in power grids or traffic in transportation networks, which can lead to reduced model performance. To address this, we propose flow attention, which adapts existing graph attention mechanisms to satisfy Kirchhoff’s first law. Furthermore, we discuss how this modification influences the expressivity and identify sets of non-isomorphic graphs that can be discriminated by flow attention but not by standard attention. Through extensive experiments on two flow graph datasets—electronic circuits and power grids—we demonstrate that flow attention enhances the performance of attention-based GNNs on both graph-level classification and regression tasks.
URL: https://openreview.net/forum?id=tOzg7UxTPD
---
Title: Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Models
Abstract: Under pure textual modality, Large Language Models (LLMs) have demonstrated remarkable success in complex reasoning tasks by decomposing them into simpler sub-problems. However, Multimodal Large Language Models (MLLMs) still struggle with some seemingly straightforward visual tasks, such as counting and solving jigsaw puzzles. We argue that these tasks challenge the ability of {\it visual-to-textual conversion}, where MLLMs convert visual information perceived from the input scene, to textual information for further reasoning and generating the answer. If the complexity of the visual input is beyond the perceptual capability of the MLLMs, without decomposing this conversion process, simply scaling inference-time reasoning cannot solve the task because it repeatedly encounters the same perceptual bottleneck. We propose an approach, {\it autonomous imagination}, to enable MLLMs to iteratively modify visual inputs (e.g. isolating objects, rearranging puzzle pieces) into intermediate visual states, decomposing visual-to-textual conversion into closed-loop visual modification steps. We show that, without any retraining, MLLMs can now solve tasks initially beyond their perceptual capability, highlighting that closed-loop visual modification can be an effective way of decomposing the visual reasoning task into solvable substeps. Our code and data are released at (publicly available upon acceptance).
URL: https://openreview.net/forum?id=MI4yIBLprs
---
Title: MCMC-Correction of Score-Based Diffusion Models for Model Composition
Abstract: Diffusion models can be parameterized in terms of either a score or an energy function. The energy parameterization is attractive as it enables sampling procedures such as Markov Chain Monte Carlo (MCMC) that incorporates a Metropolis–Hastings (MH) correction step based on energy differences between proposed samples. Such corrections can significantly improve sampling quality, particularly in the context of model composition, where pre-trained models are combined to generate samples from novel distributions. Score-based diffusion models, on the other hand, are more widely adopted and come with a rich ecosystem of pre-trained models. However, they do not, in general, define an underlying energy function, making MH-based sampling inapplicable. In this work, we address this limitation by retaining the score parameterization and introducing a novel MH-like acceptance rule based on line integration of the score function. This allows the reuse of existing diffusion models while still combining the reverse process with various MCMC techniques, viewed as an instance of annealed MCMC. Through experiments on synthetic and real-world data, we show that our MH-like samplers offer comparable improvements to those obtained with energy-based models, without requiring explicit energy parameterization.
URL: https://openreview.net/forum?id=iHe7HcjhFk
---
Title: From discrete-time policies to continuous-time diffusion samplers: Asymptotic equivalences and faster training
Abstract: We study the problem of training neural stochastic differential equations, or diffusion models, to sample from a Boltzmann distribution without access to target samples. Existing methods for training such models enforce time-reversal of the generative and noising processes, using either differentiable simulation or off-policy reinforcement learning (RL). We prove equivalences between families of objectives in the limit of infinitesimal discretization steps, linking entropic RL methods (GFlowNets) with continuous-time objects (partial differential equations and path space measures). We further show that an appropriate choice of coarse time discretization during training allows greatly improved sample efficiency and the use of time-local objectives, achieving competitive performance on standard sampling benchmarks with reduced computational cost.
URL: https://openreview.net/forum?id=xLE3xJUuDO
---
Title: Optimizing Federated Learning Client Selection via Multi-Objective Contextual Bandits
Abstract: In the rapidly evolving field of Machine Learning (ML), Federated Learning (FL) emerges as an innovative approach for training models across distributed devices without centralizing raw data. However, FL faces significant challenges due to the heterogeneous nature of client devices, leading to non-IID data distributions and various resource constraints. Moreover, the inherent bandwidth limitations in decentralized settings necessitate the efficient use of both network and energy resources. Energy-efficient clients not only reduce the frequency of battery charging but also minimize data transmissions, thereby resulting in lower overall energy consumption during model training. This reduction in energy usage not only improves network efficiency but also contributes to environmental sustainability. To address these challenges, we introduce a novel solution, Pareto Contextual Zooming for Federated Learning (PCZFL), which treats the client selection problem in FL as a multi-objective bandit problem. Our method focuses on optimizing both global accuracy and energy efficiency in parallel. By dynamically adjusting client selection based on real-time accuracy and energy context, the proposed solution ensures effective participation while minimizing energy consumption. In addition, we provide theoretical analysis on both the regret bound and time complexity of our method. Extensive experiments demonstrate that PCZFL noticeably outperforms current state-of-the-art methods, offering a robust solution that balances the competing demands of accuracy and energy efficiency in FL deployments.
URL: https://openreview.net/forum?id=DYmzaHwrT9
---