Weekly TMLR digest for May 10, 2026
TMLR
New certifications
==================
J2C Certification: Think2SQL: Blueprinting Reward Density and Advantage Scaling for Effective Text-to-SQL Reasoning
Simone Papicchio, Simone Rossi, Luca Cagliero, Paolo Papotti
https://openreview.net/forum?id=NxU1KWnpOG
---
Reproducibility Certification: Revisiting "Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing"
Luis Vitor Zerkowski, Soham Chaudhuri, Finley Helms, Jelle Sombekke, Udit Thakur
https://openreview.net/forum?id=5Q1gr80AXU
---
J2C Certification: Autofocus Retrieval: An Effective Pipeline for Multi-Hop Question Answering With Semi-Structured Knowledge
Derian Boer, Stephen Linus Roth, Stefan Kramer
https://openreview.net/forum?id=U2vqruHfQY
---
J2C Certification: Neural Fourier Transform for Multiple Time Series Prediction
Noam Koren, Kira Radinsky, Daniel Freedman
https://openreview.net/forum?id=0GBjIwRuVp
---
J2C Certification: Paradoxical noise preference in RNNs
Noah Izaac Eckstein, Manoj Srinivasan
https://openreview.net/forum?id=gqxTZRzI35
---
Accepted papers
===============
Title: LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers
Authors: Nikhil Abhyankar, Parshin Shojaee, Chandan K. Reddy
Abstract: Automated feature engineering plays a critical role in improving predictive model performance for tabular learning tasks. Traditional automated feature engineering methods are limited by their reliance on pre-defined transformations within fixed, manually designed search
spaces, often neglecting domain knowledge. Recent advances using Large Language Models (LLMs) have enabled the integration of domain knowledge into the feature engineering process. However, existing LLM-based approaches use direct prompting or rely solely on
validation scores for feature selection, failing to leverage insights from prior feature discovery experiments or establish meaningful reasoning between feature generation and data-driven performance. To address these challenges, we propose LLM-FE, a novel framework that combines evolutionary search with the domain knowledge and reasoning capabilities of LLMs to automatically discover effective features for tabular learning tasks. LLM-FE formulates feature engineering as a program search problem, where LLMs propose new feature transformation programs iteratively, and data-driven feedback guides the search process. Our results demonstrate that LLM-FE consistently outperforms state-of-the-art baselines, showcasing generalizability across diverse models, tasks, and datasets.
URL: https://openreview.net/forum?id=qvI35hkpOO
---
Title: InfoScale: Unleashing Training-free Variable-scaled Image Generation via Effective Utilization of Information
Authors: Guohui Zhang, Jiangtong Tan, Linjiang Huang, Zhonghang Yuan, Mingde Yao, Jie Huang, Feng Zhao
Abstract: Diffusion models (DMs) have become dominant in visual generation but suffer a performance drop when tested on resolutions that differ from the training scale, whether lower or higher.
Current training-free methods for DMs have shown promising results, primarily focusing on higher-resolution generation. However, most methods lack a unified analytical perspective for variable-scale generation, leading to suboptimal results.
In fact, the key challenge in generating variable-scale images lies in the differing amounts of information across resolutions, which requires information conversion procedures to be varied for generating variable-scaled images.
In this paper, we investigate the issues of three critical aspects in DMs for a unified analysis in variable-scaled generation: dilated convolution, attention mechanisms, and initial noise.
Specifically, 1) dilated convolution in DMs for the higher-resolution generation loses high-frequency information.
2) Attention for variable-scaled image generation struggles to adjust the information aggregation adaptively.
3) The spatial distribution of information in the initial noise is misaligned with the variable-scaled image.
To solve the above problems, we propose $\textbf{InfoScale}$, an information-centric framework for variable-scaled image generation by effectively utilizing information from three aspects correspondingly.
For information loss in 1), we introduce a Progressive Frequency Compensation module to compensate for high-frequency information lost by dilated convolution in higher-resolution generation.
For information aggregation inflexibility in 2), we introduce an Adaptive Information Aggregation module to adaptively aggregate information in lower-resolution generation and achieve an effective balance between local and global information in higher-resolution generation.
For information distribution misalignment in 3), we design a Noise Adaptation module to re-distribute information in initial noise for variable-scaled generation.
Our method is plug-and-play, and extensive experiments demonstrate its effectiveness in variable-scaled image generation.
URL: https://openreview.net/forum?id=iB8PzxdpMf
---
Title: Measuring Fine-Grained Relatedness in Multitask Learning via Data Attribution
Authors: Yiwen Tu, Ziqi Liu, Jiaqi W. Ma, Weijing Tang
Abstract: Measuring task relatedness and mitigating negative transfer remain a critical open challenge in Multitask Learning (MTL). This work extends data attribution---which quantifies the influence of individual training data points on model predictions---to MTL setting for measuring task relatedness. We propose the MultiTask Influence Function (MTIF), a method that adapts influence functions to MTL models with hard or soft parameter sharing. Compared to conventional task relatedness measurements, MTIF provides a fine-grained, instance-level relatedness measure beyond the entire-task level. This fine-grained relatedness measure enables a data selection strategy to effectively mitigate negative transfer in MTL. Through extensive experiments, we demonstrate that the proposed MTIF efficiently and accurately approximates the performance of models trained on data subsets. Moreover, the data selection strategy enabled by MTIF consistently improves model performance in MTL. Our work establishes a novel connection between data attribution and MTL, offering an efficient and fine-grained solution for measuring task relatedness and enhancing MTL models.
URL: https://openreview.net/forum?id=zIDGm96xwg
---
Title: A Quotient Homology Theory of Representation in Neural Networks
Authors: Kosio Beshkov
Abstract: Previous research has proven that the set of maps implemented by neural networks with a ReLU activation function is identical to the set of piecewise linear continuous maps. Furthermore, such networks induce a hyperplane arrangement splitting the input domain of the network into convex polyhedra $G_J$ over which a network $\Phi$ operates in an affine manner.
In this work, we leverage these properties to define an equivalence relation $\sim_\Phi$ on top of an input dataset, which defines a quotient space that can be split into two sets related to the local rank of $\Phi_J$ and the intersections $\cap \text{Im}\Phi_{J_i}$. We refer to the latter as the \textit{overlap decomposition} $\mathcal{O}_\Phi$ and prove that if the intersections between each polyhedron and an input manifold are convex, the homology groups of neural representations are isomorphic to quotient homology groups $H_k(\Phi(\mathcal{M})) \simeq H_k(\mathcal{M}/\mathcal{O}_\Phi)$. This lets us intrinsically calculate the Betti numbers of neural representations without the choice of an external metric. We develop methods to numerically compute the overlap decomposition through linear programming and a union-find algorithm.
URL: https://openreview.net/forum?id=RluspxztzS
---
Title: Sparse Mean Estimation in Adversarial Settings via Incremental Learning
Authors: Jianhao Ma, Rui Ray Chen, Yinghui He, Salar Fattahi, Wei Hu
Abstract: In this paper, we study the problem of sparse mean estimation under adversarial corruptions, where the goal is to estimate the $k$-sparse mean of a heavy-tailed distribution from samples contaminated by adversarial noise. Existing methods face two key limitations: they require prior knowledge of the sparsity level $k$ and scale poorly in high-dimensional settings. We propose a simple and scalable estimator that addresses both challenges. Specifically, it learns the $k$-sparse mean without knowing $k$ in advance and operates in near-linear time and memory with respect to the ambient dimension. Under a moderate signal-to-noise ratio, our method achieves the optimal statistical rate, matching the information-theoretic lower bound. Extensive simulations corroborate our theoretical guarantees. At the heart of our approach is an incremental learning phenomenon: we show that a basic subgradient method applied to a nonconvex two-layer formulation with an $\ell_1$-loss can incrementally learn the $k$ nonzero components of the true mean while suppressing the rest. More broadly, our work is the first to reveal the incremental learning phenomenon of the subgradient method in the presence of heavy-tailed distributions and adversarial corruption.
URL: https://openreview.net/forum?id=S3e7ikEZfg
---
Title: Iterative Compositional Data Generation for Robot Control
Authors: Anh-Quan Pham, Marcel Hussing, Shubhankar P. Patankar, Danielle Bassett, Jorge Mendez-Mendez, Eric Eaton
Abstract: Collecting robotic manipulation data is expensive, making it impractical to acquire demonstrations for the combinatorially large space of tasks that arise in multi-object, multi-robot, and multi-environment settings. While recent generative models can synthesize useful data for individual tasks, they do not exploit the compositional structure of robotic domains and struggle to generalize to unseen task combinations. We propose a semantic compositional diffusion transformer that factorizes transitions into robot-, object-, obstacle-, and objective-specific components and learns their interactions through attention. Once trained on a limited subset of tasks, we show that our model can zero-shot generate high-quality transitions from which we can learn control policies for unseen task combinations. Then, we introduce an iterative self-improvement procedure in which synthetic data is validated via offline reinforcement learning and incorporated into subsequent training rounds. Our approach substantially improves zero-shot performance over monolithic and hard-coded compositional baselines, ultimately solving nearly all held-out tasks and demonstrating the emergence of meaningful compositional structure in the learned representations.
URL: https://openreview.net/forum?id=cASorO1kiy
---
Title: UniRec: Unified Multimodal Encoding for LLM-Based Recommendations
Authors: Zijie Lei, Tao Feng, Zhigang Hua, Yan Xie, Guanyu Lin, Shuang Yang, Ge Liu, Jiaxuan You
Abstract: Large language models (LLMs) have recently shown promise for multimodal recommen-
dation, particularly with text and image inputs. Yet real-world recommendation signals
extend far beyond these modalities. To reflect this, we formalize recommendation features
into four modalities: text, images, categorical features, and numerical attributes, and em-
phasize unique challenges this heterogeneity poses for LLMs in understanding multimodal
information. In particular, these challenges arise not only across modalities but also within
them, as attributes (e.g., price, rating, time) may all be numeric yet carry distinct meanings.
Beyond this intra-modality ambiguity, another major challenge is the nested structure of
recommendation signals, where user histories are sequences of items, each carrying multiple
attributes. To address these challenges, we propose UniRec, a unified multimodal encoder for
LLM-based recommendation. UniRec first employs modality-specific encoders to produce
consistent embeddings across heterogeneous signals. It then applies a triplet representa-
tion—comprising attribute name, type, and value—to separate schema from raw inputs
and preserve semantic distinctions. Finally, a hierarchical Q-Former models the nested
structure of user interactions while maintaining their layered organization. On multiple
real-world benchmarks, UniRec outperforms state-of-the-art multimodal and LLM-based
recommenders by up to 15%, while extensive ablation studies further validate the contribu-
tions of each component.
URL: https://openreview.net/forum?id=WXE255GWhQ
---
Title: Topology- and Gradient-Guided Knowledge Distillation for Point Cloud Semantic Segmentation
Authors: Luu Tung Hai, Thinh D. Le, Zhicheng Ding, Qing Tian, Truong-Son Hy
Abstract: Point cloud processing has gained significant attention due to its critical role in applications such as autonomous driving and 3D object recognition. However, deploying high-performance models like Point Transformer V3 in resource-constrained environments remains challenging due to their high computational and memory demands. This work introduces a novel distillation framework that leverages topology-aware representations and gradient-guided knowledge distillation to effectively transfer knowledge from a high-capacity teacher to a lightweight student model. Our approach captures the underlying geometric structures of point clouds while selectively guiding the student model's learning process through gradient-based feature alignment. Experimental results in the Nuscenes, SemanticKITTI, and Waymo datasets demonstrate that the proposed method achieves competitive performance, with an approximately 16$\times$ reduction in model size and up to 1.9$\times$ decrease in inference time compared to its teacher model. Notably, on NuScenes, our method achieves competitive performance among knowledge distillation techniques trained solely on LiDAR data, surpassing prior knowledge distillation baselines in segmentation performance. Our implementation is available publicly at https://github.com/HySonLab/PointDistill
URL: https://openreview.net/forum?id=lP2phGa5af
---
Title: EgoPlan: Towards Effective Embodied Agents via Egocentric Planning
Authors: Zhirui Fang, Ming Yang, Weishuai Zeng, Junpeng Yue, Boyu Li, Jiafei Lyu, Xiu Li, Zongqing Lu
Abstract: We explore leveraging large multi-modal models (LMMs) and Text2image models to build a more general embodied agent. LMMs excel in planning long-horizon tasks over symbolic abstractions but struggle with grounding in the physical world, often failing to accurately identify object positions in images. A bridge is needed to connect LMMs to the physical world. The paper proposes a novel approach, egocentric vision language planning (EgoPlan), to handle long-horizon tasks from an egocentric perspective in varying household scenarios. This pipeline leverages a diffusion model to simulate the fundamental dynamics between states and actions, discusses how to integrate computer vision related techniques like style transfer and optical flow to enhance ability of modeling spatial states and generalization across different environmental dynamics. The LMM serves as a planner, breaking down instructions into sub-goals and selecting actions based on their alignment with these sub-goals, thus enabling more generalized and effective decision-making. By using LMM, we can output text actions, using a series of mechanisms such as reflection to perform high-level task decomposition and low-level action output end-to-end. Experiments show that EgoPlan improves long-horizon task success rates from the egocentric view compared to baselines across household scenarios.
URL: https://openreview.net/forum?id=KPKqTH0LTi
---
Title: OracleTSC: Oracle-Informed Reward Hurdle and Uncertainty Regularization for Traffic Signal Control
Authors: Darryl C. Jacob, Xinyu Liu, Muchao Ye, Xiaoyong Yuan, Pan He
Abstract: Transparent decision-making is essential for traffic signal control (TSC) systems to earn public trust. However, traditional reinforcement learning–based TSC methods function as black boxes, providing little to no insight into their decisions. Although large language models (LLMs) could provide the needed interpretability through natural language reasoning, they face challenges such as limited memory and difficulty in deriving optimal policies from sparse environmental feedback. Existing TSC methods that apply reinforcement fine-tuning to LLMs face notable training instability and deliver only limited improvements over pretrained models. We attribute this instability to the long-horizon nature of TSC: feedback is sparse and delayed, most control actions yield only marginal changes in congestion metrics, and the resulting weak reward signals interact poorly with policy-gradient optimization. We introduce OracleTSC, which addresses these issues through: (1) a reward hurdle mechanism that filters weak learning signals by subtracting a calibrated threshold from environmental feedback, and (2) preventing policy degeneracy by maximizing the probability of the chosen answer, which promotes consistent decision-making across multiple responses. Experiments on the standard LibSignal benchmark demonstrate that our approach enables a compact model (LLaMA3-8B) to achieve substantial improvements in traffic flow, with a $75%$ reduction in travel time and $67%$ decrease in queue lengths over the pretrained baseline while preserving interpretability through natural language explanations. Furthermore, the method exhibits strong cross-intersection generalization: a policy trained on one intersection transfers to a structurally distinct intersection with $17%$ lower travel time and $39%$ lower queue length, all without any additional finetuning for the target topology. These findings show that uncertainty-aware reward shaping could stabilize reinforcement fine-tuning and provide a new perspective for improving its effectiveness in TSC tasks.
URL: https://openreview.net/forum?id=WmJu5MkoQD
---
Title: Think2SQL: Blueprinting Reward Density and Advantage Scaling for Effective Text-to-SQL Reasoning
Authors: Simone Papicchio, Simone Rossi, Luca Cagliero, Paolo Papotti
Abstract: While Large Language Models (LLMs) have advanced the state-of-the-art in Text-to-SQL, robust reasoning in complex, multi-table environments remains a bottleneck for parameter-efficient models. This paper presents a systematic empirical study on injecting reasoning capabilities into Text-to-SQL through the lens of Reinforcement Learning with Verifiable Rewards (RLVR) for the Qwen3 model family. We uncover a critical interplay between reward density, advantage scaling, and model capacity. Our analysis yields four primary insights. First, we propose a novel execution-guided dense reward function that significantly outperforms binary signals and existing state-of-the-art rewards by providing granular feedback at the instance level. Second, we analyze the mechanics of advantage calculation, demonstrating that while large models thrive on sparse signals with aggressive advantage scaling, smaller models require dense rewards and conservative scaling to improve Text-to-SQL performance. Third, we evaluate the impact of cold start, showing that distillation does not always benefit RLVR performance, and supervised fine-tuned models are prone to distributional mimicry. Fourth, we map the Pareto frontier of training efficiency, providing insights for optimizing Text-to-SQL reasoning under computational constraints. Our findings culminate in the Think2SQL family: our 4B-parameter model demonstrates reasoning capabilities competitive with state-of-the-art models such as o3. We release our models, datasets, and code to create a blueprint for RLVR optimization in Text-to-SQL at https://github.com/spapicchio/Think2SQL.
URL: https://openreview.net/forum?id=NxU1KWnpOG
---
Title: Sequential Causal Discovery with Noisy Language Model Priors
Authors: Prakhar Verma, David Arbour, Sunav Choudhary, Harshita Chopra, Arno Solin, Atanu R. Sinha
Abstract: Causal discovery from observational data typically assumes access to complete data and availability of perfect domain experts. In practice, data often arrive in batches, are subject to sampling bias, and expert knowledge is scarce. Language Models (LMs) offer a surrogate for expert knowledge but suffer from hallucinations, inconsistencies, and bias. We present a hybrid framework that bridges these gaps by adaptively integrating sequential batch data with LM-derived noisy, expert knowledge while accounting for both data-induced and LM-induced biases. We propose a representation shift from Directed Acyclic Graph (DAG) to Partial Ancestral Graph (PAG), that accommodates ambiguities within a coherent framework, allowing grounding the global LM knowledge in local observational data. To guide LM interactions, we use a sequential optimization scheme that adaptively queries the most informative edges. Across varied datasets and LMs, we outperform prior work in structural accuracy and extend to parameter estimation, showing robustness to LM noise.
URL: https://openreview.net/forum?id=wFs71JzEO7
---
Title: Active Teacher Selection for Reward Learning
Authors: Rachel Freedman, Justin Svegliato, Kyle Hollins Wray, Stuart Russell
Abstract: Reward learning techniques enable machine learning systems to learn objectives from human
feedback. A core limitation of these systems is their assumption that all feedback comes from
a single human teacher, despite gathering feedback from large and heterogeneous populations.
We propose the Hidden Utility Bandit (HUB) framework to model differences in teacher
rationality, expertise, and costliness, formalizing the problem of learning from multiple
teachers. We develop a variety of solution algorithms and apply them to two real-world
domains: paper recommendation systems and COVID-19 vaccine testing. We find that Active
Teacher Selection (ATS) algorithms outperform baselines by actively selecting when and which
teacher to query. Our key contributions are 1) the HUB framework: a novel mathematical
framework for modeling the teacher selection problem, 2) ATS: an active-learning based
algorithmic approach that demonstrates the utility of modeling teacher heterogeneity, and
3) proof-of-concept application of the HUB framework and ATS approaches to model and
solve multiple real-world problems with complex trade-offs between reward learning and
optimization.
URL: https://openreview.net/forum?id=9OEy68av40
---
Title: A Closer Look at In-Distribution vs. Out-of-Distribution Accuracy for Open-Set Test-time Adaptation
Authors: Zefeng Li, Evan Shelhamer
Abstract: Open-set test-time adaptation (TTA) updates models on new data in the presence of input shifts and unknown output classes. While recent methods have made progress on improving in-distribution (InD) accuracy for known classes, their ability to accurately detect out-of-distribution (OOD) unknown classes remains underexplored. We benchmark robust and open-set TTA methods (SAR, OSTTA, UniEnt, and SoTTA) on the standard corruption benchmarks of CIFAR-10-C at the small scale and ImageNet-C at the large scale. For CIFAR-10-C, we use OOD data from SVHN and CIFAR-100 in their respective corrupted forms of SVHN-C and CIFAR-100-C. For ImageNet-C, we use OOD data from ImageNet-O and Textures in their respective corrupted forms of ImageNet-O-C and Textures-C. ImageNet-O is nearer to ImageNet, as unknown but related object classes (like ``garlic bread'' vs. ``hot dog'' for food, or ``highway'' vs. ``dam'' for infrastructure), while Textures is farther from ImageNet, as non-object patterns (like ``cracked'' mud, ``porous'' sponge, ``veined'' leaves). We evaluate the accuracy and confidence of TTA methods for InD vs. OOD recognition on CIFAR-10-C and ImageNet-C. We verify the accuracy of each method's own OOD detection technique on CIFAR-10-C. We also evaluate on ImageNet-C and report both accuracy and standard OOD detection metrics. We further examine more realistic settings, in which the proportions and rates of OOD data can vary. To explore the trade-off between InD recognition and OOD rejection, we propose a new baseline that replaces softmax/multi-class output with sigmoid/multi-label output. Our analysis shows for the first time that current open-set TTA methods struggle to balance InD and OOD accuracy and that they only imperfectly filter OOD data for their own adaptation updates.
URL: https://openreview.net/forum?id=4MuLx2YDmi
---
Title: What One View Reveals, Another Conceals: 3D-Consistent Visual Reasoning with LLMs
Authors: Dan Kushnir, László Freund
Abstract: Maintaining semantic label consistency across multiple views is a persistent challenge in 3D semantic object detection. Existing zero-shot approaches that combine 2D detections with vision-language features often suffer from bias toward non-descriptive viewpoints and require a fixed label list to operate on. We propose a truly open-vocabulary algorithm that uses large language model (LLM) reasoning to relabel multi-view detections, mitigating errors from poor, ambiguous viewpoints and occlusions. Our method actively samples informative views based on feature diversity and uncertainty, generates new label hypotheses via LLM reasoning, and recomputes confidences to build a spatial-semantic representation of objects. Experiments on controlled single-object and multi-object scenes show double digit improvement, in accuracy and sampling rate over ubiquitous fusion methods using YOLO, CLIP, and other LLM-based baselines. We demonstrate in multiple settings that \textbf{L}LM-guided \textbf{A}ctive \textbf{D}etection and \textbf{R}easoning (LADR) balances detail preservation with reduced ambiguity and low sampling rate. We provide theoretical convergence analysis showing exponential convergence to a stable and correct semantic label.
URL: https://openreview.net/forum?id=w8rMmoynWi
---
Title: Re:Form --- Reducing Human Priors in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny
Authors: Chuanhao Yan, Fengdi Che, Xuhan Huang, Xu Xu, Xin Li, Yizhi LI, Xingwei Qu, Jingzhe Shi, Chenghua Lin, Yaodong Yang, Binhang Yuan, Hang Zhao, Yu Qiao, Bowen Zhou, Jie Fu
Abstract: Existing informal language-based (e.g., human language) Large Language Models (LLMs) trained with Reinforcement Learning (RL) face a significant challenge: their verification processes, which provide crucial training signals, are neither reliable nor scalable. In fact, the prevalent large proprietary models could hardly generate verifiable programs. A promising yet largely uncharted alternative is formal language-based reasoning. Grounding LLMs in rigorous formal systems where generative models operate in formal language spaces (e.g., Dafny) enables the automatic and mathematically provable verification of their reasoning processes and outcomes. This capability is pivotal for achieving large-scale, reliable formal software verification. It is a common practice to employ human-annotated chain-of-thought and other human priors to induce the reasoning and coding capabilities of LLMs. Unfortunately, it becomes unacceptably all-consuming to provide such priors for supervising complex programming tasks. In this work, we systematically explore ways to reduce human priors with the formal language, Dafny, as the main environment for our pilot study. Our pipeline mainly relies on introducing an automatic and scalable data curation pipeline, and careful RL designs integrated with feedback from the formal language verifier. We introduce DafnyComp, a benchmark of compositional formal programs with auto-formalized specifications for specification reasoning. Our supervised fine-tuning (SFT) stage enables even small models (e.g., 0.5B) to generate syntactically valid and verifiable Dafny code, surpassing proprietary models. RL with regularization further improves performance, achieving stronger generalization to out-of-domain tasks and outperforming all strong baselines on the challenging DafnyComp benchmark. Anonymized code and models are available at https://github.com/ReFormDafny/ReForm and https://huggingface.co/ReFormDafny.
URL: https://openreview.net/forum?id=cAQmIS4GOe
---
Title: Compromising Honesty and Harmlessness in Language Models via Covert Deception Attacks
Authors: Laurène Vaugrante, Francesca Carlon, Maluna Menke, Thilo Hagendorff
Abstract: Recent research on large language models (LLMs) has demonstrated their ability to understand and employ deceptive behavior, even without explicit prompting. Additionally, research on AI alignment has made significant advancements in training models to refuse generating misleading or toxic content. As a result, LLMs generally became honest and harmless. In this study, we introduce “deception attacks” that undermine both of these traits while keeping models seemingly trustworthy, revealing a vulnerability that, if exploited, could have serious real-world consequences. We introduce fine-tuning methods that cause models to selectively deceive users on targeted topics while remaining accurate on others, to maintain a high user trust. Through a series of experiments, we show that such targeted deception is effective even in high-stakes domains or ideologically charged subjects. In addition, we find that deceptive fine-tuning often compromises other safety properties: deceptive models are more likely to produce toxic content, including hate speech and stereotypes. Finally, since self-consistent deception across turns gives users few cues to detect manipulation and thus can preserve trust, we test for multi-turn deception and observe mixed results. Given that millions of users interact with LLM-based chatbots, voice assistants, agents, and other interfaces where trustworthiness cannot be ensured, securing these models against covert deception attacks is critical.
URL: https://openreview.net/forum?id=2KPIDIeLE2
---
Title: Variance Matters: Improving Domain Adaptation via Stratified Sampling
Authors: Andrea Napoli, Paul White
Abstract: Domain shift remains a key challenge in deploying machine learning models to the real world. Unsupervised domain adaptation (UDA) aims to address this by minimising domain discrepancy during training, but the discrepancy estimates suffer from high variance in stochastic settings, which can stifle the theoretical benefits of the method. This paper proposes Variance-Reduced Domain Adaptation via Stratified Sampling (VaRDASS), the first specialised stochastic variance reduction technique for UDA. We consider two specific discrepancy measures -- correlation alignment and the maximum mean discrepancy (MMD) -- and derive ad hoc stratification objectives for these terms. We then present expected and worst-case error bounds, and prove that our proposed objective for the MMD is theoretically optimal (i.e., minimises the variance) under certain assumptions. Finally, a practical k-means style optimisation algorithm is introduced and analysed. Experiments on four domain shift datasets demonstrate improved discrepancy estimation accuracy and target domain performance.
URL: https://openreview.net/forum?id=MVwgedTIUs
---
Title: Symbolic Quantile Regression for the Interpretable Prediction of Conditional Quantiles
Authors: Cas Oude Hoekstra, Floris den Hengst
Abstract: Symbolic Regression (SR) is a well-established framework for generating interpretable or white-box predictive models.
Although SR has been successfully applied to create interpretable estimates of the average of the outcome, it is currently not well understood how it can be used to estimate the relationship between variables at other points in the distribution of the target variable. Such estimates of e.g. the median or an extreme value provide a fuller picture of how predictive variables affect the outcome and are necessary in high-stakes, safety-critical application domains. This study introduces Symbolic Quantile Regression (SQR), an approach to predict conditional quantiles with SR. In an extensive evaluation, we find that SQR outperforms transparent models and performs comparably to a strong black-box baseline without compromising transparency. We also show how SQR can be used to explain differences in the target distribution by comparing models that predict extreme and central outcomes in an airline fuel usage case study. We conclude that SQR is suitable for predicting conditional quantiles and understanding interesting feature influences at varying quantiles.
URL: https://openreview.net/forum?id=x9OYbyPJOG
---
Title: Scaling Large Language Models with Fully Sparse Activations
Authors: Hongyu Wang, Shuming Ma, Ruiping Wang, Furu Wei
Abstract: Activation sparsity can reduce the inference cost of large language models (LLMs) by lowering both compute and memory traffic. Yet most existing approaches sparsify only FFN intermediate states, leaving substantial portions of inference effectively dense. We study how to scale fully sparsely activated LLMs, in which every activation participating in linear transformations is sparse. We focus on two questions: how to train such models effectively, and how activation sparsity affects model quality as scale increases. We develop a pre-training recipe that enables effective training fully sparsely activated LLMs from scratch, including using squared ReLU as activation function, top-K sparsification and a straight-through estimator for the remaining linear layers. Extensive experiments spanning model sizes, training-token budgets, and target sparsity levels reveal that its performance gap to dense baselines narrows with model scale, increases nonlinearly with sparsity, while remaining largely insensitive to the training-token budget. Finally, we investigate post-training activation sparsification of pre-trained dense models via both training-free techniques and supervised fine-tuning, and observe a similar trend as pre-training experiments: larger models are more robust to sparsification, and exhibit increasingly sparse activation patterns. Overall, our results provide practical training recipes and empirical guidance for building and scaling LLMs with fully sparse activations.
URL: https://openreview.net/forum?id=MntjMCroiE
---
Title: Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approach
Authors: Haoxuan Chen, Yinuo Ren, Martin Renqiang Min, Lexing Ying, Zachary Izzo
Abstract: Diffusion models (DMs) have proven to be effective in modeling high-dimensional distributions, leading to their widespread adoption for representing complex priors in Bayesian inverse problems (BIPs). However, current DM-based posterior sampling methods proposed for solving common BIPs rely on heuristic approximations to the generative process. To exploit the generative capability of DMs and avoid the usage of such approximations, we propose an ensemble-based algorithm that performs posterior sampling without the use of heuristic approximations. Our algorithm is motivated by existing work that combines DM-based methods with the sequential Monte Carlo (SMC) method. By examining how the prior evolves through the diffusion process encoded by the pre-trained score function, we derive a modified partial differential equation (PDE) governing the evolution of the corresponding posterior distribution. This PDE includes a modified diffusion term and a reweighting term, which can be simulated via stochastic weighted particle methods. Theoretically, we prove that the error between the true posterior and the empirical distribution of the generated samples can be bounded in terms of the training error of the pre-trained score function and the number of particles in the ensemble. Empirically, we validate our algorithm on several inverse problems in imaging to show that our method gives more accurate reconstructions compared to existing DM-based methods. Our code is available at the following Github repository~\url{https://github.com/HaoxuanSteveC00/AFDPS-TMLR}.
URL: https://openreview.net/forum?id=qN8ASsfjKs
---
Title: The Pitfalls of Text Degeneration when Aligning LLMs through Model Merge
Authors: Peijun Qing, Lei Hsiung, Hefan Zhang, Haiquan Lu, Xingjian Diao, Chiyu Ma, Saeed Hassanpour, Soroush Vosoughi
Abstract: Model merge offers a cost-efficient method for integrating multiple specialized large language models (LLMs) into one comprehensive model. While it shows promise for encoder-decoder models in standard Natural Language Processing (NLP) tasks, \textbf{we find that merging decoder-based LLMs may lead to localized text degeneration, even when overall performance appears to improve.} We specifically assess the applications of model merge in steering LLMs to align better with diverse human preferences through interpolation and extrapolation merge. Our extensive experiments, covering model sizes ranging from $\mathtt{7b}$ to $\mathtt{70b}$ parameters, and including sixteen models with varying post-training, employ three popular merging methods: $\mathtt{Task~Arithmetic}$, $\mathtt{TIES}$-$\mathtt{Merging}$, and $\mathtt{Dare}$-$\mathtt{TIES}$. Our results uncover inherent limitations in current model merge applications for alignment, which can lead to text degeneration. We hope our findings will offer valuable insights for employing model merging in alignment scenarios and can help practitioners avoid potential pitfalls.
URL: https://openreview.net/forum?id=zJAy9Tt9DX
---
Title: ADiff4TPP: Asynchronous Diffusion Models for Temporal Point Processes
Authors: Amartya Mukherjee, Ruizhi Deng, He Zhao, Yuzhen Mao, Leonid Sigal, Frederick Tung
Abstract: This work introduces a diffusion model-based approach to modelling temporal point processes via an asynchronous noise schedule.
Existing methods typically rely on parametric conditional intensity functions or autoregressive next-event prediction, which can limit distributional expressivity and make long-horizon forecasting computationally expensive. We address this limitation by using diffusion models to learn the joint distribution of event sequences in latent space without imposing restrictive parametric assumptions.
At each step of the diffusion process, the noise schedule injects noise of varying scales into different parts of the data.
With a careful design of the noise schedules, earlier events are generated faster than later ones, thus providing stronger conditioning for forecasting the more distant future.
We derive an objective to effectively train these models for a general family of noise schedules based on conditional flow matching.
Our method models the joint distribution of the latent representations of events in a sequence and achieves state-of-the-art results in predicting both the next inter-event time and event type on benchmark datasets.
Additionally, it flexibly accommodates varying lengths of observation and prediction windows in different forecasting settings by adjusting the starting and ending points of the generation process.
Finally, our method shows strong performance in long horizon prediction tasks, outperforming existing baseline methods.
URL: https://openreview.net/forum?id=bwnZW4wXh4
---
Title: Hessian-aware Training for Enhancing DNN Resilience to Bitwise Corruptions in Their Parameters
Authors: Tahmid Hasan Pranto, Seijoon Kim, Lizhong Chen, Sanghyun Hong
Abstract: Deep neural networks are not resilient to parameter corruptions: even a single-bitwise error in their parameters in memory can cause an accuracy drop of over 10%, and in the worst-cases, up to 99%. This susceptibility poses great challenges in deploying models on computing platforms, where adversaries can induce random/targeted bit-flips, e.g., through software-induced fault attacks like Rowhammer. Most prior work addresses this issue with hardware or system-level approaches, such as integrating additional hardware components to verify a model’s integrity at inference. However, these methods have not been widely deployed as they require infrastructure or platform-wide modifications.
In this paper, we propose a new approach to addressing this issue: training models to be more resilient to bitwise corruptions to their parameters. Our approach, Hessian-aware training, promotes models to learn flatter loss surfaces. We show that existing training methods designed to improve generalization (e.g., through sharpness-aware minimization) do not enhance resilience to parameter corruptions. In contrast, models trained with our method demonstrate improved resilience to parameter corruptions, particularly with a 20–50% reduction in the number of bits whose individual flipping leads to a 90–100% accuracy drop. We also characterize the factors that may influence this increased resilience. Moreover, we show the synergy between ours and existing hardware and system-level defenses.
URL: https://openreview.net/forum?id=XxlQF4muso
---
Title: Super-Linear: A Lightweight Pretrained Mixture of Linear Experts for Time Series Forecasting
Authors: Liran Nochumsohn, Raz Marshanski, Hedi Zisling, Omri Azencot
Abstract: Time series forecasting (TSF) is critical in domains like energy, finance, healthcare, and logistics, requiring models that generalize across diverse datasets. Large pre-trained models such as Chronos and Time-MoE show strong zero-shot (ZS) performance but suffer from high computational costs. In this work, we introduce Super-Linear, a lightweight and scalable mixture-of-experts (MoE) model for general forecasting. It replaces deep architectures with simple frequency-specialized linear experts. A lightweight spectral gating mechanism dynamically selects relevant experts, enabling efficient, accurate forecasting. Crucially, resampling during training exposes the model to diverse frequency regimes, while a flexible input adaptation strategy allows it to handle varying inference lengths. Despite its simplicity, Super-Linear demonstrates strong performance across benchmarks, while substantially improving efficiency, robustness to sampling rates, and interpretability.
URL: https://openreview.net/forum?id=av8niDGYMk
---
Title: Modality-Inconsistent Continual Learning of Multimodal Large Language Models
Authors: Weiguo Pian, Shijian Deng, Shentong Mo, Mingrui Liu, Yunhui Guo, Yapeng Tian
Abstract: In this paper, we introduce Modality-Inconsistent Continual Learning (MICL), a new continual learning scenario for Multimodal Large Language Models (MLLMs) that involves tasks with inconsistent modalities (image, audio, or video) and varying task types (captioning or question-answering). Unlike existing vision-only or modality-incremental settings, MICL combines modality and task type shifts, both of which drive catastrophic forgetting. To address these challenges, we propose MoInCL, which employs a Pseudo Targets Generation Module to mitigate forgetting caused by task type shifts in previously seen modalities. It also incorporates Instruction-based Knowledge Distillation to preserve the model's ability to handle previously learned modalities when new ones are introduced. We benchmark MICL using a total of six tasks and conduct experiments to validate the effectiveness of our MoInCL. The experimental results highlight the superiority of MoInCL, showing significant improvements over representative and state-of-the-art continual learning baselines.
URL: https://openreview.net/forum?id=FD8or43fBU
---
Title: SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Improved Vision-Language Model Adaptation
Authors: Shuvendu Roy, Ali Etemad
Abstract: We present SelfPrompt, a novel prompt-tuning approach for vision-language models (VLMs) in a semi-supervised learning setup. Existing methods for tuning VLMs in semi-supervised setups struggle with the negative impact of the miscalibrated VLMs on pseudo-labelling, and the accumulation of noisy pseudo-labels. SelfPrompt addresses these challenges by introducing a cluster-guided pseudo-labelling method that improves pseudo-label accuracy, and a confidence-aware semi-supervised learning module that maximizes the utilization of unlabelled data by combining supervised learning and weakly-supervised learning. Additionally, we investigate our method in an active semi-supervised learning setup, where the labelled set is strategically selected to ensure the best utilization of a limited labelling budget. To this end, we propose a weakly-supervised sampling technique that selects a diverse and representative labelled set, which can be seamlessly integrated into existing methods to enhance their performance. We conduct extensive evaluations across 13 datasets, significantly surpassing state-of-the-art performances with average improvements of 6.23% in standard semi-supervised learning, 6.25% in active semi-supervised learning, and 4.9% in base-to-novel generalization, using a 2-shot setup. Furthermore, SelfPrompt shows excellent generalization in single-shot settings, achieving an average improvement of 11.78%.
URL: https://openreview.net/forum?id=cP6USDUjK8
---
Title: Learning Robust Penetration Testing Policies under Partial Observability: A systematic evaluation
Authors: Raphael Simon, Pieter Jules Karel Libin, Wim Mees
Abstract: Penetration testing, the simulation of cyberattacks to identify security vulnerabilities, presents a sequential decision-making problem well-suited for reinforcement learning (RL) automation. Like many applications of RL to real-world problems, partial observability presents a major challenge, as it invalidates the Markov property present in Markov Decision Processes (MDPs). Partially Observable MDPs require history aggregation or belief state estimation to learn successful policies. We investigate stochastic, partially observable penetration testing scenarios over host networks of varying size, aiming to better reflect real-world complexity through more challenging and representative benchmarks. This approach leads to the development of more robust and transferable policies, which are crucial for ensuring reliable performance across diverse and unpredictable real-world environments. Using vanilla Proximal Policy Optimization (PPO) as a baseline, we compare a selection of PPO-based variants designed to mitigate partial observability, including frame-stacking, augmenting observations with historical information, and employing LSTM or TrXL architectures. We conduct a systematic empirical analysis of these algorithms across different host network sizes. We find that this task greatly benefits from history aggregation. Converging up to four times faster than other approaches. Manual inspection of the learned policies by the algorithms reveals clear distinctions and provides insights that go beyond quantitative results.
URL: https://openreview.net/forum?id=YkUV7wfk19
---
Title: GENIE: A Visual-Only Diffusion Framework for Task- Agnostic Image Transformation
Authors: Uddeshya Singh, Aniket Thomas, Aishwarya Agarwal, Srikrishna Karanam, Biplab Banerjee
Abstract: Designing a unified vision model capable of handling diverse visual transformation tasks without task-specific modifications remains a significant challenge, particularly in scaling and generalizing beyond narrowly defined objectives. We propose GENIE, a novel ControlNet-Diffusion framework that performs task-based image generation solely through visual exemplars, eliminating dependence on textual prompts or auxiliary metadata. Unlike conventional prompt-driven diffusion models, GENIE employs a dual visual conditioning mechanism—combining implicit guidance via ControlNet and explicit task encoding through CLIP-based visual arithmetic—to infer task intent directly from reference input-output pairs. To improve semantic alignment between visual exemplars and generated outputs, we introduce a lightweight task consistency loss, which encourages representational coherence in the embedding space across transformed pairs. While not a multitask learner in the classical sense, GENIE enables task switching across multiple tasks without any task-specific modifications in architecture or task-specific loss functions. Evaluations across seven vision tasks—inpainting, colorization, edge detection, deblurring, denoising, semantic segmentation, and depth estimation—and two out-of-distribution (OOD) tasks—deraining and contrast enhancement—demonstrate that GENIE achieves an average performance gain of 10% over visual-conditioned baselines, showcasing its effectiveness for scalable and text-free visual generation.
URL: https://openreview.net/forum?id=vtth9hOwoP
---
Title: Discontinuity-Preserving Image Super-Resolution using MRF-Regularized One-Step Diffusion
Authors: Sanchar Palit, Subhasis Chaudhuri, Biplab Banerjee
Abstract: We propose a real-world image super-resolution framework that leverages a pretrained text-to-image Stable Diffusion model optimized for single-step sampling. Unlike traditional multi-step diffusion-based methods, which are computationally intensive, our approach enables fast inference while preserving high perceptual quality. To this end, we integrate a lightweight image enhancement module trained jointly with the diffusion model under a Maximum A Posteriori (MAP) formulation. The optimization includes a compound Markov Random Field (MRF) prior, derived from the anticipated discontinuity line field energy, which functions as a structural regularizer to preserve fine image details and facilitate deblurring. Existing single-step diffusion approaches often rely on distillation or noise map estimation, which limits their ability to generate rich pixel-space details. In contrast, our method explicitly models high-frequency line field consistency between the low- and high-resolution domains, guiding the image enhancer to reconstruct sharp outputs. By preserving and enhancing structural features such as edges and textures, our framework effectively handles complex degradations commonly encountered in real-world scenarios. Experimental results demonstrate that our method achieves performance that is comparable to or exceeds that of state-of-the-art single-step and multi-step diffusion-based image super-resolution methods qualitatively, quantitatively, and computationally.
URL: https://openreview.net/forum?id=CLrWXyyL5c
---
Title: Revisiting "Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing"
Authors: Luis Vitor Zerkowski, Soham Chaudhuri, Finley Helms, Jelle Sombekke, Udit Thakur
Abstract: Recent advances in diffusion-based image editing have enabled highly realistic and accessible manipulation of facial images, raising serious concerns about biometric privacy and malicious misuse. FaceLock, introduced in Edit Away and My Face Will Not Stay: Personal Biometric Defense against Malicious Generative Editing, proposes an optimization-based defense that embeds subtle perturbations into images at publication time to induce identity distortion in downstream generative edits. The method claims prompt-agnostic effectiveness and strong performance across multiple editing scenarios, supported by open-source code. In this paper, we present a systematic reproducibility study of FaceLock that evaluates its technical, quantitative, and qualitative reproducibility. We assess whether the reported results can be obtained using the released codebase, analyze the correspondence between the paper’s algorithmic description and its implementation, and document ambiguities that impact reproducibility. We further examine quantitative reproducibility by attempting to recover the reported performance trends and relative ranking against baselines. We, however, were not able to reproduce the originally reported performance trends, and our outputs were generally worse than those presented in the original paper. Beyond that, we expand the qualitative analysis to a broader set of image–prompt pairs and an additional, harder facial dataset to better test generalization behavior. While we obtained some successful outputs, only a small fraction of our qualitative results matched the consistently high quality reported by the authors. Finally, we introduce an extension to the FaceLock method that helps with robustness, and we critically examine the evaluation criteria used to measure defense effectiveness, highlighting limitations of prompt fidelity as a primary metric and arguing for a more explicit consideration of the trade-off between identity protection and preservation of the original image. We provide a link to our GitHub repository $\footnote{https://github.com/Luizerko/revisiting_facelock}$.
URL: https://openreview.net/forum?id=5Q1gr80AXU
---
Title: A Second Order Majorant Algorithm for Nonnegative Matrix Factorization
Authors: Mai Quyen PHAM, Jeremy Cohen, Thierry CHONAVEL
Abstract: Nonnegative Matrix Factorization (NMF) is a fundamental tool in unsupervised learning, widely used for tasks such as dimensionality reduction, feature extraction, representation learning, and topic modeling. Many algorithms have been developed for NMF, including the well-known Multiplicative Updates (MU) algorithm, which belongs to a broader class of majorization-minimization techniques.
In this work, we introduce a general second-order optimization framework for NMF under both quadratic and $\beta$-divergence loss functions.
This approach, called Second-Order Majorant (SOM), constructs a local quadratic majorization of the loss function by majorizing its elementwise nonnegative Hessian matrix.
It includes MU as a special case, while enabling faster variants. In particular, we propose mSOM, a new algorithm within this class that leverages a tighter local approximation to accelerate convergence. We provide a convergence analysis, showing linear convergence for individual factor updates and global convergence to a stationary point for the alternating version, AmSOM. Numerical experiments on both synthetic and real datasets demonstrate that AmSOM is a promising algorithm for NMF.
URL: https://openreview.net/forum?id=lm16IQmimK
---
Title: Autofocus Retrieval: An Effective Pipeline for Multi-Hop Question Answering With Semi-Structured Knowledge
Authors: Derian Boer, Stephen Linus Roth, Stefan Kramer
Abstract: In many real-world settings, machine learning models and interactive systems have access to both structured knowledge, e.g., knowledge graphs or tables, and unstructured content, e.g., natural language documents. Yet, most rely on either. Semi-Structured Knowledge Bases (SKBs) bridge this gap by linking unstructured content to nodes within structured data.
In this work, we present Autofocus-Retriever (AF-Retriever), a modular framework for SKB-based, multi-hop question answering. It combines structural and textual retrieval through novel integration steps and optimizations, achieving the best zero- and one-shot results across all three STaRK QA benchmarks, which span diverse domains and evaluation metrics. AF-Retriever’s average first-hit rate surpasses the second-best method by 32.1%.
Its performance is driven by (1) leveraging exchangeable large language models (LLMs) to extract entity attributes and relational constraints for both parsing and reranking the top-$k$ answers, (2) vector similarity search for ranking both extracted entities and final answers, (3) a novel incremental scope expansion procedure that prepares for the reranking on a configurable amount of suitable candidates that fulfill the given constraints the most, and (4) a hybrid retrieval strategy that reduces error susceptibility.
In summary, while constantly adjusting the focus like an optical autofocus, AF-Retriever delivers a configurable amount of answer candidates in four constraint-driven retrieval steps, which are then supplemented and ranked through four additional processing steps.
An ablation study and a detailed error analysis, including a comparison of three different LLM reranking strategies, provide component-level insights that are valuable for advancing the model and for enabling researchers and users to adapt, optimize, or extend its parts. The source code is publicly available at https://github.com/kramerlab/AF-Retriever.
URL: https://openreview.net/forum?id=U2vqruHfQY
---
Title: Neural Fourier Transform for Multiple Time Series Prediction
Authors: Noam Koren, Kira Radinsky, Daniel Freedman
Abstract: Multivariate time series forecasting is an important task in various fields such as economic planning, healthcare management, and environmental monitoring.
In this work, we present a novel methodology for improving multivariate forecasting, particularly, in data sets with strong seasonality.
We frame the forecasting task as a Multi-Dimensional Fourier Transform (MFT) problem and propose the Neural Fourier Transform (NFT) that leverages a deep learning model to predict future time series values by learning the MFT coefficients.
The efficacy of NFT is empirically validated on 7 diverse datasets, demonstrating improvements over multiple forecasting horizons and lookbacks, thereby establishing new state-of-the-art results.
Our contributions advance the field of multivariate time series forecasting by providing a model that excels in predictive accuracy.
The code of this study is publicly available.
URL: https://openreview.net/forum?id=0GBjIwRuVp
---
Title: Provably Efficient Reward Transfer in Reinforcement Learning with Discrete Markov Decision Processes
Authors: Kevin Jatin Vora, Yu Zhang
Abstract: In this paper, we propose a new solution to reward adaptation (RA) in reinforcement learning, where the agent adapts to a target reward function based on one or more existing source behaviors learned a priori under the same domain dynamics but different reward functions. While learning the target behavior from scratch is possible, it is often inefficient given the available source behaviors. Our work introduces a new approach to RA through the manipulation of Q-functions. Assuming the target reward function is a known function of the source reward functions, we compute bounds on the Q-function and present an iterative process (akin to value iteration) to tighten these bounds. The iteration process is based on a lite-model, which is assumed to be given or can be learned. The computed bounds enable action pruning in the target domain before learning even starts. We refer to this method as "$Q-Manipulation$" (Q-M). We formally prove that Q-M, under discrete domains and an accurate lite-model, does not affect the optimality of the returned policy and show that it is provably efficient in terms of sample complexity. Q-M is evaluated in a variety of synthetic and simulation domains to demonstrate its effectiveness, generalizability, and practicality.
URL: https://openreview.net/forum?id=u2b31c9Noe
---
Title: Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay
Authors: Leyan Pan, Xinyuan Cao
Abstract: Neural Collapse (NC) is a geometric structure recently observed at the terminal phase of training deep neural networks, which states that last-layer feature vectors for the same class would `collapse' to a single point, while features of different classes become equally separated. We demonstrate that batch normalization (BN) and weight decay (WD) critically influence the emergence of NC. In the near-optimal loss regime, we establish an asymptotic lower bound on the emergence of NC that depends only on the WD value, training loss, and the presence of last-layer BN. Our experiments substantiate theoretical insights by showing that models demonstrate a stronger presence of NC with BN, appropriate WD values and lower loss. Our findings offer a novel perspective in studying the role of BN and WD in shaping neural network features.
URL: https://openreview.net/forum?id=eKqgCPDBFg
---
Title: Symbolic Recovery of PDEs from Measurement Data
Authors: Erion Morina, Philipp Scholl, Martin Holler
Abstract: Models based on partial differential equations (PDEs) are powerful for describing a wide range of complex phenomena in the natural sciences. Accurately identifying the PDE model, which represents the underlying physical law, is essential for a proper understanding of the problem. This reconstruction typically relies on indirect and noisy measurements of the system’s state and, without specifically tailored methods, rarely yields symbolic expressions, thereby limiting interpretability.
In this work, we address this limitation by considering neural network architectures based on rational functions for the symbolic representation of physical laws. These networks combine the approximation power of rational functions with the flexibility to represent arithmetic operations, and generalize ParFam and EQL-type architectures used in symbolic regression for physical law learning. We further establish regularity results for these symbolic networks.
Our main contribution is a reconstruction result showing that, if there exists an admissible physical law that is expressible within the symbolic network architecture, then in the limit of noiseless and complete measurements, symbolic networks recover a physical law within the PDE model that is representable by the architecture. Moreover, the recovered law corresponds to a regularization-minimizing parameterization, promoting interpretability and sparsity in case of $L^1$-regularization. Under an additional identifiability condition, the unique true physical law is recovered.
These reconstruction and regularity results are derived at the continuous level prior to discretization due to a formulation in function space. Empirical results using the ParFam architecture are consistent with the theoretical findings and suggest the feasibility of reconstructing interpretable physical laws in practice.
URL: https://openreview.net/forum?id=TbHfgo10W3
---
Title: Paradoxical noise preference in RNNs
Authors: Noah Izaac Eckstein, Manoj Srinivasan
Abstract: In recurrent neural networks (RNNs) used to model biological neural networks, noise is typically introduced during training to emulate biological variability and regularize learning. The expectation is that removing the noise at test time should preserve or improve performance. Contrary to this intuition, we find that continuous-time recurrent neural networks (CTRNNs) often perform best at a nonzero noise level, often approximately the same level used during training. This noise preference typically arises when noise is injected inside the neural activation function; networks trained with noise injected outside the activation function perform best with zero noise. The phenomenon arises robustly in diverse tasks for large enough training noise including function approximation, maze navigation, 2D path integration, and a multi-task suite from cognitive neuroscience; we also show the phenomenon arising in feedforward neural networks, not just in RNNs. Through analyses of simple function-approximation and single-neuron regulator tasks, we show that the phenomenon stems from noise-induced shifts of fixed points (stationary distributions) in the underlying stochastic dynamics of the RNNs, thereby providing some mechanistic interpretability of the phenomenon. These fixed point shifts are noise-level dependent and bias the network outputs when the noise is removed, degrading performance. Analytical and numerical results show that the bias arises when neural states operate near activation-function nonlinearities, where noise is asymmetrically attenuated, and that performance optimization incentivizes operation near these nonlinearities; such performance incentives exist for networks with noise inside the activation function, but not for networks with noise outside the activation function, explaining why only noise-in networks show preference. Thus, networks can overfit to the stochastic training environment itself rather than just to the input–output data. The phenomenon is distinct from stochastic resonance, wherein nonzero noise enhances signal processing. Our findings reveal that training noise can become an integral part of the computation learned by recurrent networks, with implications for understanding neural population dynamics and for the design of robust artificial RNNs.
URL: https://openreview.net/forum?id=gqxTZRzI35
---
Title: Seeing is Simulating: Differentiable Physics for Interaction-Aware Material Estimation
Authors: Chun Feng, Hao Zhang, Haolan Xu, Narendra Ahuja
Abstract: Modeling human-object interactions is crucial for creating immersive virtual experiences. However, synthesizing 3D object dynamics conditioned on actions remains a challenging problem. Existing approaches equip static 3D objects with motion priors distilled from video diffusion models. This methodology has two drawbacks: (i) video diffusion models are not physically grounded. Thus, the generated videos may contain physical inaccuracies; (ii) video diffusion models cannot generate complex dynamics where multiple objects interact under actions with long durations and large spatial extent. We present $\textbf{PhysInteract}$, a physics-based framework that (i) models interactions with a representation that captures their duration and contact information; (ii) estimates object material properties (e.g., Young's modulus) from objects' deformation caused by interactions; (iii) uses physics simulation to reproduce realistic object dynamics based on estimated interactions and material properties. We highlight that PhysInteract is fully differentiable, enabling joint optimization of interaction representations and object material properties. PhysInteract achieves better performance than existing methods. We demonstrate its superiority by quantitatively testing PhysInteract on a curated dataset. In conjunction with an additional user study, our method shows a step towards more realistic and immersive virtual experiences.
URL: https://openreview.net/forum?id=lwuaTI4ISa
---
Title: Learning Where It Matters: Responsible and Interpretable Text-to-Image Generation with Background Consistency
Authors: Sayedmoslem Shokrolahi, Jae-Mo Kang, Il-Min Kim
Abstract: Text-to-image diffusion models have achieved remarkable progress, yet they still struggle to produce unbiased and responsible outputs. A promising direction is to manipulate the bottleneck space of the U-Net (the $h$-space), which provides \textit{interpretability} and \textit{controllability}. However, existing methods rely on learning attributes from the entire image, entangling them with spurious features and offering no corrective mechanisms at inference. This uniform reliance leads to poor subject alignment, fairness issues, reduced photorealism, and incoherent backgrounds in scene-specific prompts. To address these challenges, we propose two complementary innovations for training and inference. First, we introduce a spatially focused concept learning framework that disentangles target attributes into concept vectors by suppressing target attribute features within the multi-head cross-attention (MCA) modules and attenuating the encoder output (i.e., $h$-vector) to ensure the concept vector exclusively captures target attribute features. In addition, we introduce a spatially weighted reconstruction loss to emphasize regions relevant to the target attribute. Second, we design an inference-time strategy that improves background consistency by enhancing low-frequency components in the $h$-space. Experiments demonstrate that our approach improves fairness, subject fidelity, and background coherence while preserving visual quality and prompt alignment, outperforming state-of-the-art $h$-space methods. The code is provided at https://github.com/Moslem-Sh21/learning-where-it-matters.
URL: https://openreview.net/forum?id=sCOJGbJwAJ
---
Title: Harnessing Heterogeneity: Improving Convergence Through Partial Variance Control in Federated Learning
Authors: Pranab Sahoo, Ashutosh Tripathi, Sriparna Saha, Samrat Mondal
Abstract: Federated Learning (FL) has emerged as a promising paradigm for collaborative model training without sharing local data. However, a significant challenge in FL arises from the heterogeneous data distributions across participating clients. This heterogeneity leads to highly variable gradient norms in the model's final layers, resulting in poor generalization, slower convergence, and reduced robustness of the global model. To address these issues, we propose a novel technique that incorporates a gradient penalty term into partial variance control. Our method enables diverse representation learning from heterogeneous client data in the initial layers while modifying standard SGD in the final layers. This approach reduces the variance in the classification layers, aligns the gradients, and mitigates the effects of data heterogeneity. Through theoretical analysis, we establish convergence rate bounds for the proposed algorithm, demonstrating its potential for competitive convergence compared to current FL methods in highly heterogeneous data settings. Empirical evaluations on five benchmark datasets validate our approach, showing enhanced performance and faster convergence over state-of-the-art baselines across various levels of data heterogeneity.
URL: https://openreview.net/forum?id=I9VhJ5iLNr
---
Title: Models with a Cause: Causal Discovery with Language Models on Temporally Ordered Text Data
Authors: Bruce Rushing, Javier Gomez-Lavin
Abstract: While language models (LMs) have been proposed for causal discovery tasks, it remains unclear whether they possess the inductive biases necessary to identify causal structures in token generation processes. We investigate whether LMs can learn the causal structure governing how tokens depend on their predecessors by testing if they possess the temporal and statistical properties required for causal discovery. We prove that existing algorithms can recover a unique causal model when token sequences satisfy standard causal assumptions and have temporal ordering. LMs' sequential processing and positional encodings enable them to leverage this temporal information. Using controlled experiments on synthetic data generated by mixtures of Markov chains, we test whether LMs learn conditional independencies and Markov exchangeability properties necessary for causal discovery. We find that transformers successfully learn these properties, achieving this not by approximating exact probability distributions but by learning qualitative probability rankings. These synthetic experiments provide initial evidence that LMs possess inductive biases suitable for discovering token-level causal structures.
URL: https://openreview.net/forum?id=YJddclPGuY
---
Title: Source-Optimal Training is Transfer-Suboptimal
Authors: C. Evans Hedges
Abstract: We prove that training a source model optimally for its own task is generically suboptimal when the objective is downstream transfer. We study the source-side optimization problem in L2-SP (L2-distance to Starting Point) ridge regression, where the target estimator is regularized toward the source model parameters, and show a fundamental mismatch between the source-optimal regularization $\tau_S^*$ (minimizing source risk) and the transfer-optimal regularization $\tau_0^*$ (maximizing downstream transfer): outside of a measure-zero set, $\tau_0^* \neq \tau_S^*$. We characterize $\tau_0^*$ as a function of the normalized task alignment $\rho = \braket{w_0, w_1}/\|w_0\|^2$ and identify an alignment-dependent reversal: with imperfect alignment ($0<\rho<1$), transfer benefits from stronger source regularization, while in super-aligned regimes ($\rho>1$), transfer benefits from weaker regularization. In isotropic settings, whether transfer helps is independent of target sample size and noise. We verify the phase transition in synthetic experiments across overparameterization ratios and covariance structures, and present nonlinear experiments on MNIST, CIFAR-10, and 20 Newsgroups showing that the mismatch persists in standard transfer learning pipelines, with explicit L2-SP fine-tuning closely tracking standard SGD and the target sample-size independence prediction confirmed empirically.
URL: https://openreview.net/forum?id=CMlpokFXfA
---
Title: Primus: Enforcing Attention Usage for 3D Medical Image Segmentation
Authors: Tassilo Wald, Saikat Roy, Fabian Isensee, Constantin Ulrich, Sebastian Ziegler, Dasha Trofimova, Raphael Stock, Michael Baumgartner, Gregor Koehler, Klaus Maier-Hein
Abstract: Transformers have achieved remarkable success across multiple fields, yet their impact on 3D medical image segmentation remains limited with convolutional networks still dominating major benchmarks. In this work, (A) we analyze current Transformer-based segmentation models and identify critical shortcomings, particularly their over-reliance on convolutional blocks. Further, we demonstrate that in some architectures, performance is unaffected by the absence of the Transformer, thereby demonstrating their limited effectiveness. To address these challenges, we move away from hybrid architectures and (B) introduce Transformer-centric segmentation architectures, termed Primus and PrimusV2. Primus leverages high-resolution tokens, combined with advances in positional embeddings and block design, to maximally leverage its Transformer blocks, while PrimusV2 expands on this through an iterative patch embedding. Through these adaptations, Primus surpasses current Transformer-based methods and competes with a default nnU-Net while PrimusV2 exceeds it and is on par with the state-of-the-art CNNs such as ResEnc-L and MedNeXt architectures across nine public datasets. In doing so, we introduce the first competitive Transformer-centric model, making Transformers state-of-the-art in 3D medical segmentation. Code is made available.
URL: https://openreview.net/forum?id=x4vZE4PDEu
---
Title: SensX: Model-Agnostic Local Feature Attribution via Calibrated Global Sensitivity Analysis
Authors: Manu Aggarwal, Nicholas Cogan, Vipul Periwal
Abstract: Local feature attribution is a standard tool for auditing and debugging deep learning predictions, but existing attribution methods are not designed for systems that chain pretrained, frozen, or API-only modules. Gradient-based methods such as Integrated Gradients require an end-to-end computational graph that may be unavailable. Perturbation-based methods such as KernelSHAP require a reference input or background distribution whose choice can substantially alter attributions and may not be defensible for composite pipelines. We present SensX, a local attribution method that treats the model as a black box and replaces arbitrary design choices with interpretable, application-grounded parameters. SensX adapts Morris-style coordinate walks from global sensitivity analysis to local attribution. It requires no access to model internals, training data, or arbitrary reference inputs. We validate SensX across four case studies, each targeting a distinct limitation of existing methods. On a synthetic benchmark where ground-truth relevant features vary per input, SensX reaches $95\%$ top-$2$ attribution accuracy versus $58\%$ for the best KernelSHAP/Integrated Gradients variant. On a ViT with $>150{,}000$ pixel-channel features, SensX produces spatially coherent maps and exposes systematic intra-patch bias where KernelSHAP is infeasible and Integrated Gradients yields task-irrelevant attributions. On single-cell classifiers with unstructured gene-expression features, SensX attains the lowest top-$k$ perturbation AUC. On a composite spatial transcriptomics system where neither method is applicable, SensX reveals reliance on preprocessing grid artifacts and a bias toward low-staining regions.
URL: https://openreview.net/forum?id=dKzReyfUeW
---
New submissions
===============
Title: pTopoFL: Privacy-Preserving Personalised Federated Learning via Persistent Homology
Abstract: Federated learning (FL) faces two structural tensions: gradient sharing enables datareconstruction attacks, while non-IID client distributions degrade aggregation quality. We introduce pTopoFL, a framework that addresses both challenges simultaneously by replacing gradient communication with topological descriptors derived from persistent homology (PH). Clients transmit only PH feature vectors—shape summaries whose many-to-one structure makes inversion provably ill-posed—rather than model gradients. The server performs
topology-guided personalised aggregation: clients are clustered by Wasserstein similarity between their PH diagrams, intra-cluster models are topology-weighted, and clusters are blended with a global consensus. We prove an information-contraction theorem showing
that PH descriptors leak strictly less mutual information per sample than gradients, and we establish linear convergence of the Wasserstein-weighted aggregation scheme. Evaluated against FedAvg, FedProx, SCAFFOLD, and pFedMe on a non-IID healthcare scenario (8 hospitals) and a pathological benchmark (10 clients), pTopoFL achieves AUC 0.841 and 0.910 respectively—the highest in both settings—while reducing reconstruction risk 4.5× relative to gradient sharing. Code and data are publicly available at X.
URL: https://openreview.net/forum?id=T4bwr2ZJlH
---
Title: QMoE+: Hybrid Quantum Mixture of Experts
Abstract: Quantum mixture of experts (QMoE) extends conditional computation to the NISQ setting by distributing learning across parameterized quantum circuit (PQC) experts selected via a routing mechanism. Existing approaches are limited by single-block experts, lack of load balancing, and aggregation schemes that ignore routing amplitudes. We propose QMoE+, which uses two-block data re-uploading experts with learnable offsets, a coherent aggregation circuit over the joint routing-data Hilbert space, and a Switch-style load-balancing loss. Under top-k=1 sparse routing, QMoE+ activates only ∼28% of its parameters per inference while achieving consistent accuracy gains across seven datasets and four gate sets, winning 27/28 configurations with a mean improvement of +5.11% in the noiseless setting and +4.71% under depolarising noise. A decomposed ablation further shows that quantum coherence in the aggregation circuit outperforms an incoherent baseline in all seven datasets under p=0.01 noise (mean +1.80%), establishing an independent contribution beyond learnable aggregation parameters alone. Ablations confirm that load balancing is consistently beneficial, while data re-uploading provides the largest gains on complex tasks. Code is available at https://anonymous.4open.science/r/qmoe-plus/.
URL: https://openreview.net/forum?id=l1JaPqZ6K5
---
Title: Task-Relevant Language-conditioned Segmentation for Robust Generalization in Reinforcement Learning
Abstract: Humans possess a remarkable ability to filter out irrelevant sensory clutter, extracting only the information needed to anticipate and act within dynamic environments. Prior attempts to mitigate this through augmentation and masking strategies have improved robustness, but remain limited by computational overhead, weak semantic grounding, or instability in actor-critic training. Inspired by how language guides human perception, we introduce Task Relevant Language-conditioned Segmentation (TaLaS), a framework that leverages language-conditioned segmentation to impose semantic structure on visual observations. TaLaS employs a two-phase design where in the first phase, a lightweight masker is pretrained on unaugmented, language-guided masks; in the second phase, a student masker is regularized with strong augmentations to enforce consistency. This yields a task-relevant feature extractor that improves policy stability and removes the need for online segmentation at inference time. To address the actor’s deployment distribution shift, we employ asymmetric actor-critic training. TaLaS improves robustness to distractors and achieves particularly strong performance under challenging visual shifts on RL-ViGen, while remaining competitive in easier settings. The benchmark includes challenging variants of the DeepMind Control Suite, Quadruped Locomotion and Dexterous Manipulation tasks. https://talas-rl.github.io/.
URL: https://openreview.net/forum?id=MRHXB6eooE
---
Title: Graph Contrastive Learning via Weisfeiler-Leman Dual-View Sampling
Abstract: Graph contrastive learning (GCL) approaches have gained momentum over the past few years. By augmenting the original graph data, the common GCL pipeline learns from such multiple contrastive graph views in a self-supervised manner, tackling critical issues in the literature, such as node label scarcity. To obtain contrastive views, most GCL techniques heavily rely on feature-space similarity measures. We consider this as a limiting factor in GCL, since it implies that node features are (in general) informative and closely aligned with the graph topology, an assumption that does not hold, for instance, in the case of heterophilic graphs. In this work, we propose to address the problem by coupling the usual feature-space similarities with structure-based measures, which we propose to implement through the Weisfeiler-Leman (WL) family of algorithms. Our framework, dubbed WLGCL, introduces a dual-view sampling strategy that works on features- and WL-level to construct more reliable contrastive pairs. WLGCL integrates a multi-positive and hard-negative contrastive loss to ensure the alignment-uniformity trade-off without modifying the design of the graph encoder. Extensive experiments on six benchmark datasets and against seven state-of-the-art baselines demonstrate the efficacy of WLGCL, where additional empirical evaluations justify the adoption of our architectural choices for the model.
URL: https://openreview.net/forum?id=uuk14WVKyj
---
Title: Efficient Fine-Tuning of Large Language Models with Zeroth-Order Model Parallelism
Abstract: Model parallelism (MP) is a widely adopted paradigm for scaling large language model (LLM) training across multiple nodes. Yet, existing methods mainly rely on first-order optimization, which suffer from two key bottlenecks: high communication overhead due to frequent transmission of activations and gradients, and substantial memory consumption caused by caching these intermediate states. Zeroth-order (ZO) optimization offers a compelling alternative by eliminating explicit gradient computation and storage, naturally reducing communication and memory costs. Despite these advantages, ZO methods remain largely unexplored in the context of MP for LLM fine-tuning. In this work, we first investigate activation sparsity patterns induced by common activation functions (e.g., ReLU, GELU, SwiGLU) during LLM fine-tuning. Motivated by these key observations, we propose SparQ, a ZO-based MP framework that exploits quantization-induced activation sparsity to reduce memory footprint and communication overhead. SparQ consists of three key components: (1) using the gradient-free nature of ZO optimization to eliminate gradients; (2) applying activation quantization to induce sparsity that enables efficient sparse encoding; and (3) strategically placing split layers at sparsity-rich regions and transmitting activations in sparse form, significantly reducing communication cost with minimal impact on model performance. We theoretically establish that SparQ achieves a sublinear convergence rate for non-convex objectives. Extensive experiments show that SparQ reduces GPU memory usage by over 3× and communication cost by 50%+ compared to state-of-the-art MP baselines, while maintaining comparable LLM fine-tuning performance across multiple tasks.
URL: https://openreview.net/forum?id=9wpm4fbBJI
---
Title: Number Value Loss in LLMs and N-adic Tokenization
Abstract: This paper provides a theoretical analysis of metrics for numerical value comparison and reasoning in machine learning. We address the root causes of optimization failure in numerical representations through three primary contributions.
First, we introduce MALL (Magnitude-Aware Log Loss), a metric designed to ensure gradient stability and sensitivity across more than 30 orders of magnitude. We demonstrate that MALL maintains a robust signal for both global magnitude and local precision across the entire $\mathbb{R}^{+}$ domain, resolving the vanishing and exploding gradient problems inherent in traditional metrics. This ensures a stable foundation for numerical reasoning of bijective to number value tokenizations, decoded token sequences or regressions and makes MALL a superior drop-in replacement and a standalone baseline for numerical comparison.
Second, we identify the Softmax boundary problem --- a fundamental structural failure at digit-order transitions caused by the interplay between independent positional distributions and positional tokenization. We establish a No-Go theorem proving that additive per-token continuous losses are mathematically incompatible with numerical stability over large ranges. Consequently, we demonstrate that structured discontinuities in the gradient field act as a necessary catalyst for global consistency and propose a deferred global loss with hardmax as a regularization strategy to stabilize this behavior.
Third, we propose a geometrical embedding regularizer, Triangle Loss, based on the triangle inequality to enforce numerical continuity within the embedding manifold. By ensuring that the geometric relationships between embeddings reflect their numerical distances, Triangle Loss improves generalization for rare tokens in any numerical bijective tokenization and provides a structural basis for learning numerical proximity at extreme scales.
Through mathematical proofs and gradient field visualizations, we demonstrate that our framework addresses the fundamental limitations of current numerical objectives, providing a robust foundation for coherent numerical intelligence in neural architectures.
URL: https://openreview.net/forum?id=8NULHa8vrm
---
Title: $\phi$-Table: A Statistical Explanation for Global SHAP
Abstract: Global SHAP explanations are typically presented as feature-importance rankings, which identify variables that matter to a black-box model but do not indicate whether their effects admit clear directional summaries, how uncertain those summaries are, or how faithfully they represent the fitted response. This paper proposes the $\phi$-table, a SHAP-based statistical explanation table for tabular black-box regression models. The procedure selects features by SHAP importance and fits a standardized linear surrogate to the fitted model response $f(X)$, reporting SHAP importance together with model-response coefficients, uncertainty summaries, surrogate fidelity, and bootstrap coefficient stability. The resulting coefficients are interpreted as projections of the fitted model response onto the SHAP-selected feature set. Across synthetic, semi-synthetic, and real-data experiments, the $\phi$-table extends ranking-only SHAP into a statistical global explanation by exposing direction, uncertainty, fidelity, and stability as distinct components of fitted model behavior.
URL: https://openreview.net/forum?id=3fW4xShek8
---
Title: Anderson Accelerated Asynchronous Method for Distributed Optimization
Abstract: Anderson acceleration (AA) is an effective technique for accelerating fixed-point iterations, but it is rarely applied to distributed optimization or distributed machine learning. In this paper, we apply AA to accelerate an asynchronous distributed gradient method over the master-worker architecture, resulting in the Asynchronous Distributed Gradient Method with Anderson Acceleration (ADGM-AA). In particular, we first transform the asynchronous gradient method into a fixed-point iteration, and then incorporate it with AA. To ensure the global convergence of ADGM-AA, we equip it with a novel reference-path-based safe-guard scheme. We prove that under mild conditions, ADGM-AA converges with fixed step-sizes that are independent of the delays. Compared with the delay-dependent step-size in most existing works, our delay-free step-size is easier to determine and often leads to faster convergence. To emphasize, numerical experiments demonstrate that by incorporating the AA scheme, the proposed ADGM-AA significantly outperforms the vanilla asynchronous distributed gradient method.
URL: https://openreview.net/forum?id=Nm7dTogzfa
---
Title: Generative Modeling with Bayesian Sample Inference
Abstract: We derive a novel generative model from iterative Gaussian posterior inference. By treating the generated sample as an unknown variable, we can formulate the sampling process in the language of Bayesian probability. Our model uses a sequence of prediction and posterior update steps to iteratively narrow down the unknown sample starting from a broad initial belief. In addition to a rigorous theoretical analysis, we establish a connection between our model and diffusion models and show that it includes Bayesian Flow Networks (BFNs) as a special case. In our experiments, we demonstrate that our model improves sample quality on ImageNet32 over both BFNs and the closely related Variational Diffusion Models, while achieving equivalent log-likelihoods on ImageNet32 and ImageNet64.
URL: https://openreview.net/forum?id=n8qQPnalVW
---
Title: They Are Not Static: A Survey of Dynamic Agent Skills
Abstract: Large language model agents increasingly externalize procedural knowledge into reusable skills: invocable code, natural-language procedures, SKILL.md packages, graphs, or parametric adapters. This externalization turns adaptation into a new learning problem. The agent does not only update its prompt or weights; it updates a library of artifacts that changes what future policies can retrieve, compose, execute, and trust. This survey studies the rapidly growing 2023–2026 literature on dynamic or self-evolving skill systems and argues that such systems are best understood as lifecycle-managed, verified, evolving artifact stores for LLM agents. We extend the options-based skill formalism to a seven-tuple—applicability, policy, termination, interface, edit, verification, and lineage—that makes edits, admission verification, and provenance explicit. We further lift this view to library-level dynamics, in which a library at time t is transformed into a new library at time t+1 by a ten-operator algebra: ADD, REFINE, MERGE, SPLIT, PRUNE, DISTILL, ABSTRACT, COMPOSE, REWRITE, and RERANK. Using this formalism, we organize a 94-paper modern audit set of dynamic-skill and boundary/context papers around a skill lifecycle: evidence acquisition, proposal, verification and admission, organization, retrieval and composition, maintenance and repair, distillation and portability, and governance. The resulting taxonomy separates artifact families, update loci, assurance models, storage topologies, maintenance regimes, and governance maturity without reducing the field to a list of systems. We then synthesize the mechanisms that make lifecycle-managed stores improve: edit repertoires, admission gates, storage and retrieval structure, and fast-slow update clocks. The most consistent evidence is that admission and repair matter more than raw skill count, verifier quality is often load-bearing in skill-aware reinforcement learning, flat retrieval degrades in the moderate-library-size regime, and benchmarks still under-report library trajectories. We close with a research agenda for compositional verifiers, maintenance schedules, registry-scale retrieval, cross-library portability, provenance, and lifecycle-aware evaluation.
URL: https://openreview.net/forum?id=cjU3YbcRr8
---
Title: Beyond Row-Level Prediction: A Unified Evaluation of Table Representation Methods and Recoverable Table-Level Geometry
Abstract: Tables exhibit multiple interacting levels of structure, and useful table representations must compose fine-grained local signals into reusable whole-table embeddings. Yet table representation methods are still often assessed through row-level prediction or downstream supervised tasks rather than through the quality of the table-level representations they produce. We introduce a unified evaluation framework for table representation methods built around four practical desiderata: consistency under partial views, discriminability across label granularities, robustness to benign perturbations, and efficiency. Across controlled synthetic families and real open-source corpora, we find a consistent pattern: lightweight schema- and text-based methods often outperform naive mean-pooled embeddings from state-of-the-art tabular foundation models on the practical quality-cost frontier. This suggests that table-level representations are not an automatic byproduct of predictive training, but depend critically on how local tabular signals are composed into a global representation. To test this hypothesis, we freeze the encoder of a tabular foundation model and train lightweight representation heads on top of its outputs. The learned heads substantially improve table-level geometry over naive pooling, showing that useful compositional structure is recoverable from the same encoder states, although bounded by the information ceiling of the frozen backbone. A closed-corpus parent-table retrieval proof of concept mirrors the benchmark trends and again shows simple methods outperforming pooled tabular foundation models. Together, these results position table-level representation as a first-class problem beyond row-level prediction and highlight learned composition as a key ingredient in reusable representations.
URL: https://openreview.net/forum?id=52Iefj80vA
---
Title: Energy-Based Constraint Networks: Learning Structural Coherence Across Modalities
Abstract: We introduce energy-based constraint networks -- a modality-agnostic architecture that learns structural coherence from contrastive pairs. The system processes frozen encoder embeddings through a state-space model with dual-head attention, producing a scalar energy measuring structural consistency alongside per-position energy scores that localize violations. Multiple independently trained branches detect different violation types and compose at inference without interference.
We demonstrate the framework in two domains. In text, the system achieves 93.4% accuracy on trained corruption types and 87.2% on 9 unseen types, using frozen BERT and 7.4M trainable parameters. In vision, the same architecture achieves competitive deepfake detection: 0.959 AUC on FaceForensics++ Deepfakes and 0.870 on Celeb-DF without any Celeb-DF training data, using frozen DINOv2 and 3.6M parameters per branch.
The framework supports flexible training: branches learn from designer-specified corruptions, real-world paired data, or both. Composable branches require representation compatibility -- a finding validated through extensive experimentation where five incompatible approaches failed before the compatible one succeeded. The architecture is encoder-agnostic and domain-agnostic: changing the domain requires only new corruption strategies; changing the encoder requires only a new input projection layer. To our knowledge, this is the first architecture to learn within-modality structural coherence as an explicit energy landscape with per-position decomposition, and to demonstrate that the same architecture transfers across modalities via corruption respecification alone.
URL: https://openreview.net/forum?id=gl6l8nTXBB
---
Title: FairNVT: Dual-Level Fairness via Noise Injection in Vision Transformers
Abstract: This paper presents FairNVT, a lightweight debiasing framework for pretrained transformer-based encoders that improves both representation and prediction level fairness while preserving task accuracy. Unlike many existing debiasing approaches that address these notions separately, we argue they are inherently connected: suppressing sensitive information at the representation level can facilitate fairer predictions. Our approach learns task-relevant and sensitive embeddings via lightweight adapters, applies calibrated Gaussian noise to the sensitive embedding, and fuses it with the task representation. Together with orthogonality constraints and fairness regularization, these components jointly reduce sensitive-attribute leakage in the learned embeddings and encourage fairer downstream predictions. The framework is compatible with a wide range of pretrained transformer encoders. Across three datasets spanning vision and language, FairNVT reduces sensitive-attribute attacker accuracy, improves demographic-parity and equalized-odds metrics, and maintains high task performance.
URL: https://openreview.net/forum?id=rzm6gZrYgl
---
Title: CURE-OOD: Benchmarking Out-of-Distribution Detection for Survival Prediction
Abstract: ``How long can I live and remain free of cancer?'' is often the first question a patient asks after receiving a cancer diagnosis and treatment.
Accurate survival prediction helps alleviate psychological distress and supports risk stratification and personalized treatment planning.
Recent survival prediction frameworks have shown strong performance using computed tomography (CT) images. However, variations in imaging acquisition introduce out-of-distribution (OOD) samples caused by covariate shifts that undermine model reliability. Despite this challenge, to our knowledge, no existing benchmark systematically studies OOD detection in cancer survival prediction. To address this gap, we introduce the Cancer sURvival bEnchmark for OOD Detection (CURE-OOD), the first benchmark for systematically evaluating OOD detection in survival prediction under controlled acquisition-induced distribution shifts. CURE-OOD defines scanner-parameter-based training, in-distribution (ID), and OOD test splits across four survival prediction tasks. Our experiments show that covariate shifts notably reduce survival prediction performance. It also shows that mainstream classification-oriented OOD detectors can fail in survival prediction. Finally, we include HazardDev as a simple survival-aware reference baseline for OOD detection. CURE-OOD enables systematic analysis of how distribution shifts affect both downstream survival performance and OOD detectability.
URL: https://openreview.net/forum?id=fziI7nE1vO
---
Title: Instructions shape Production of Language, not Processing
Abstract: Instructions trigger a production-centered mechanism in language models. Through a cognitively inspired lens that separates language processing and production, we reveal this mechanism as an asymmetry between the two stages by probing task-specific information layer-wise across five binary judgment tasks. Specifically, we measure how instruction tokens shape information both when sample tokens — the input under evaluation — are processed and when output tokens are produced. Across prompting variations, task-specific information in sample tokens stays largely stable and correlates only weakly with behavior, whereas the same information in output tokens varies substantially and correlates strongly. Attention-based interventions confirm this pattern causally: blocking instruction flow to all subsequent tokens reduces both behavior and information in output tokens, whereas blocking it only to sample tokens has minimal effect on either. The asymmetry generalizes across model families and tasks, and sharpens with model scale and instruction-tuning — both of which disproportionately affect the production stage. Our findings suggest that understanding model capabilities requires both jointly assessing internals and behavior, and decomposing the internal perspective by token position to separate the processing of input tokens from the production of output tokens.
URL: https://openreview.net/forum?id=XhU5Q7RNtK
---
Title: ACTIVA: Amortized Causal Effect Estimation via Variational Autoencoders
Abstract: Predicting post-intervention distributions from observational data is central to many scientific and decision-making problems, but remains challenging due to causal ambiguity, restrictive modeling assumptions, and the lack of amortization across tasks. We introduce ACTIVA, a transformer-based conditional variational autoencoder for amortized estimation of full interventional distributions from observational data and intervention queries. ACTIVA learns a conditional latent prior that supports zero-shot inference by amortizing causal knowledge across diverse training tasks.
We provide a consistency result showing that, under idealized conditions, ACTIVA's learning objective targets a mixture over the interventional distributions of causal models that are observationally compatible with the input. Empirically, on synthetic datasets and biologically realistic gene-expression simulations, ACTIVA substantially outperforms a correlational baseline, reduces spurious non-descendant effects, and achieves competitive performance relative to strong amortized baselines. Our results show that ACTIVA is a promising approach for estimating interventional distributions from observational data.
URL: https://openreview.net/forum?id=819pIIPuVl
---
Title: Enhance the Safety in Reinforcement Learning by ADRC Lagrangian Methods
Abstract: Safe reinforcement learning (Safe RL) seeks to maximize rewards while satisfying safety constraints, typically addressed through Lagrangian-based methods. However, existing approaches, including PID and classical Lagrangian methods, suffer from oscillations and frequent safety violations due to parameter sensitivity and inherent phase lag. To address these limitations, we propose ADRC-Lagrangian methods that leverage Active Disturbance Rejection Control (ADRC) for enhanced robustness and reduced oscillations. Our unified framework encompasses classical and PID Lagrangian methods as special cases while significantly improving safety performance. Extensive experiments demonstrate that our approach reduces safety violations by up to 74\%, constraint violation magnitudes by 89\%, and average costs by 67\%, establishing superior effectiveness for Safe RL in complex environments.
URL: https://openreview.net/forum?id=3IbuT8uzYS
---
Title: REPA-FPO: A Fisher Policy Optimization for Efficient Flow Matching Training
Abstract: Flow Matching (FM) models are a leading class of generative models, widely used across diverse domains. However, FM models require large-scale training datasets, which makes training computationally expensive. Existing feature alignment (REPA) improves training efficiency but overlooks the role of the data itself, leaving further room for improvement. In this paper, we observe that different samples carry different amounts of Fisher information and thus contribute unequally to parameter learning in FM. This heterogeneity highlights the importance of accounting for sample-wise contributions during training. However, computing per-sample Fisher information accurately is prohibitively expensive in practice. To overcome this limitation, we provide a mathematical analysis showing that the loss magnitude can serve as an effective proxy for the trace of the Fisher Information Matrix (FIM), enabling efficient estimation. Building on this insight, we propose Fisher Policy Optimization (FPO), a strategy that dynamically reweights samples during training by shifting weight from low-FIM samples to high-FIM samples. Extensive experiments demonstrate that FPO improves both training efficiency and generation quality, while generalizing well across inference samplers, model architectures, and diffusion spaces.
URL: https://openreview.net/forum?id=mRHipMopOC
---
Title: Strategies for Cold-Starting Active Learning Loops Using Multiple Data Modalities
Abstract: In experimental materials science, every measurement counts.
Discovering new materials in the context of compositionally complex materials is time-consuming and costly because of the high number of measurements required to screen the large composition-property space.
To address this, there is a growing need for acceleration strategies that minimize data collection combined with acceptable surrogate model accuracy.
Active learning can significantly reduce the number of labeled data points (measurements) required to train surrogate machine learning models while still achieving high predictive performance with low uncertainty.
However, a major challenge in active learning is the cold-start problem: How to select informative initial points when no labeled data are yet available?
We present and systematically evaluate multiple cold-start initialization strategies for active learning loops based on different existing ``cheap'' data modalities, as well as their multimodal combination.
These strategies provide diverse and representative starting points and lead to rapid model convergence.
Two acquisition functions, \textit{Uncertainty Sampling} (US) and \textit{Self-Adjusting Weighted Expected Improvement} (SAWEI), are compared for iterative point selection, automatically balancing exploration and exploitation.
Active learning is stopped dynamically by monitoring the normalized mean predictive variance of the surrogate model.
We apply our approach to eight experimental composition-spread materials libraries, a common setup for high-throughput screening, with different levels of compositional complexity.
For those materials libraries we learn a surrogate model to predict electrical resistance as a function of composition.
Our active learning framework significantly reduces the number of required measurements, achieving a reduction of 87\% for some materials libraries using a single modality and a reduction of 85\% on average for all materials libraries using a multimodal cold-start strategy.
On average, we find that SAWEI outperforms uncertainty sampling.
In summary, we demonstrate a practical, cold start active learning framework using a multimodal approach that accelerates autonomous experimental characterization on the path to autonomous materials discovery.
URL: https://openreview.net/forum?id=ZLoSdR9Xqb
---
Title: Diffusion-Based Hypothesis Testing and Change-Point Detection
Abstract: Score-based methods have recently seen increasing popularity in modeling and generation. Methods have been constructed to perform hypothesis testing and change-point detection with score functions, but these methods are in general not as powerful as their likelihood-based peers. Recent works consider generalizing the score-based Fisher divergence into a diffusion-divergence by transforming score functions via multiplication with a matrix-valued function or a weight matrix. In this paper, we extend the score-based hypothesis test and change-point detection stopping rule into their diffusion-based analogs. Additionally, we theoretically quantify the performance of these diffusion-based algorithms and study scenarios where optimal performance is achievable. We propose a method of numerically optimizing the weight matrix and present numerical simulations to illustrate the advantages of diffusion-based algorithms.
URL: https://openreview.net/forum?id=gaf6vUQSSr
---
Title: Discretization and Predictive Complexity of Time Series
Abstract: Discretization is widely used in time-series analysis to convert continuous observations into
symbolic sequences before sequence modeling. Its effect on forecasting, however, is not
merely representational: discretization may preserve the predictive structure of the original
process, or it may destroy it by merging histories that imply different future distributions.
In this paper, we study discretization
through the lens of predictive states and Hankel-rank-based predictive complexity.
We first formalize predictive-sufficient discretization and review how predictive-state
collapse under coarsening reduces predictive complexity. We then introduce synthetic same-\(K\)
hidden Markov model families that share the same hidden-state cardinality but exhibit different
Bayes-level context gaps. These families allow us to separate nominal hidden-state size from
observable predictive difficulty in a controlled setting. Our experiments show that hidden-state
count alone does not determine forecasting difficulty, even when the latent-state cardinality is
fixed. Moreover, learner-side recovery of Bayes-level context sensitivity is family-dependent and
non-monotone in hidden dimension: some families benefit from moderate increases in representation
size, whereas others degrade when the model dimension becomes unnecessarily large. Taken together,
these results suggest that, in the present same-\(K\) setting, representation requirements are not
explained by hidden-state count alone, but also depend on the family-specific predictive structure
that remains observable after discretization.
URL: https://openreview.net/forum?id=ebUNKFaxSp
---
Title: GATTA: Graph Active Learning with Test-Time Augmentation
Abstract: Test-time augmentation (TTA) has proven effective for improving model robustness and uncertainty estimation in computer vision, yet its application to graph-structured data remains largely unexplored. We introduce GATTA (Graph Active Learning with Test-Time Augmentation), a framework for enhancing active learning by aggregating predictions across multiple augmented views to produce more reliable uncertainty estimates. To address the challenge of label-preserving graph augmentations, GATTA incorporates a consistency-based filtering mechanism that discards augmented views yielding unreliable predictions.
We systematically evaluate GATTA across multiple graph datasets, GNN architectures, and acquisition strategies. Our results show that simple uncertainty-based methods, such as Entropy and Least Confidence, benefit most from TTA, achieving performance competitive with more sophisticated and computationally expensive approaches. GATTA generalizes across architectures, outperforms model-side ensemble methods such as MC Dropout. We further show that GATTA scales efficiently with both ensemble size and graph size. Extensive analysis of augmentation types, strengths, and filtering strategies provides practical guidelines for effective deployment.
Our findings demonstrate that augmenting simple methods with TTA offers a more efficient path to strong active learning performance than engineering complex acquisition functions, enabling practitioners to achieve competitive results with lower computational overhead and reduced implementation complexity.
URL: https://openreview.net/forum?id=gQjXARc6tt
---
Title: A Hierarchical Geometric Observation Interface for Spatial Planning in Reinforcement Learning
Abstract: In reinforcement learning (RL), spatial planning is often mediated through rasterized observations processed by convolutional networks, even when the underlying task is continuous and geometric. This discretization can introduce aliasing and obscure topological structure, increasing the difficulty of the spatial problem. We study a hierarchical set-valued geometry-first observation interface for sparse-reward navigation that operates directly on triangulated obstacle geometry. This interface uses learned multi-token aggregation to compress variable-sized geometry into a bounded fixed-size representation while preserving local spatial structure relevant for spatial decision making. In a controlled goal-conditioned point-navigation setting with a fixed RL backbone, we compare it against raster--CNN baselines across bounded and unbounded procedural training regimes. The empirical results of our work demonstrate that its advantage is most pronounced under continual exposure to newly generated environments, where the agent must learn reusable spatial structure rather than rely on memorizing a fixed environment support.
URL: https://openreview.net/forum?id=U10DFNcMrW
---
Title: Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery
Abstract: Circuit discovery methods identify subgraphs that explain specific model behaviors, and structural differences between discovered circuits are commonly interpreted as evidence of distinct mechanisms. We test this assumption by varying input statistics while holding the task fixed, and show that the resulting structural differences exhibit apparent specialization but do not correspond to functional differences, a pattern we term phantom specialization. Using Literal Sequence Copying across four token-frequency bands in five Pythia models (70M-1.4B), we extract 75 circuits and find that structurally distinct circuits implement the same computation: band-specific edges transfer broadly across bands, a shared core recovers at least 99% of circuit performance, and causal interchange interventions confirm that internal representations are interchangeable across frequency bands. Repeated extractions within the same frequency band further suggest that discovery algorithms sample from an equivalence class of valid subgraphs rather than recovering a unique mechanism. Standard evaluation practice obscures this pattern: source-level evaluation inflates apparent faithfulness, while edge-level evaluation reveals the many-to-one mapping from structure to function. Our results show that structural differences between circuits are not sufficient evidence for distinct mechanisms, and that exposing this requires edge-level evaluation and cross-condition transfer tests.
URL: https://openreview.net/forum?id=Ur42Xx1QPv
---
Title: VAE-Inf: A statistically interpretable generative paradigm for imbalanced classification
Abstract: Imbalanced classification remains a pervasive challenge in machine learning, particularly when minority samples are too scarce to provide a robust discriminative boundary. In such extreme scenarios, conventional models often suffer from unstable decision boundaries and a lack of reliable error control. To bridge the gap between generative modeling and discriminative classification, we propose a two-stage framework \textbf{VAE-Inf} that integrates deep representation learning with statistically interpretable hypothesis testing.
In the first stage, we adopt a one-class modeling perspective by training a variational autoencoder (VAE) exclusively on majority-class data to capture the underlying reference distribution. The resulting latent posteriors are aggregated via a Wasserstein barycenter to construct a global Gaussian reference model, providing a geometrically principled baseline for the majority class. In the second stage, we transform this generative foundation into a discriminative classifier by fine-tuning the encoder with limited minority samples. This is achieved through a novel distribution-aware loss that enforces probabilistic separation between classes based on variance-normalized projection statistics.
For inference, we introduce a projection-based score that admits a natural hypothesis testing interpretation, allowing for a distribution-free calibration procedure. This approach yields exact finite-sample control of the Type-I error (false positive rate) without relying on restrictive parametric assumptions. Extensive experiments on diverse real-world benchmarks demonstrate that our framework achieves competitive performance against other approaches. The codes are available upon request.
URL: https://openreview.net/forum?id=XSgjCi4iCf
---
Title: SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign
Abstract: Proteins are fundamental to biological processes, with their function determined by the complex interplay between the amino acid sequence and the three-dimensional structure. Developing generative models capable of understanding this intrinsically multi-modal relationship is crucial for fields like drug discovery and protein engineering. Existing models often rely on a multi-stage training process where autoencoders that tokenize data into latent representations are trained in a first stage. Secondly, a generative model is trained on the latent representation of the autoencoder(s), i.e., generative modeling in a latent space. We hypothesize that this multi-stage training is not necessary to obtain performant co-design models and thus present SimpleDesign, an effective multi-modal protein design model trained directly in the data space. SimpleDesign leverages a single-stage end-to-end objective that combines discrete cross-entropy for sequences and a regression objective for structures. In order to effectively model the difference in sequence and structure modalities, we develop a Mixture-of-Transformer architecture that allows modality-specific processing while keeping global self-attention over both modalities. We train SimpleDesign on over 2M sequence-structure pairs achieving strong performance across co-design and unconditional sequence/structure generation benchmarks.
URL: https://openreview.net/forum?id=wPfw7GkMns
---
Title: ReDiTT: Retrieval Augmented Conditional Diffusion Transformers for Asynchronous Time Series
Abstract: We present a diffusion based model for asynchronous time series prediction, where the goal is to predict the next inter event time and event type. To address the inherent uncertainty of future events, we introduce ReDiTT, a retrieval augmented conditional diffusion transformer that operates in latent space. ReDiTT retrieves structurally similar latent sequences from a memory bank during both training and inference and incorporates them as reference conditions through cross attention. This retrieval based conditioning allows the model to attend to relevant temporal dynamics and provides global structural guidance for generation. As a result, ReDiTT stabilizes long horizon forecasting and improves sample diversity. Experiments on seven real world datasets demonstrate state of the art performance on next event prediction and long horizon forecasting.
URL: https://openreview.net/forum?id=sv35KiCipb
---
Title: ShapeY: A Principled Framework for Measuring Shape Recognition Capacity via Nearest-Neighbor Matching
Abstract: Object recognition (OR) in humans relies heavily on shape cues and the ability to recognize objects across varying 3D viewpoints. Unlike humans, deep networks often rely on non-shape cues such as texture and background, leading to vulnerabilities in generalization and robustness. To address this gap, we introduce ShapeY, a novel and principled benchmarking framework designed to evaluate shape-based recognition capability in OR systems. ShapeY comprises 68,200 grayscale images of 200 3D objects rendered from multiple viewpoints and optionally subjected to non-shape ``appearance'' changes. Using a nearest-neighbor matching task, ShapeY specifically probes the fine-grained structure of an OR system's embedding space by evaluating whether object views are clustered by 3D shape similarity across varying 3D viewpoints and other non-shape changes. ShapeY provides a suite of quantitative and qualitative performance readouts, including error rate graphs, viewpoint tuning curves, histograms of positive and negative matching scores, and grids showing ordered best matches, which together offer a comprehensive evaluation of an OR system's shape understanding capability. Testing of 321 pre-trained networks with diverse architectures reveals significant challenges in achieving robust shape-based recognition: even state-of-the-art models struggle to generalize consistently across 3D viewpoint and appearance changes, and are prone to infrequent but egregious matches of objects of obviously completely different shape. ShapeY establishes a principled framework for advancing artificial vision systems toward human-like shape recognition capabilities, emphasizing the importance of disentangled and invariant object encodings.
URL: https://openreview.net/forum?id=cTm6EecEBY
---
Title: Behavioral AI: Building Algorithms That Understand Us
Abstract: While AI research has historically worked toward developing tools to help people understand AI models, the emergence of generative AI into our daily lives suddenly makes the reverse question salient: how well can AI models understand people? Today's AI systems fall short; these deficiencies demand a focus toward building new systems that can understand people. In this Perspective, we endeavor to channel this focus into a new academic subfield, which we call Behavioral AI. This Perspective lays out dimensions of understanding that are currently deficient in AI systems, including emotional, intellectual, and preferential understanding. While improving AI systems along these dimensions faces a unique set of challenges, we show there has already been a flurry of progress across disciplines. As we build systems that better understand people, it will not only improve AI tools; if Behavioral AI succeeds as a field, these systems too can unlock new insights in the behavioral sciences that help us understand ourselves.
URL: https://openreview.net/forum?id=yyO6qlwxyu
---
Title: Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
Abstract: The rapid growth of large language models (LLMs) with diverse capabilities, costs, and domains has created a critical need for intelligent model selection at inference time. While smaller models suffice for routine queries, complex tasks demand more capable models. However, static model deployment does not account for the complexity and domain of incoming queries, leading to suboptimal performance and increased costs. Dynamic routing systems that adaptively select models based on query characteristics have emerged as a solution to this challenge.
We provide a systematic analysis of state-of-the-art multi-LLM routing and cascading approaches. In contrast to mixture-of-experts architectures, which route within a single model, we study routing across multiple independently trained LLMs. We cover diverse routing paradigms, including query difficulty, human preferences, clustering, uncertainty quantification, reinforcement learning, multimodality, and cascading. For each paradigm, we analyze representative methods and examine key trade-offs. Beyond taxonomy, we introduce a conceptual framework that characterizes routing systems along three dimensions: when decisions are made, what information is used, and how they are computed. This perspective highlights that practical systems are often compositional, integrating multiple paradigms under operational constraints.
Our analysis demonstrates that effective multi-LLM routing requires balancing competing objectives. Choosing the optimal routing strategy depends on deployment and computational constraints. Well-designed routing systems can outperform even the most powerful individual models by strategically leveraging specialized capabilities across models while maximizing efficiency gains. Meanwhile, open challenges remain in developing routing mechanisms that generalize across diverse architectures, modalities, and applications.
URL: https://openreview.net/forum?id=ypRg1TvQaM
---
Title: A Survey on Industrial Anomaly Synthesis
Abstract: This paper presents a comprehensive review of industrial anomaly synthesis (IAS). Existing surveys on industrial anomalies mainly focus on anomaly detection, while anomaly synthesis is typically treated as an auxiliary component rather than as an independent topic. However, owing to its increasing importance in data augmentation, downstream model training, and controllable industrial inspection, IAS has become a research direction of growing interest. To address the lack of a dedicated review, we survey a broad range of representative methods and organize them into four paradigms: hand-crafted synthesis, distribution hypothesis-based synthesis, generative model (GM)-based synthesis, and vision-language model (VLM)-based synthesis. We further establish a dedicated taxonomy for IAS, which supports more systematic comparison across methods and offers a clearer view of the field’s development. Beyond methodological categorization, we summarize the datasets, benchmarks, and evaluation metrics commonly adopted in IAS, and review recent advances in multimodal anomaly synthesis that remain underexplored in prior surveys. Overall, this survey provides a structured understanding of existing IAS methods, evaluation settings, current limitations, and promising future directions, and is intended to serve as a reference for subsequent research in this area. More resources are available at https://anonymous.4open.science/status/IAS.
URL: https://openreview.net/forum?id=f9qjl5xCVW
---
Title: Mutual Information Collapse Explains Disentanglement Failure in $\beta$-VAEs
Abstract: The $\beta$-VAE is a foundational framework for unsupervised disentanglement, utilizing a regularization parameter $\beta$ to balance latent factorization against reconstruction fidelity. However, disentanglement performance often exhibits a non-monotonic dependence on $\beta$: standard metrics, such as MIG and SAP, typically peak at intermediate values and deteriorate under stronger regularization. We characterize this phenomenon as informational collapse---an information-theoretic failure in which excessive regularization drives the mutual information between latent variables and ground-truth generative factors toward zero. By analyzing the stationarity conditions in a linear-Gaussian setting, we prove that for $\beta > 1$, alternating optimization induces a spectral contraction of the encoder gain. This leads to an exponential decay of its spectral norm and the subsequent vanishing of latent--factor mutual information. To mitigate this failure mode, we investigate the $\lambda\beta$-VAE, which augments the objective with an auxiliary $L_2$ reconstruction penalty. Our analysis demonstrates that this term modifies the encoder stationarity conditions to counteract spectral decay, thereby stabilizing information flow within the latent representation. Extensive experiments on dSprites, Shapes3D, and MPI3D-real confirm that $\lambda > 0$ enhances the stability of disentanglement and preserves latent informativeness across a significantly broader range of $\beta$, providing a principled justification for dual-parameter regularization in variational inference.
URL: https://openreview.net/forum?id=P0E4LsRvGY
---
Title: Structural Knowledge Informed Continual Multivariate Time Series Forecasting
Abstract: Recent studies in multivariate time series (MTS) forecasting reveal that explicitly modeling the hidden dependencies among different time series can yield promising forecasting performance and reliable explanations. However, modeling variable dependencies remains underexplored when MTS is continuously collected under different regimes (stages). Due to the potential distribution and dependency disparities, the underlying model may encounter the catastrophic forgetting problem, i.e., it is challenging to memorize and infer different types of variable dependencies across different regimes while maintaining forecasting performance. To address this issue, we propose a novel Structural Knowledge Informed Continual Learning (SKI-CL) framework to perform MTS forecasting within a continual learning paradigm, which leverages structural knowledge to steer the forecasting model toward identifying and adapting to different regimes, and selects representative MTS samples from each regime for memory replay. Specifically, we develop a forecasting model based on graph structure learning, where a consistency regularization scheme is imposed between the learned variable dependencies and the structural knowledge (\textit{e.g.}, physical constraints, domain knowledge, feature similarity, which provides regime characterization) while optimizing the forecasting objective over the MTS data. As such, MTS representations learned in each regime are associated with distinct structural knowledge, which helps the model memorize a variety of conceivable scenarios and results in accurate forecasts in the continual learning context. Meanwhile, we develop a representation-matching memory replay scheme that maximizes the temporal coverage of MTS data to efficiently preserve the underlying temporal dynamics and dependency structures of each regime. Thorough empirical studies on synthetic and real-world benchmarks validate SKI-CL's efficacy and advantages over the state-of-the-art for continual MTS forecasting tasks. SKI-CL can also infer faithful dependency structures that closely align with structural knowledge in the test stage.
URL: https://openreview.net/forum?id=7tVdc31VmA
---
Title: Deep Discrete-Time Survival Analysis with Guaranteed Monotonicity
Abstract: Discrete-time neural survival models trained with binary cross-entropy are attractive due
to their simplicity. However, they can produce invalid patient-specific survival curves that
increase over time when survival probabilities at different time points are learned without
structural constraints. We propose Kaplan–Meier Net (KMNet), a discrete-time neural sur
vival model that predicts interval-wise conditional survival probabilities and constructs the
survival curve through a Kaplan–Meier style product, guaranteeing non-increasing survival
predictions by design. KMNet is trained with a censoring-aware weighted binary cross
entropy objective and is further augmented with a smooth ranking term that compares
individuals using the conditional survival probability at the event interval of the anchor
observation, which differs from the global ranking losses used in existing deep survival mod
els. We evaluate KMNet on eight benchmark datasets and compare it with seven strong
neural baselines. Across datasets, KMNet achieves the best overall average rank in both
time-dependent concordance and integrated brier score, while consistently producing valid
survival curves.
URL: https://openreview.net/forum?id=VXIk9iY10Q
---
Title: Efficient Multi-Scale Deformable Attention on GPUs
Abstract: Multi-scale deformable attention (MSDA) is a core operator in DETR-family vision transformers
whose scattered bilinear sampling pattern defeats the tile-based strategies on which
FlashAttention-style kernels depend. We present a diagnostic study of GPU kernel optimization
for MSDA on NVIDIA A100 (SM 8.0) and H100 (SM 9.0), identifying two failure modes of
conventional heuristics and a root cause that is both hardware- and compiler-gated.
Dispatch-order reordering does not pay: seven query orders (linear, Morton Z-order, random,
scanline, Hilbert, centroid, and a clustering-and-packing analogue) produce within-±2%
forward latency at K=4, L=4 because L2 locality is tile-set by the query-block kernel rather
than by the dispatch order. Throughput proxies mislead: an 85%-occupancy point-parallel
tiling delivers only 5.1% of A100 peak bandwidth, while a 17%-occupancy query-block tiling
delivers 36% and runs 7.4× faster. The backward-pass bottleneck is scattered-gradient atomic
contention: at BF16, the backward kernel attains 2.4% of A100 peak bandwidth versus 21.3% on
H100. The gap is hardware- and compiler-gated: Ampere has no native BF16 atomic instruction
(forcing a 32-bit compare-and-swap emulation), and on H100 the standard CUDA atomic
still lowers to that emulation while a relaxed-ordering variant reaches Hopper's native
reduction primitive; an FP32-accumulator variant closes the A100 gap entirely. The resulting
backends deliver 2.4–14× forward speedup and up to 88% peak VRAM reduction over the reference
implementation at numerical parity.
URL: https://openreview.net/forum?id=Q4jZ7zKNKx
---
Title: FedSCM: Federated Domain Adaptation via Sparse Consensus Matching
Abstract: We address the problem of client selection for source-free domain adaptation in heterogeneous federated learning (FL). In this setting, a central server possesses an unlabeled target-domain dataset and aims to learn a model by leveraging locally trained models from a large pool of K non-IID clients. Crucially, only a small subset of clients have data distributions that meaningfully align with the server’s target domain, making effective client selection essential. However, due to strict privacy constraints, the server cannot access raw client data, client-side statistics, or labels for its own dataset—it can only evaluate the initially-trained client models on unlabeled target samples. To tackle this challenge, we propose Federated Sparse Consensus Matching (FedSCM), a principled, optimization-based method for label-free and data-free client selection. FedSCM selects clients whose predictions are both confident and mutually consistent by solving an entropy-regularized sparse optimization problem over client weights. We prove that FedSCM always yields a sparse solution, and under a novel Dirichlet-based expertise model, it identifies the correct subset of relevant clients with high probability, provided n ≥ O(logK) target samples. We further establish local and global convergence guarantees under mild conditions. Extensive experiments on CIFAR-10, CIFAR-100, and SVHN demonstrate that FedSCM consistently outperforms existing approaches to federated domain adaptation, while significantly reducing both communication and computation overhead. Our framework offers a general and theoretically grounded approach to selective model aggregation under extreme data heterogeneity and limited supervision.
URL: https://openreview.net/forum?id=MXKRLoCi1o
---
Title: POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
Abstract: Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators---including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.
URL: https://openreview.net/forum?id=KUyeGxlirZ
---
Title: A friendly introduction to triangular transport
Abstract: Decision making under uncertainty is a cross-cutting challenge in science and engineering. Most approaches to this challenge employ probabilistic representations of uncertainty. In complicated systems accessible only via data or black-box models, however, these representations are rarely known. We discuss how to characterize and manipulate such representations using \textit{triangular transport maps}, which approximate any complex probability distribution as a transformation of a simple, well-understood distribution. The particular structure of triangular transport guarantees many desirable mathematical and computational properties that translate well into solving practical problems. Triangular maps are actively used for density estimation, (conditional) generative modelling, Bayesian inference, data assimilation, optimal experimental design, and related tasks. While there is ample literature on the development and theory of triangular transport methods, this manuscript provides a detailed introduction for scientists interested in employing measure transport without assuming a formal mathematical background. We build intuition for the key foundations of triangular transport, discuss many aspects of its practical implementation, and outline the frontiers of this field.
URL: https://openreview.net/forum?id=PL0vAVlmAH
---
Title: MonoQPM: Splitting Features into Concepts for Inherent Interpretability and Predictive Performance
Abstract: Alongside superposition, polysemanticity is one of the primary obstacles to achieving interpretable deep neural networks. While these phenomena are typically intrinsically linked, the Quadratic Programming Enhanced Model (QPM) naturally decouples them. By representing classes with a sparse binary assignment of very few features, QPM prevents superposition by design on its final features but still exhibits polysemanticity. However, measuring polysemanticity is an open problem. This work proposes a utility-focused approach to measuring polysemanticity by quantifying the decrease in interference-induced activation errors, which yields the practical utility of tighter prediction sets using Conformal Prediction (CP). We explicitly disentangle the polysemantic features of QPM into monosemantic concepts to create the Monosemantic QPM (MonoQPM). Because its features are disentangled, MonoQPM acts as a significantly more efficient Conformal Predictor. Additionally, we introduce CUBCars, a dataset providing ground truth information about shared concepts. Using this and other datasets, we demonstrate that polysemanticity emerges in QPM across all tested architectures, but is effectively alleviated by MonoQPM. For instance, MonoQPM guarantees 88% coverage using Adaptive Prediction Sets on ImageNet with just 66% of the frozen QPM’s set size.
URL: https://openreview.net/forum?id=8bQNRlFQhZ
---
Title: Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
Abstract: As large language models (LLMs) continue to advance, improving them solely through human supervision is becoming increasingly costly and limited in scalability. As models approach human-level capabilities in certain domains, human feedback may no longer provide sufficiently informative signals for further improvement. At the same time, the growing ability of models to make autonomous decisions and execute complex actions naturally enables abstractions in which components of the model development process can be progressively automated. Together, these challenges and opportunities have driven increasing interest in self-improvement, where models autonomously generate data, evaluate outputs, and iteratively refine their own capabilities. In this paper, we present a system-level perspective on self-improving language models and introduce a unified framework that organizes existing techniques. We conceptualize the self-improvement system as a closed-loop lifecycle, consisting of four tightly coupled processes: data acquisition, data selection, model optimization, and inference refinement, along with an autonomous evaluation layer throughout the process. Within this framework, the model itself plays a central role in driving each stage: collecting or generating data, selecting informative signals, updating its parameters, and refining outputs, while the autonomous evaluation layer continuously monitors progress and guides the improvement cycle across stages. Following this lifecycle perspective, we systematically review and analyze representative methods for each component from a technical standpoint. We further discuss current limitations and outline our vision for future research toward fully self-improving LLMs.
URL: https://openreview.net/forum?id=7Sc51w6mIC
---
Title: The Learnability of an Unknown System From Input-Output Data
Abstract: Artificial intelligence is transforming how scientists build models, test ideas, and make predictions. Beneath these advances lies a fundamental question: from the data we collect, when is learning possible in principle, and when is it not? We cast inference as a game between a learner, who holds a pool of candidate models, and an adversary, who holds the unknown ground-truth system. The learner observes the system and selects a candidate model to achieve one of three goals: identify the system, predict its output, or verify an input. We analyze 81 cases that arise by varying these goals, the available observations from both ground truth and candidates, and whether systems are single- or multi-valued. For each case, we prove whether universal solvability is possible. By clarifying which observations make success achievable in principle, our results explain why certain data-driven problems are solvable and guide how to collect data and evaluate models.
URL: https://openreview.net/forum?id=vAhvicqZLG
---
Title: MR.AsyncFL: Staleness-Aware Aggregation for Asynchronous Federated Learning via Model Replacement
Abstract: Asynchronous federated learning (AFL) has become increasingly popular and crucial for scalable, privacy-preserving machine learning across diverse and distributed edge devices. However, a fundamental challenge in FL is the staleness of client updates, which can degrade global model accuracy and slow down convergence as clients operate and communicate independently. Existing AFL methods typically address this issue by down-weighting stale updates, but outdated client information may still persist in the global model over time. In this paper, we propose MR.AsyncFL, a fully AFL framework based on model replacement. Upon receiving a new local model from a client, the server replaces that client’s previously cached contribution in the global model with the updated one, preserving the invariant that the global model remains a convex combination of the most recent available client models. To support this mechanism, we propose a recursive weight update scheme that preserves normalization in a lightweight and fully asynchronous manner. We further provide a convergence analysis for MR.AsyncFL under bounded staleness and client participation assumptions, and derive an $\mathcal{O}(T^{-1/4})$ convergence rate under a specific parameter scaling. Experiments on CIFAR-10 and CIFAR-100 under both IID and non-IID settings, with and without staleness thresholds, show that MR.AsyncFL consistently outperforms representative asynchronous baselines, such as FedAsync, TWAFL, and Rolling FedAvg, while maintaining strong robustness under severe staleness and system heterogeneity.
URL: https://openreview.net/forum?id=iwdvaRbrnU
---
Title: NeMoS: Nearest Neighbors Bandit meets Active Learning for Online Model Selection
Abstract: The proliferation of open-platform text-to-image generative models has made prompt-wise model selection critical to maximize generation quality and semantic alignment. However, current strategies, such as contextual bandits, often converge slowly and fail to exploit the semantic relationships across prompts. To bridge this gap, we propose NeMoS, a non-parametric bandit framework that couples nearest neighbor reward estimation with a budget-constrained active learning strategy.
Specifically, our approach operates in the prompt embedding space and estimates the reward of incoming prompts based on feedback from their nearest neighbors. By limiting ground-truth queries to ambiguous ``near-tie'' scenarios, NeMoS resolves uncertainty efficiently and accelerates convergence. We prove that this active mechanism yields a poly-logarithmic regret bound, marking a significant theoretical improvement over its passive version.
Extensive experiments on four datasets with six image generative models show that NeMoS reduces regret by up to 60\% compared to state-of-the-art baselines, while being robust to model addition or removal.
\textit{We provide experimental code in the supplementary material.}
URL: https://openreview.net/forum?id=CSjewjplO1
---
Title: SEER: Label-Structured Modality Routing for Multimodal Sentiment Analysis and Intent Recognition
Abstract: Multimodal sentiment analysis and intent recognition require models to combine textual, acoustic, and visual evidence whose reliability varies across utterances. Although adaptive fusion can address this variability by assigning sample-specific modality weights, many existing routing mechanisms estimate confidence from raw feature statistics, generic similarity measures, or prototype assignments that are only indirectly related to the downstream label structure. This can make routing sensitive to modality style or feature magnitude rather than to the evidence most relevant for sentiment or intent prediction. To study this issue, we introduce a staged routing framework. First, Emotion-Aware Modality Calibration (EAMC) serves as an encoded-space routing baseline that estimates modality reliability after semantic encoding while keeping the backbone and weighted-sum fusion rule fixed. Building on this baseline, we propose Structured Evidence Estimation and Routing (SEER), which incorporates label structure into the representation space used for confidence estimation. SEER-L0 adds label-aware contrastive supervision to organize modality representations according to task labels, while SEER-L1 estimates modality confidence by matching modality-adapted representations to shared label-structured anchors. We also evaluate SEER-L2, a prototype-guided temporal evidence extraction extension. Experiments on aligned CMU-MOSI, aligned CMU-MOSEI, and MIntRec under a multi-run evaluation protocol show that SEER-L1 provides the most consistent improvement over EAMC on the primary metrics, namely binary F1 for sentiment analysis and Weighted F1 for intent recognition. In contrast, SEER-L2 does not improve performance in the current setting. These results suggest that, for the evaluated benchmarks, adaptive multimodal routing benefits more from label-structured confidence estimation than from adding temporal pooling complexity.
URL: https://openreview.net/forum?id=UjVqsEbVzs
---
Title: The Encoding-Behavior Dissociation: How Distributed Safety Representations Yield Single-Direction Vulnerabilities in Vision-Language Model
Abstract: Safety refusal in large language models (LLMs) has been shown to be mediated by a single linear direction in residual-stream activation space. The safety refusal geometry of vision-language models (VLMs), however, under the Linear Representation Hypothesis, is scarcely investigated. Unlike LLMs, they introduce a dedicated visual encoder and cross-modal fusion, which greatly expands the representation space as compared to textual modalities only. The understanding of safety behavior in VLM representation spaces has direct implications for multimodal safety alignment. We conduct an investigation of refusal geometry in VLMs spanning multiple models and experiments, and uncover a fundamental dissociation between how safety is encoded and how it is acted upon. At the encoding level, VLM safety representations are markedly higher-dimensional than those of text-only LLMs: in pretrained PaliGemma, separating harmful from benign inputs requires ~50 PCA components (versus one for text-only LLMs), with signal distributed across all token positions and all attention heads, and is robust to iterative direction ablation, a sharp contrast to Gemma-IT, whose safety separation collapses after ablating ~11 directions. At the behavioral level, instruction-tuned VLMs (Qwen-base, Qwen-small, LLaVA, Qwen-tiny, Phi-Vision) are nevertheless governed by a threshold along the single dominant Difference-in-Means (DIM) direction: activation steering along this direction drives harmful-prompt refusal effectively and induces refusal on benign prompts from 52% to 98%. Exploiting this geometry, we derive per-image and universal PGD attacks that achieve 98.4% and 96.9% refusal bypass, respectively, exceeding prior white-box and transfer-based baselines, and remain effective under ℓ∞ imperceptibility constraints (ε = 8/255) on three of four tested architectures. Larger models require stronger perturbations but are not protected, while cross-model transfer is weak (≤11%), indicating that safety geometry is model-specific. This encoding–behavior dissociation, a high-dimensional safety sensor wired to a one-dimensional refusal gate, which exposes a structural limitation of current VLM alignment and motivates a shift from encoding-centric toward mechanism-centric safety design.
URL: https://openreview.net/forum?id=GGFjPyO3mZ
---
Title: Interpreting the Synchronization Gap: The Hidden Mechanism Inside Diffusion Transformers
Abstract: Recent theoretical models of diffusion processes, conceptualized as coupled Ornstein-Uhlenbeck systems, predict a hierarchy of interaction timescales, and consequently, the existence of a synchronization gap between modes that commit at different stages of the reverse process. However, because these predictions rely on continuous time and analytically tractable score functions, it remains unclear how this phenomenology manifests in the deep, discrete architectures deployed in practice. In this work, we investigate how the synchronization gap is mechanistically realized within pretrained Diffusion Transformers (DiTs). We construct an explicit architectural realization of replica coupling by embedding two generative trajectories into a joint token sequence, modulated by a symmetric cross attention gate with variable coupling strength $g$. Through a linearized analysis of the attention difference, we show that the replica interaction decomposes mechanistically. We empirically validate our theoretical framework on a pretrained DiT-XL/2 model by tracking commitment and per layer internal mode energies. Our results reveal that: (1) the synchronization gap is an intrinsic architectural property of DiTs that persists even when external coupling is turned off; (2) as predicted by our spatial routing bounds, the gap completely collapses under strong coupling $g\rightarrow1$; (3) the gap is strictly depth localized, emerging sharply only within the final layers of the Transformer; and (4) global, low frequency structures consistently commit before local, high frequency details. Ultimately, our findings provide a mechanistic interpretation of how Diffusion Transformers resolve generative ambiguity, isolating speciation transitions to the terminal layers of the network.
URL: https://openreview.net/forum?id=qMgUhYYBJW
---
Title: Memory Makes The Poisons: Understanding and Mitigating Data Poisoning in LVLMs
Abstract: Large Vision-Language Models (LVLMs) are increasingly deployed in high-stakes applications, yet their training-time security remains poorly understood. As a prominent data poisoning attack specifically designed for LVLMs, ShadowCast achieves significant success in inducing targeted hallucinations, posing a serious threat to LVLM safety. ShadowCast’s success has been attributed to injected visual perturbations. Consequently, subsequent defenses have focused on visual purification; however, their effectiveness remains limited. In this paper, we present a re-analysis of the ShadowCast mechanism. Our key finding is that memorization during LVLM fine-tuning is an overlooked but major contributor to attack success, and it dominates at higher poison ratios. This factor has been largely overlooked in previous work. We further show that multimodal training exacerbates this vulnerability compared to unimodal settings. This insight fundamentally reframes both the threat model and the defense objective: if memorization is a major contributor, purification-only defenses are inherently insufficient in multimodal regimes. Motivated by this perspective, we propose RejectShield, a rejection-based defense that filters suspicious training samples prior to fine-tuning. Across extensive evaluations spanning 4 attack goals, 3 LVLMs, black-box and white-box attack settings, and 3 poisonings, RejectShield reduces the attack success rate by up to 99% while largely preserving model utility, significantly advancing defense effectiveness against LVLM poisoning. Code and additional results are provided in the Supp.
URL: https://openreview.net/forum?id=bnWb25IRxx
---
Title: HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning
Abstract: Existing LLM-based automatic test generation methods mainly produce input and expected output pairs to categorize the intended behavior of correct programs. Although straightforward, these methods have limited diversity in generated tests and cannot provide enough debugging information.
We propose HarnessLLM, a two-stage training pipeline that enables LLMs to write harness code for testing. Particularly, LLMs generate code that synthesizes inputs and validates the observed outputs, allowing complex test cases and flexible output validation such as invariant checking.
To achieve this, we train LLMs with SFT followed by RLVR with a customized reward design.
Experiments show that HarnessLLM outperforms input-output-based testing in bug finding and testing strategy diversity.
HarnessLLM further benefits the code generation performance through test-time scaling with our generated test cases as inference-phase validation.
URL: https://openreview.net/forum?id=dZN9ZWa2kU
---
Title: GLEN-Bench: A Graph-Language based Benchmark for Nutritional Health
Abstract: Nutritional interventions are important for managing chronic health conditions, but current computational methods provide limited support for personalized dietary guidance. We identify three key gaps: (1) dietary pattern studies often ignore real-world constraints such as socioeconomic status, comorbidities, and limited food access; (2) recommendation systems rarely explain why a particular food helps a given patient; and (3) no unified benchmark evaluates methods across the connected tasks needed for nutritional interventions. We introduce GLEN-Bench, the first comprehensive graph-language based benchmark for nutritional health assessment. We combine NHANES health records, FNDDS food composition data, and USDA food-access metrics to build a knowledge graph that links demographics, health conditions, dietary behaviors, poverty-related constraints, and nutrient needs. We test the benchmark using opioid use disorder, where models must detect subtle nutritional differences across disease stages. GLEN-Bench includes three linked tasks: risk detection identifies at-risk individuals from dietary and socioeconomic patterns; recommendation suggests personalized foods that meet clinical needs within resource constraints; and question answering provides graph-grounded, natural-language explanations to facilitate comprehension. We evaluate these graph-language approaches, including graph neural networks, large language models, and hybrid architectures, to establish solid baselines and identify practical design choices. Our analysis identifies clear dietary patterns linked to health risks, providing insights that can guide practical interventions.
URL: https://openreview.net/forum?id=XMUbOKWsem
---
Title: Global Motif Embedding Meet Few-Shot Graph Learning
Abstract: Graph Neural Networks (GNNs) have shown strong performance in node classification tasks. However, in real-world scenarios, only a limited number of nodes are often labeled, leading to the few-shot node classification problem, which is a significant challenge for GNNs. Most existing research focuses on designing new models to adapt to this setting, but often overlooks structural information in the graph data, such as motif patterns, which can provide crucial cues for learning from a few examples. In this paper, we propose a novel framework that integrates motif representations into graph few-shot learning models. Specifically, we extract unique motif representations from the graph and introduce them as virtual nodes. To capture richer structural patterns, we further enhance motif extraction by adding cluster labels based on node similarity, thereby incorporating both structural and feature information. Additionally, we assign TF-IDF scores as edge weights between virtual motif nodes and original nodes to quantify the importance of their connections. Experimental results demonstrate that our approach consistently improves the performance of various graph few-shot learning methods.
URL: https://openreview.net/forum?id=M3uAgivbay
---
Title: Heavy-Tailed Class-Conditional Priors for Long-Tailed Generative Modeling
Abstract: Variational Autoencoders (VAEs) with global priors trained under an imbalanced empirical class distribution can lead to underrepresentation of tail classes in the latent space. While $t^3$VAE improves robustness via heavy-tailed Student's $t$-distribution priors, its single global prior still allocates mass proportionally to class frequency. We address this latent geometric bias by introducing C-$t^3$VAE, which assigns a per-class Student's $t$ joint prior over latent and output variables. This design promotes uniform prior mass across class-conditioned components. To optimize our model we derive a closed-form objective from the $\gamma$-power divergence, and we introduce an equal-weight latent mixture for class-balanced generation. On SVHN-LT, CIFAR100-LT, and CelebA datasets, C-$t^3$VAE consistently attains lower FID scores than $t^3$VAE and Gaussian-based VAE baselines under severe class imbalance while remaining competitive in balanced or mildly imbalanced settings. In per-class F1 evaluations, our model outperforms the conditional Gaussian VAE across highly imbalanced settings. Moreover, we identify the mild imbalance threshold $\rho < 5$, for which Gaussian-based models remain competitive. However, for $\rho \geq 5$ our approach yields improved class-balanced generation and mode coverage.
URL: https://openreview.net/forum?id=iTngjtzYzk
---
Title: CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding
Abstract: Although Multimodal Large Language Models (MLLMs) have shown remarkable potential in Visual Document Retrieval (VDR) through generating high-quality multi-vector embeddings, the substantial storage overhead caused by representing a page with thousands of visual tokens limits their practicality in real-world applications. To address this challenge, we propose an auto-regressive generation approach, CausalEmbed, for constructing multi-vector embeddings. By incorporating iterative margin loss during contrastive training, CausalEmbed encourages the embedding models to learn compact and well-structured representations. Our method enables efficient VDR tasks using only dozens of visual tokens, achieving a 30–155$\times$ reduction in token count while maintaining highly competitive performance across various backbones and benchmarks. Theoretical analysis and empirical results demonstrate the unique advantages of auto-regressive embedding generation in terms of training efficiency and scalability at test time. As a result, CausalEmbed introduces a flexible test-time scaling strategy for multi-vector VDR representations and sheds light on the generative paradigm within multimodal document retrieval.
URL: https://openreview.net/forum?id=mZXoOgBxX2
---
Title: Neural Operators for Multi-Task Control and Adaptation
Abstract: Neural operator methods have emerged as powerful tools for learning mappings between infinite-dimensional function spaces, yet their potential in optimal control remains largely unexplored. We focus on multi-task control problems, whose solution is a mapping from task description (e.g., cost or dynamics functions) to optimal control law (e.g., feedback policy). We approximate these solution operators using a permutation-invariant neural operator architecture. Across a range of parametric optimal control environments and a locomotion benchmark, a single operator trained via behavioral cloning accurately approximates the solution operator and generalizes to unseen tasks, out-of-distribution settings, and varying amounts of task observations. We further show that the branch—trunk structure of our neural operator architecture enables efficient and flexible adaptation to new tasks. We develop structured adaptation strategies ranging from lightweight updates to full-network fine-tuning, achieving strong performance across different data and compute settings. Finally, we introduce meta-trained operator variants that optimize the initialization for few-shot adaptation. These methods enable rapid task adaptation with limited data and consistently outperform a popular meta-learning baseline. Together, our results demonstrate that neural operators provide a unified and efficient framework for multi-task control and adaptation.
URL: https://openreview.net/forum?id=jciOb0z5Wm
---
Title: One Spike Decision Reinforcement Learning Framework for Dynamic Environments
Abstract: Deep reinforcement learning (DRL) agents face challenges in natural environments that are similar to those encountered by biological organisms: they must make actions that are both accurate and timely in response to dynamic, non-stationary conditions. However, achieving such behavior incurs significant computational overhead, limiting the scalability of DRL in real-world applications. Spiking Neural Networks (SNNs), as the most biologically plausible computational model of neurons, offer a promising energy-efficient alternative for reinforcement learning due to their low computational cost. Existing SNN-based methods, however, often rely on multiple simulation time steps to approximate analog activations, which compromises their low-latency and low-power advantages. To address this, we propose a novel DRL framework based on one-spike firing decision (OSFD), which redefines the use of SNNs in DRL. In OSFD, each decision step
triggers only a single spike to produce an action, while the residual membrane potential is incrementally accumulated across steps. In addition, we introduce Bayesian variational inference to dynamically regulate the contribution of residual potentials based on state information gain, thereby optimizing policy learning. Experimental results demonstrate that our method not only surpasses conventional artificial neural network (ANN)-based frameworks in performance but also significantly reduces computational cost.
URL: https://openreview.net/forum?id=3lnHCkW5eU
---
Title: ChatAni: Language-Driven Multi-Actor Animation Generation in Street Scenes
Abstract: Generating interactive and realistic traffic participant animations from instructions is essential for autonomous driving simulations. Existing methods, however, fail to comprehensively address the diverse participants and their dynamic interactions in street scenes. In this paper, we present ChatAni, the first system capable of generating interactive, realistic, and controllable multi-actor animations based on language instructions. To produce fine-grained, realistic animations, ChatAni introduces two novel animators: PedAnimator, a unified multi-task animator that generates interaction-aware pedestrian animations under varying task plans, and VehAnimator, a kinematics-based policy that generates physically plausible vehicle animations. For precise control through complex language, ChatAni employs a multi-LLM-agent role-playing approach, using natural language to plan the trajectories and behaviors of different participants. Extensive experiments demonstrate that ChatAni can generate realistic street scenes with interacting vehicles and pedestrians, benefiting tasks like prediction and understanding. All related code, data, and checkpoints will be open-sourced.
URL: https://openreview.net/forum?id=zxQ4pjTiG2
---
Title: MDP Planning as Policy Inference
Abstract: We formulate episodic Markov decision process (MDP) planning as Bayesian inference over policies. The primary contribution is conceptual: the policy itself is treated as the latent variable, and expected return defines an unnormalized posterior density over policies. This preserves the standard expected-return objective, in contrast to trajectory-centric planning-as-inference formulations that introduce auxiliary optimality variables and to entropy-regularized policy optimization methods that solve a different objective.
In the exact formulation, the posterior over deterministic policies induces what we define here as an optimal stochastic policy under preference uncertainty, namely the stochastic policy induced by that posterior. For discrete MDPs with stochastic transitions, we study variational sequential Monte Carlo (VSMC) as one approximate inference method for this posterior, introducing policy consistency under state revisitation and coupled transition randomness across particles.
Experiments on grid worlds, Blackjack, Triangle Tireworld, and Academic Advising examine the consequences of inference over policies and compare its induced behavior with entropy-regularized policy optimization. The results support the view that MDP planning can be naturally cast as Bayesian inference over policies.
URL: https://openreview.net/forum?id=6LoJnJKcqf
---
Title: From Queries to Clones: A Systematic Study of Encoder Stealing Attacks
Abstract: Model stealing is a growing threat for online ML services. Encoder models are more vulnerable to stealing than classifiers because their high-dimensional embeddings reveal substantially richer information than class logits. This risk is amplified by Encoder-as-a-Service platforms that expose foundation encoders. Prior work mostly studied encoder stealing attacks in isolation with inconsistent setups, so practical trade-offs and failure modes across attacks remain unclear. We present a comprehensive benchmark and comparative study of a representative set of encoder stealing attacks on two widely used encoders, CLIP and DINO. We consider three threat scenarios with increasing realism: (i) the attacker has access to the victim’s training data, (ii) the attacker knows the victim’s training distribution and uses disjoint data, and (iii) the attacker has no reliable knowledge of the victim’s training distribution. We also evaluate a novel setting where the attacker uses queries from multiple datasets to steal a more generalizable surrogate. Finally, we vary data and query budgets, surrogate capacity, and resource constraints to understand practical attack scenarios. Across our settings, we observe that high utility on the stealing distribution does not necessarily translate to high utility on the victim’s training distribution under shift. We find that contrastive objectives with strong augmentations are the most reliable, conventional methods can be brittle, and prototype alignment is query-efficient but shifts cost to local compute and memory. In our experiments, mixed-source queries reveal a data density-diversity trade-off, and DINO is consistently easier to steal than CLIP, with cross-modal text guidance partially narrowing the gap. Overall, our results map practical attack operating points and highlight vulnerabilities relevant to the foundation model era.
URL: https://openreview.net/forum?id=7Kr7IDbYIi
---
Title: Contrastive Counterfactual Generation for Imperceptible Adversarial Attack
Abstract: Imperceptible adversarial attacks aim to mislead deep neural networks by adding signal-domain perturbations that induce misclassification while remaining visually indistinguishable from the original signal. Existing methods rely on untargeted loss maximization, producing perturbations poorly aligned with decision boundaries and providing limited control over locality and perceptual cost. To address these limitations, we propose $\textbf{Contrastive Counterfactual Generation}$ ($\texttt{CoCoGen}$), a cross-domain adversarial attack framework that formulates perturbation synthesis as a constrained optimisation problem. \texttt{CoCoGen} explicitly targets the nearest decision boundary by minimising the $\textit{contrastive counterfactual margin}$ under a strict signal-energy budget. Perturbations are localised via gradient-based Top-$k$ spatial projection and confined to the high-frequency subspace using a Fourier-domain projection operator, leveraging reduced human sensitivity to high spatial frequencies. The objective is optimised using masked gradient descent with
momentum, while an adaptive sparsity grid search identifies minimal feasible signal support. Experiments across multiple architectures show that $\texttt{CoCoGen}$ achieves $100\%$ Attack Success Rate (vs. $80-100\%$ for prior methods, with most below $99\%$) while maintaining a MUSIQ score of $61-63$ (vs. $36-55$), outperforming prior methods in both attack efficacy and visual quality.
URL: https://openreview.net/forum?id=dnme31GvOd
---
Title: Random Logit Scaling: Defending Deep Neural Networks Against Black-Box Score-Based Adversarial Example Attacks
Abstract: Machine learning models are increasingly adapted in various domains. However, adversarial examples pose a significant threat to the reliable deployment of these models. In recent years, some powerful adversarial example attacks have been proposed for the fast and query-efficient generation of adversarial examples, even in black-box scenarios, highlighting the need for scalable, low-cost, and powerful defenses. In this work, we present two contributions to the domain of black-box adversarial example attacks and defenses. First, we propose Random Logit Scaling (RLS), a randomization-based defense against black-box score-based adversarial example attacks. RLS is a plug-and-play, post-processing defense that can be implemented on top of any existing ML model with minimal effort. The idea behind RLS is to confuse an attacker by outputting falsified scores resulting from randomly scaled logits while maintaining the model accuracy. We show that RLS significantly reduces the success rate of state-of-the-art black-box score-based attacks while preserving the accuracy and minimizing confidence score distortion compared to state-of-the-art randomization-based defenses. Second, we introduce a novel adaptive attack against AAA, a SOTA non-randomized black-box defense against black-box score-based attacks that also modifies output logits to confuse attackers. With our adaptive attack, we demonstrate the vulnerability of AAA, establishing RLS as the effective SOTA defense against black-box score-based attacks.
URL: https://openreview.net/forum?id=CXafPv4aAG
---
Title: Reproducibility study on how to find Spurious Correlations, Shortcut Learning, Clever Hans or Group-Distributional non-robustness and how to fix them
Abstract: Deep Neural Networks (DNNs) are increasingly utilized in high-stakes domains like medical diagnostics and autonomous driving where model reliability is critical. However, the research landscape for ensuring this reliability is terminologically fractured across communities that pursue the same goal of ensuring models rely on causally relevant features rather than confounding signals. While frameworks such as distributionally robust optimization (DRO), invariant risk minimization (IRM), shortcut learning, simplicity bias, and the Clever Hans effect all address model failure due to spurious correlations, researchers typically only reference work within their own domains. This reproducibility study unifies these perspectives through a comparative analysis of correction methods under challenging constraints like limited data availability and severe subgroup imbalance. We evaluate recently proposed correction methods based on explainable artificial intelligence (XAI) techniques alongside popular non-XAI baselines using both synthetic and real-world datasets. Findings show that XAI-based methods generally outperform non-XAI approaches, with Counterfactual Knowledge Distillation (CFKD) proving most consistently effective at improving generalization. Our experiments also reveal that the practical application of many methods is hindered by a dependency on group labels, as manual annotation is often infeasible and automated tools like Spectral Relevance Analysis (SpRAy) struggle with complex features and severe imbalance. Furthermore, the scarcity of minority group samples in validation sets renders model selection and hyperparameter tuning unreliable, posing a significant obstacle to the deployment of robust and trustworthy models in safety-critical areas.
URL: https://openreview.net/forum?id=FyFTTV3dTC
---
Title: Scaling Laws: A Model-Based Optimization Perspective
Abstract: Scaling laws have become indispensable for guiding the pre-training of large language models, enabling optimal decision-making---such as determining the scale of model and data---under a fixed compute budget. Standard practice involves fitting parametric functions~(predominantly power laws) from small-scale experiments, which allows researchers to extrapolate trends and predict compute-optimal configurations at larger scales. This neural scaling paradigm is fundamentally a specialized instantiation of \textit{Model-Based Optimization} (MBO): constructing a surrogate model (the scaling law) from experimental data to predict validation metrics, and subsequently optimizing pre-training configurations as design variables against this surrogate. Despite this equivalence, existing literature primarily focuses on neural scaling priors while neglecting the broader MBO perspective. In this position paper, we bridge this gap by formally mapping the neural scaling paradigm to the three stages of MBO: \textit{design space}, \textit{surrogate modeling}, and \textit{guided optimization}, and distinguish the three unique characteristics---low-dimensional spaces, strong power-law priors, and strict compute constraints---that separate it from standard MBO problems. Furthermore, we systematically partition the design space into three subspaces: \textit{model}, \textit{data}, and \textit{hyperparameters}. Crucially, we formalize their relationship as a bi-level optimization problem, wherein hyperparameters are optimized at the lower level to ensure convergence for specific model and data configurations. To demonstrate the practical utility of adopting MBO techniques, we focus on the surrogate modeling stage and provide an illustrative proof-of-concept by applying \textit{autofocus}---an established MBO technique---to mitigate extrapolation-induced covariate shifts. Finally, we conclude by providing a principled roadmap for future research, highlighting uncertainty quantification and multi-objective optimization.
URL: https://openreview.net/forum?id=w982lDpJrT
---
Title: Do Instance Priors Help Weakly Supervised Semantic Segmentation?
Abstract: Semantic segmentation requires dense pixel-level annotations, which are costly and time-consuming to acquire. To address this, we present SeSAM, a framework that uses a foundational segmentation model, i.e. Segment Anything Model (SAM), with weak labels, including coarse masks, scribbles, and points. SAM, originally designed for instance-based segmentation, cannot be directly used for semantic segmentation tasks. In this work, we identify specific challenges faced by SAM and determine appropriate components to adapt it for class-based segmentation using weak labels. Specifically, SeSAM decomposes class masks into connected components, samples point prompts along object skeletons, selects SAM masks using weak-label coverage, and iteratively refines labels using pseudo-labels, enabling SAM-generated masks to be effectively used for semantic segmentation. Integrated with a semi-supervised learning framework, SeSAM balances ground-truth labels, SAM-based pseudo-labels, and high-confidence pseudo-labels, significantly improving segmentation quality. Extensive experiments across multiple benchmarks and weak annotation types show that SeSAM consistently outperforms weakly supervised baselines while substantially reducing annotation cost relative to fine supervision.
URL: https://openreview.net/forum?id=4bHFNKe8OU
---