Weekly TMLR digest for Aug 02, 2026
TMLR
New certifications
==================
Survey Certification: What Survives Privatization? A Guide to Structure and Utility in Differentially Private Genome-Wide Association Studies
Shadi Rahimian, Akito Yamamoto, Mario Fritz
https://openreview.net/forum?id=6BWikkmkOH
---
J2C Certification: Random features for Grassmannian kernel approximation with bounded rank-one projections
Remi Delogne, Laurent Jacques
https://openreview.net/forum?id=wq18dZJ2pA
---
Featured Certification: ARC-Encoder: learning compressed text representations for large language models
Hippolyte Pilchen, Edouard Grave, Patrick Perez
https://openreview.net/forum?id=lU1P9dsqfn
---
Survey Certification: Automata Learning from Recurrent Networks: A Critical Synthesis for Verification, Testing, and Interpretability
Jaouhar Slimi, Augustin Lemesle, Tristan Le Gall, Zakaria Chihani
https://openreview.net/forum?id=R52ETbUBVo
---
J2C Certification: On the Convergence Analysis of Muon
Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, Jiawei Zhang
https://openreview.net/forum?id=4nH4CulGaP
---
Accepted papers
===============
Title: Verify What Matters: Budgeted Verification for Tool-Using Agents under Counterfactual Downstream Harm
Authors: Yueh Tang, Justin Zhan
Abstract: Tool-using agents make intermediate decisions that alter persistent state, shape later observations, and create failures that are not equally easy to recover from. When verification is costly, the central question is not whether checking helps in general, but which decisions are worth checking. Policies driven only by local uncertainty capture whether a step may be wrong, but not how much that error would matter if left uncorrected. We formulate budgeted verification for tool-using agents as an intervention-allocation problem in which the value of checking a step depends on verifier efficacy, local error probability, downstream harm, and intervention cost. This factorization makes uncertainty-only routing a restricted approximation appropriate when downstream harm is nearly constant, but incomplete when actions differ in persistence, reversibility, and dependency structure. Empirically, we evaluate this framing in a controlled OpenClaw-facing sandbox, not a full gateway-paired OpenClaw benchmark. On the dependency-sensitive slice with an exact budget of 150 verifications over 300 episodes, uncertainty-only routing achieves 0.813 success, 0.187 irreversible-failure rate, and 3.455 average net reward, while harm-aware routing under the same verify rate achieves 0.817 success, 0.183 irreversible-failure rate, and 3.700 average net reward. The paired bootstrap intervals for these main differences overlap zero, so we do not present this table as a large-margin benchmark result. Instead, the evidence is diagnostic: harm-aware routing selects higher-consequence episodes, H-only is the strongest ablation, and p × H improves over p-only while not outperforming the structural harm cue alone. Additional diagnostics include structural-proxy alignment with exact branch harm, a multi-budget curve at 10%, 25%, 50%, and 75% verification rates, pairwise ablation confidence intervals, cross-slice behavior, verifier-quality sensitivity, a disjoint held-out seed split, a small true held-out procedural variant, a monotone-binned calibration probe, learned PRM-style baselines, and stress tests showing failure modes under harm-cue corruption and correlated verifier failures. Taken together, the results support the paper’s central allocation claim in a bounded form: under scarce verification, local uncertainty alone is an incomplete routing signal, and downstream consequence must be modeled explicitly. The current transparent p × H score should be viewed as an interpretable diagnostic instantiation rather than a universally dominant router.
URL: https://openreview.net/forum?id=nv1jzr0FaZ
---
Title: Cross-Fitted Clipped Covariance Estimation with a Data-Driven Tail-Energy Criterion
Authors: Even He, Zaizai Yan
Abstract: Heavy-tailed data make covariance estimation sensitive to the clipping level: stronger clipping reduces variance but increases bias. We study how to choose this clipping level from the data within a radial clipped covariance family. We propose the quantile tail-energy surrogate (QTES), a fully data-driven rule that combines a cross-fitted variance certificate with a held-out estimate of the tail energy removed by clipping. QTES requires no distributional prior parameters. For Euclidean clipping, the operator-norm bias is bounded by this scalar tail-energy quantity. Under a finite $L_4$ moment condition, together with a mild feasibility condition for the QTES block construction, we prove a uniform finite-sample calibration bound over a positive clipping grid and a finite-sample guarantee for the estimator selected by QTES, relative to the best candidate on that grid under the same variance-plus-bias criterion. Experiments on clean heavy-tailed and contaminated benchmarks show that QTES performs strongly across both regimes and remains competitive with the best reference methods considered.
URL: https://openreview.net/forum?id=MyNXLdRFJ3
---
Title: ViP$^2$-CLIP: Visual-Perception Prompting with Unified Alignment for Zero-Shot Anomaly Detection
Authors: Ziteng Yang, Jingzehua Xu, Cong Liu, Yanshu Li, Zepeng Li, Yeqiang Wang, Xinghui Li
Abstract: Zero-Shot Anomaly Detection (ZSAD) aims to detect anomalies in a target dataset without any training samples, leveraging models trained on auxiliary data. While CLIP offers strong cross-modal representations for ZSAD, its pretraining objective inherently emphasizes global foreground semantics over fine-grained local defects. Consequently, its anomaly localization remains highly sensitive to prompt wording, limiting the effectiveness of existing methods that rely on explicit category labels. To overcome this limitation, we introduce ViP$^{2}$-CLIP, a lightweight CLIP-based ZSAD framework featuring Visual-Perception Prompting (ViP-Prompt) and Unified Text-Patch Alignment (UTPA). ViP-Prompt replaces fixed class-name tokens with image-conditioned cues to adaptively generate fine-grained prompts, obviating the need for manual templates and class-name priors. Furthermore, UTPA enforces a unified text-patch alignment strategy across multiple visual scales, jointly optimizing image-level detection and pixel-level localization. These mechanisms enable the model to precisely localize abnormal regions, exhibiting particular robustness in scenarios with ambiguous or privacy-constrained category labels. Extensive experiments on 14 industrial and medical benchmarks demonstrate that ViP$^{2}$-CLIP achieves competitive performance against existing state-of-the-art approaches, with particular strengths in pixel-level localization capability. Code is available at: https://github.com/kim0806/ViP2-CLIP.
URL: https://openreview.net/forum?id=KCRRuiQSIm
---
Title: RIGID: A Training-Free and Generator-Agnostic Framework for Robust AI-Generated Image Detection
Authors: Zhiyuan He, Pin-Yu Chen, Tsung-Yi Ho
Abstract: The rapid advances in generative AI models have empowered the creation of highly realistic images with arbitrary content, raising concerns about potential misuse and harm, such as Deepfakes. Current research focuses on training detectors using large datasets of generated images. However, these training-based solutions are often computationally expensive and show limited generalization to unseen generated images. In this paper, we propose a training-free method to distinguish between real and AI-generated images. We first observe that real images are more robust to tiny noise perturbations than AI-generated images in the representation space of vision foundation models. Based on this observation, we propose RIGID, a training-free and generator-agnostic method for robust AI-generated image detection. RIGID is a simple yet effective approach that identifies whether an image is AI-generated by comparing the representation similarity between the original and the noise-perturbed counterpart. Our comprehensive evaluation demonstrates RIGID’s practical effectiveness. On the ImageNet and LSUN-Bedroom averages, RIGID improves AP over AEROBLADE by 26.07 and 28.49 points, respectively. Remarkably, RIGID performs comparably to training-based methods, particularly on out-of-domain data. Additionally, RIGID maintains competitive performance across a broad range of generation techniques and demonstrates strong resilience to common image corruptions.
URL: https://openreview.net/forum?id=NBkBI2Zjlm
---
Title: FieldFormer: Locality-Aware Transformers for Spatio-Temporal Modeling on Sparse Sensor Networks
Authors: Ankit Bhardwaj, Ananth Balashankar, Lakshmi Subramanian
Abstract: Spatio-temporal sensor data in real-world systems is often sparse, noisy, and irregular, making it difficult to infer global structure from limited observations. Under extreme sparsity, we run into the limits of identifiability of latent system states, making latent field reconstruction fundamentally underconstrained. In such scenarios, multiple physically plausible fields may remain consistent with the same observations, requiring reconstruction models to rely heavily on inductive biases regarding locality, transport structure, and spatial regularity.
Under such sparsity regimes, reliable reconstruction becomes concentrated around the observational support induced by the sensor network, making sensor-space modeling a more identifiable objective than unconstrained global field recovery. We introduce FieldFormer, a mesh-free transformer architecture designed for locality-aware sensor-space modeling in persistent sensor networks. For each query, FieldFormer aggregates local evidence using a learnable velocity-scaled distance metric that adapts neighborhood geometry to heterogeneous spatio-temporal relationships. Neighborhoods are constructed as fixed maximal sparse contexts over nearby sensors and bounded temporal windows, while learned velocity-scaled offsets modulate token geometry within this context, enabling stable and scalable inference under extreme sparsity. A local transformer encoder integrates neighborhood information, while global consistency is modeled through coordinate-based neural field formulation.
We evaluate FieldFormer across five benchmarks spanning synthetic and real-world spatio-temporal systems, including anisotropic heat diffusion, shallow-water dynamics, atmospheric transport fields, and pollution monitoring datasets. Our results reveal that locality-aware reconstruction provides strong advantages in persistent sparse sensor networks where local domains of dependence remain observed, enabling FieldFormer to consistently outperform state-of-the-art baselines on sensor-space prediction tasks under highly sparse and noisy sensing regimes.
URL: https://openreview.net/forum?id=we4FYGOE2y
---
Title: CDG-MAE: Cross-view Masked Modeling using Diffusion Generated Views
Authors: Varun Belagali, Pierre Marza, Srikar Yellapragada, Zilinghan Li, Tarak Nath Nandi, Ravi K Madduri, Joel Saltz, Stergios Christodoulidis, Maria Vakalopoulou, Dimitris Samaras
Abstract: Cross-view masked autoencoding has emerged as a powerful pretext task for learning dense correspondences, which are essential for applications such as video label propagation. The cross-view pretext task is modeled with a masked autoencoder, where a masked target view is reconstructed from an anchor view. However, acquiring effective training data remains a challenge - collecting diverse video datasets is costly, while simple image crops lack the necessary pose variations, underperforming video-based methods. This paper introduces CDG-MAE, a novel MAE-based self-supervised method that uses diverse synthetic views generated from static images via an image-conditioned diffusion model. We present a quantitative method to evaluate the local and global consistency of the generated views to choose the right diffusion model for cross-view self-supervised pretraining. These generated views exhibit substantial changes in pose and perspective, providing a rich training signal that overcomes the limitations of video and crop-based anchors. Furthermore, we enhance the standard single-anchor MAE setting to a multi-anchor masking strategy to increase the difficulty of the pretext task. CDG-MAE substantially narrows the gap to video-based MAE methods, while maintaining the data advantages of image-only MAEs.
URL: https://openreview.net/forum?id=7XIymKIA0v
---
Title: Benford’s Law as a Distributional Prior for Post-Training Quantization of Large Language Models
Authors: Arthur Negrão de Faria Martins da Costa, Pedro Silva, Vander L. S. Freitas, Gladston Moreira, Eduardo Luz
Abstract: Post-training quantization (PTQ) is a practical way to reduce the memory footprint of large language models, but low-bit quantization is sensitive to mismatches between the quantization codebook and the empirical weight/activation distributions. We revisit Benford-like leading-digit statistics as a lightweight diagnostic of scale-broad behavior in transformer tensors. Across several model families, we observe a consistent functional dichotomy: transformational nn.Linear weights tend to be Benford-like, whereas LayerNorm and embedding parameters systematically deviate. Motivated by this observation, we propose BenQ, a data-free PTQ codebook that uses a simple log-spaced grid as a proxy for scale-broad distributions and applies it selectively to transformational layers while keeping stability-critical parameters in higher precision. In 4-bit group-wise PTQ, BenQ consistently improves over uniform RTN and is often competitive with NF4, while remaining substantially simpler than optimization-based methods. We additionally report activation quantization results as an exploratory stress test: BenQ can improve robustness over uniform baselines in some families, but performance remains mixed across models, highlighting open challenges for static-grid activation PTQ. Code is available at https://github.com/ufopcsilab/benford-quant.
URL: https://openreview.net/forum?id=YiLcQY4Nje
---
Title: Unified Sample Difficulty Estimation in Pathology Foundation Models
Authors: Tianyu Liu, Lijun Wang, Tinglin Huang, Xiao Luo, Wengong Jin, Hongyu Zhao
Abstract: The fast scaling speed of histopathology datasets allows researchers to train various foundation models for disease-centered research with applications in classifying disease-state information and predicting gene expression levels. However, it has been shown that current models tend to be overconfident and make classification at a low-calibration level. This case is underexplored for regression-type tasks such as gene expression prediction as well, which could seriously affect the diagnosis and treatment based on the developed models. To resolve this critical issue, we propose a \underline{u}niversal framework\footnote{Full codes can be found here: \url{https://anonymous.4open.science/r/USD-F176/} (also in supplementary files).} to estimate the \underline{s}ample \underline{d}ifficulty (USD) in both regression and classification tasks. In particular, we fit the data in the embedding space with Gaussian distribution and then utilize prior-informed relative Mahalanobis distance to estimate sample difficulty. Moreover, we incorporate such difficulty as a weight to regularize the model prediction, which can improve model performance by emphasizing challenging samples. Our method can be seamlessly extended to regression tasks by the incorporation of discrete targets. Compared with the second-best method on average, extensive experiments demonstrate that our proposed USD can improve the disease-state classification accuracy by 1.0\% at least and gene-level correlation by 3.5\% at least using UNI. Finally, we provide comprehensive ablation tests to demonstrate the importance of including sample difficulty in the training stage and case studies for assigning samples with different difficulty levels.
URL: https://openreview.net/forum?id=LLlOJs4o2N
---
Title: What Survives Privatization? A Guide to Structure and Utility in Differentially Private Genome-Wide Association Studies
Authors: Shadi Rahimian, Akito Yamamoto, Mario Fritz
Abstract: Single nucleotide polymorphisms (SNPs) are among the most common and informative forms of genetic variation in the human genome and constitute the primary data representation used in genome-wide association studies (GWAS). Due to their extreme dimensionality, strong correlation structure, and the presence of both population-level and familial dependencies, SNP datasets exhibit structural properties that fundamentally distinguish them from standard tabular data. At the same time, genomic data is uniquely sensitive; it is immutable, identifying, and shared across relatives, and has been shown to be vulnerable to a wide range of attacks, including membership inference, reconstruction, and kinship inference. As a result, protecting SNP data has become a critical and practically unavoidable requirement.
Differential privacy (DP) provides a rigorous mathematical framework for protecting sensitive data under strong adversarial assumptions. However, in the context of GWAS, the design and evaluation of meaningful DP mechanisms crucially depend on understanding the biological, statistical, and structural properties of SNP data and the downstream analysis pipelines. For a typical privacy researcher, acquiring even the minimal domain knowledge required to reason correctly about the structure of genomic data and the associated analysis pipelines represents a substantial and time-consuming barrier. Yet, without this understanding, progress in private genomic data analysis risks being misguided or misleading.
This survey explicitly bridges this gap. We provide a structured, self-contained primer on the structural properties of SNP data and the core analytical workflows of GWAS, focusing on the aspects most consequential for privacy definitions, mechanism design, and utility. Building on this foundation, we present a comprehensive and systematic overview of differentially private methods for SNP datasets. We organize the literature through a release-oriented taxonomy that reframes existing approaches in terms of what survives privatization, revealing the design choices and trade-offs that shape their scientific and practical utility. Finally, we identify key open challenges arising from mismatches between existing DP methodologies and the scientific, statistical, and operational realities of genomic data analysis, and outline future research directions toward principled and deployable privacy-preserving GWAS.
URL: https://openreview.net/forum?id=6BWikkmkOH
---
Title: On-the-go Forgetting without Explicit Unlearning via ERASE
Authors: Kushal Chakrabarti, Mayank Baranwal
Abstract: Existing unlearning approaches typically rely on post hoc weight adaptation or distillation, leading to duplicated memory costs, degraded generalization, and limited scalability. In this work, we introduce ERASE, Erasure via Reconstructive Adversarial Signal Editing, a framework for on-the-go forgetting that suppresses the observable influence of private data without modifying model weights. ERASE leverages structured, class-conditioned input perturbations to induce selective forgetting during inference, eliminating the need for retraining, fine-tuning, or model copies. We rigorously characterize sufficient conditions when ERASE provably achieves functional forgetting of designated subclasses while preserving predictions across other subclasses within the same superclass. This analysis offers a principled foundation for inference-time forgetting under mild regularity assumptions. Across diverse architectures and benchmark datasets, ERASE maintains the best observed balance between forgetting efficacy, computational efficiency, and retention fidelity over recent unlearning-based methods. By reimagining data removal as forgetting without unlearning, our work establishes a scalable, regulation-aligned pathway for continual, privacy-conscious learning.
URL: https://openreview.net/forum?id=PIXVov5LQq
---
Title: Watermarking Language Models with Error Correcting Codes
Authors: Patrick Chao, Yan Sun, Edgar Dobriban, Hamed Hassani
Abstract: Recent progress in large language models enables the creation of realistic machine-generated content. Watermarking is a promising approach to distinguish machine-generated text from human text, embedding statistical signals in the output that are ideally undetectable to humans. We propose a watermarking framework that encodes such signals through an error correcting code. Our method, termed robust binary code (RBC) watermark, introduces no noticeable degradation in quality. We evaluate our watermark on base and instruction fine-tuned models and find that our watermark is robust to edits, deletions, and translations. We provide an information-theoretic perspective on watermarking, a powerful statistical test for detection and for generating p-values, and theoretical guarantees. Our empirical findings suggest our watermark is fast, powerful, and robust, comparing favorably to the state-of-the-art.
URL: https://openreview.net/forum?id=H6oBZxNQk2
---
Title: Random features for Grassmannian kernel approximation with bounded rank-one projections
Authors: Remi Delogne, Laurent Jacques
Abstract: We propose a family of random feature maps for scalable kernel machines defined over low-dimensional subspaces in high dimensions, \ie over the Grassmannian manifold. This is typically useful in a machine learning context when data classes or clusters are well represented by the span of a few data points. Classical Grassmannian kernels such as the \emph{projection} or \emph{Binet–Cauchy} kernels require constructing full Gram matrices for practical applications, leading to prohibitive computational and memory costs for large subspace datasets in high dimensions. We address this limitation by computing specific random features of subspaces. These combine random rank-one projections of the subspace projection matrices with bounded non-linear transforms---periodic or binary---to tame the resulting heavy-tailed distribution.
We show that, in the random feature space, inner products approximate well-defined, rotation-invariant Grassmannian kernels, \ie depending only on the principal angles of the considered subspaces. Provided the number of random features is large compared to the subspace intrinsic dimension, we show that this approximation holds uniformly over all subspaces of fixed dimensions with high probability.
When the non-linear transform is periodic, the approximated kernel admits a closed-form expression with a tunable behaviour bridging inverse Binet–Cauchy and Gaussian-type regimes, while the binarised feature has no known closed-form kernel but lends itself to even more compactly represented one-bit subspace features. Moreover, we show how structured rank-one projections, leveraging randomised fast Fourier transforms, further reduce the random feature computational complexity without sacrificing accuracy in practical experiments.
We demonstrate the practicality of these techniques with synthetic experiments and classification tasks on the ETH-80 dataset representing visual object images from different viewpoints. The proposed random features recover Grassmannian geometry with high accuracy while reducing computation, memory, and storage requirements. This demonstrates that rank-one embeddings offer a practical and scalable alternative to classical Grassmannian kernels.
URL: https://openreview.net/forum?id=wq18dZJ2pA
---
Title: On Almost Surely Safe Alignment of Large Language Models at Inference Time
Authors: Xiaotong Ji, Shyam Sundhar Ramesh, Matthieu Zimmer, Ilija Bogunovic, Jun Wang, Haitham Bou Ammar
Abstract: We introduce a novel inference-time alignment approach for LLMs that aims to generate safe responses almost surely, i.e., with probability approaching one w.r.t. a given cost model. Our approach models the generation of safe responses as a constrained Markov Decision Process (MDP) within the LLM's latent space. We augment a safety state that tracks the evolution of safety constraints and dynamically penalize unsafe generations to ensure the generation of safe responses. Consequently, we demonstrate formal safety guarantees w.r.t. the given cost model upon solving the MDP in the latent space with sufficiently large penalties. Building on this foundation, we propose $\texttt{InferenceGuard}$, a practical implementation that safely aligns LLMs without modifying the model weights. Empirically, we demonstrate that $\texttt{InferenceGuard}$ effectively balances safety and task performance, outperforming existing inference-time alignment methods in generating safe and aligned responses. Our findings contribute to the advancement of safer LLM deployment through alignment at inference time, thus presenting a promising alternative to resource-intensive, overfitting-prone alignment techniques like RLHF.
URL: https://openreview.net/forum?id=FlnokjaSEu
---
Title: Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis
Authors: Yuichi Kondo, Hideaki Iiduka
Abstract: We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function. We extend the existing theoretical framework to cover three practical scheduling strategies commonly used in deep learning: a constant batch size with a decaying learning rate, an increasing batch size with a decaying learning rate, and an increasing batch size with an increasing learning rate. Our results reveal a clear hierarchy in convergence: a constant batch size does not guarantee convergence of the expected gradient norm under our Lyapunov-based analysis, whereas an increasing batch size does, and simultaneously increasing both the batch size and learning rate achieves a provably faster decay. Empirical results validate our theory, showing that dynamically scheduled SGDM significantly outperforms its fixed-hyperparameter counterpart in convergence speed. We also evaluated a warmup schedule in experiments, which empirically outperformed all other strategies in convergence behavior.
URL: https://openreview.net/forum?id=s6DTv7Sorj
---
Title: Revisiting Learning-based Video Motion Magnification for Real-time Processing
Authors: Hyunwoo Ha, Oh Hyun-Bin, Kim Jun-Seong, Kwon Byung-Ki, Kim Sung-Bin, Linh-Tam Tran, Ji-Yun Kim, Sung-Ho Bae, Tae-Hyun Oh
Abstract: Video motion magnification is a technique to capture and amplify subtle motion in a video that is invisible to the naked eye. The deep learning-based prior work successfully models outstanding quality better than conventional signal processing-based ones. However, it still lags behind real-time performance, which prevents it from being extended to various online systems. In this paper, we revisit the first learning-based model and present experimental analyses, in particular on the identification of redundant components, the insertion of spatial bottlenecks, and the trade-off relationship between channel reduction and layer addition. By integrating the findings of each experiment, we present a real-time, deep learning-based motion magnification model that achieves a computational speed ranging from a minimum of 2.7 times to a maximum of 34.9 times faster than existing learning-based methods, while maintaining perceptually sufficient generation quality. To the best of our knowledge, this is the first learning-based motion magnification model that runs in real-time on Full-HD resolution videos even without ad hoc quantization. Project page: https://fastdmm.github.io/
URL: https://openreview.net/forum?id=TAmmPuExE1
---
Title: Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts
Authors: Sanae Lotfi, Lucas Caccia, Alessandro Sordoni, Jordan T. Ash, Miroslav Dudík
Abstract: While large language models (LLMs) fine-tuned with lightweight adapters achieve strong performance across diverse tasks, their performance on individual tasks depends on the fine-tuning strategy. Fusing independently trained models with different strengths has shown promise for multi-task learning through three main strategies: ensembling, which combines outputs from independent models; merging, which fuses model weights via parameter averaging; and routing, which integrates models in an input-dependent fashion. However, many design decisions in these approaches remain understudied, and the relative benefits of more sophisticated ensembling, merging and routing techniques are not fully understood. We empirically evaluate their trade-offs, addressing two key questions: What are the advantages of going beyond uniform ensembling or merging? And does the flexibility of routing justify its complexity? Our findings indicate that non-uniform ensembling and merging improve performance, but routing offers even greater gains. Surprisingly, uniform ensembling with no learning outperforms all merging methods, suggesting that mode connectivity constraints fundamentally limit parameter-space fusion in the multi-task setting. Meanwhile, SGD-optimized routing achieves the best non-oracle performance with less than $1\%$ inference compute overhead. To mitigate the computational cost of routing, we analyze expert selection techniques, showing that clustering and greedy subset selection can maintain reasonable performance with minimal overhead. These insights advance our understanding of model fusion for multi-task learning.
URL: https://openreview.net/forum?id=bnRCvRtZv5
---
Title: FUND: Density Flow for Sampling Unnormalised Distributions
Authors: Vikas Kanaujia, Vipul Arora
Abstract: Efficient sampling from Boltzmann distributions is central to modelling complex physical systems. Markov Chain Monte Carlo (MCMC) methods suffer from critical slowing down, high autocorrelation, and poor mode-mixing, limiting their scalability. Recent advances, like Boltzmann Generators, offer a promising alternative but remain constrained by costly MCMC-based training, inefficient sampling, and poor ergodicity. We introduce an algorithm for learning Boltzmann distributions that does not require any true samples for training. Our approach draws inspiration from flow matching but departs fundamentally from sample-trajectory matching to distribution-trajectory matching. The algorithm iteratively reshapes the target distribution, using model generated samples to guide learning and ensure comprehensive mode coverage. We validate our method on standard benchmarks, including a 2D Gaussian mixture, Many-Well distributions, and high-dimensional scalar $\phi^4$ theory. The proposed approach not only improves sampling performance and accuracy over traditional MCMC and flow-based baselines but also establishes a new method for sample-free learning of complex physical distributions.
URL: https://openreview.net/forum?id=O05dDDVcyZ
---
Title: ARC-Encoder: learning compressed text representations for large language models
Authors: Hippolyte Pilchen, Edouard Grave, Patrick Perez
Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs. Context compression techniques can reduce these costs, but the most effective approaches require fine-tuning the target model or even modifying its architecture. This can degrade its general abilities when not used for this specific purpose. Here we explore an alternative approach: an encoder that compresses the context into continuous representations which replace token embeddings in decoder LLMs. First, we perform a study of training strategies and architecture choices for the encoder. Our findings led to the design of an Adaptable text Representations Compressor, named ARC-Encoder, which outputs $x$-times fewer continuous representations (typically $x \in \{4,8\}$) than text tokens. We evaluate ARC-Encoder across a variety of LLM usage scenarios, ranging from in-context learning to context window extension, on both instruct and base decoders. Results show that ARC-Encoder achieves strong performance on several benchmarks and tasks while improving computational efficiency at inference. Finally, we demonstrate that our models can be adapted to multiple decoders simultaneously, allowing a single encoder to generalize across different decoder LLMs. This makes ARC-Encoder a flexible and efficient solution for portable encoders that can support multiple LLMs with only small MLPs.
URL: https://openreview.net/forum?id=lU1P9dsqfn
---
Title: Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs
Authors: Arjun Ashok, Andrew Robert Williams, Vincent Zhihao Zheng, Irina Rish, Nicolas Chapados, Étienne Marcotte, Valentina Zantedeschi, Alexandre Drouin
Abstract: Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the “Execution Gap” where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. Code is made available at \url{https://github.com/ashok-arjun/beyond-naive-prompting}.
URL: https://openreview.net/forum?id=dkjHHFJkVI
---
Title: Automata Learning from Recurrent Networks: A Critical Synthesis for Verification, Testing, and Interpretability
Authors: Jaouhar Slimi, Augustin Lemesle, Tristan Le Gall, Zakaria Chihani
Abstract: Recurrent Neural Networks (\RNNs) have demonstrated their effectiveness in modeling sequential data and are a key building block of modern deep learning architectures. In this review paper, we study recurrent networks through the lens of automata theory. Given an \RNN, automata learning seeks to model its behavior with an automaton, which enables better interpretability and eases our understanding of its working mechanisms. We begin by examining the theoretical foundations of this approach, demonstrating how it can be applied to learn automata from various types of recurrent architectures, including the Elman Recurrent Network (ERN), Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU). Next, we review the applications of this approach in formal verification, model-based testing, and the interpretability of these deep learning models. We finish with a discussion on the advantages and critical problems of this method, while outlining key goals for future research, such as defining standard benchmarks and identifying limitations that need to be addressed to advance this field further.
URL: https://openreview.net/forum?id=R52ETbUBVo
---
Title: Quantification and Control of LSTM Resilience Based on Stability Theory
Authors: Sota Yoshihara, Ryosuke Yamamoto, Hiroyuki Kusumoto, Masanari Shimura
Abstract: This paper proposes a novel theoretical framework for guaranteeing and evaluating the resilience of long short-term memory (LSTM) networks in control systems. We introduce *recovery time* as a new metric of resilience in order to quantify the time required for an LSTM to return to its normal state after anomalous inputs. By mathematically refining incremental input-to-state stability ($\delta$ISS) theory for LSTM, we derive a practical data-independent upper bound on recovery time. This upper bound gives us resilience-aware training. Experimental validation on simple models demonstrates the effectiveness of our resilience estimation and control methods, enhancing a foundation for rigorous quality assurance in safety-critical AI applications.
URL: https://openreview.net/forum?id=hFmlMUNEsR
---
Title: Learning 3D Hypersonic Flow with Physics-Enhanced Neural Fields: A Case Study on the Orion Reentry Capsule
Authors: Haitz Sáez de Ocáriz Borde, Pietro Innocenzi, Flavio Savarino, Andrei Cristian Popescu, Pantelis Papageorgiou
Abstract: We develop a 3D aerothermodynamic surrogate for the Orion reentry capsule at hypersonic speeds, a timely case study given its role in upcoming lunar missions. The large computational meshes required for these scenarios make traditional computational fluid dynamics impractical for full-mission performance prediction and control. In this work, we propose physics-enhanced 3D neural fields for predicting steady hypersonic flow around aerodynamic bodies. The model maps spatial coordinates and angle of attack to pressure, temperature, and velocity components. We enhance the base model with Fourier positional feature mappings, which allow it to capture the sharp discontinuities typical of hypersonic flows, and further constrain the solution by imposing no-slip and isothermal wall conditions. We compare our proposed approach to other surrogate alternatives, such as graph neural networks, and demonstrate its superior performance in capturing the steep gradients ubiquitous in this regime. Our formulation yields a continuous and computationally efficient aerothermodynamic surrogate that supports rapid exploration of operating conditions based on angle of attack variation under realistic flight profiles (for a fixed capsule geometry). While we focus on Orion, the proposed framework provides a general methodology for data-driven simulation in 3D hypersonic aerothermodynamics.
URL: https://openreview.net/forum?id=ce2X1X3l0Y
---
Title: GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI
Authors: Naomi Simumba, Nils Lehmann, Paolo Fraccaro, Hamed Alemohammad, Geeth De Mel, Salman Khan, Manil Maskey, Nicolas Longépé, Xiao Xiang Zhu, Hannah Kerner, Juan Bernabe Moreno, Alexandre Lacoste
Abstract: Geospatial Foundation Models (GeoFMs) are transforming Earth Observation (EO), but
evaluation lacks standardized protocols. GEO-Bench-2 addresses this with a com-
prehensive framework spanning classification, segmentation, regression, object detection,
and instance segmentation across 19 permissively-licensed datasets. We introduce capabil-
ity groups to rank models on datasets that share common characteristics (e.g., resolution,
spectral bands, temporality), enabling users to identify which models excel in each capa-
bility and to determine where future work should focus. To support both fair comparison
and methodological innovation, we define a prescriptive yet flexible evaluation protocol.
This ensures consistency in benchmarking while facilitating research into model adapta-
tion strategies—a key open challenge in advancing GeoFMs for downstream tasks. Our
experiments show that no single model dominates across all tasks, confirming the specificity
of choices made during architecture design and pretraining. While models pretrained on
natural images (ConvNext-ImageNet, DINOv3) excel on high-resolution tasks, EO-specific
models (TerraMind, Prithvi, and Clay) outperform them on multispectral applications such
as agriculture and disaster response. These findings demonstrate that optimal model choice
depends on task requirements, data modalities, and operational constraints, and that the
goal of a single GeoFM that performs well across all tasks remains open for future research.
GEO-Bench-2 enables informed, reproducible GeoFM evaluation tailored to specific
use cases. Code, data, and the leaderboard are publicly released under a permissive license.
URL: https://openreview.net/forum?id=NPf175jnP1
---
Title: Revisiting Neighbourhoods in Mean Field Reinforcement Learning
Authors: Sriram Ganapathi Subramanian, Matthew E. Taylor, Kate Larson, Mark Crowley, Pascal Poupart
Abstract: Many multi-agent reinforcement learning (MARL) algorithms do not scale well as the number of agents increases due to an exponential time and space complexity dependency on the number of agents in the environment. Mean field theory has been used to address this problem by approximating the effect of neighbourhoods of agents by a single representative agent. While this approximation allows MARL algorithms to scale to environments with many agents, approaches typically assumed that agents 1) inside a neighbourhood are homogeneous, and 2) outside a neighbourhood have no influence (and can therefore be ignored). This paper relaxes these assumptions and proposes a novel framework, mean field attention (MFA), which uses an attention mechanism for local responses and the mean field approximation for global responses. We implement MFA with two new algorithms leveraging Q-learning and actor-critic. These novel MFA algorithms consistently outperform other MARL algorithms, including prior mean field-based algorithms, across multiple metrics and benchmarks.
URL: https://openreview.net/forum?id=PQ5R7K0WDc
---
Title: On the Convergence Analysis of Muon
Authors: Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, Jiawei Zhang
Abstract: The majority of parameters in neural networks are naturally represented as matrices. However, most commonly used optimizers treat these matrix parameters as flattened vectors during optimization, potentially overlooking their inherent structural properties. Recently, an optimizer called Muon has been proposed, specifically designed to optimize matrix-structured parameters. Extensive empirical evidence shows that Muon can significantly outperform traditional optimizers when training neural networks. Nonetheless, the theoretical understanding of Muon’s convergence behavior and the reasons behind its superior performance remain limited. In this work, we present a comprehensive convergence rate analysis of Muon and its comparison with Gradient Descent (GD). We characterize the conditions under which Muon can outperform GD. Our theoretical results reveal that Muon can benefit from the low-rank structure of Hessian matrices, a phenomenon widely observed in practical neural network training. Our experimental results support and corroborate the theoretical findings.
URL: https://openreview.net/forum?id=4nH4CulGaP
---
Title: Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
Authors: Piotr Piękos, Róbert Csordás, Firas Laakom, Li Nanbo, Jürgen Schmidhuber
Abstract: Quadratic self-attention remains a central obstacle to scaling sequence length, but many sparse attention methods lose quality relative to dense attention under comparable compute budgets.
This paper studies whether learned, content-based tokezzn selection can make sparse attention competitive with dense attention as a method for training transformer language models.
We present Mixture of Sparse Attention (MoSA), an attention mechanism inspired by Mixture of Experts with expert-choice routing, where each attention head selects its own subset of tokens and computes attention only within that subset.
We create a hybrid attention layer that keeps four dense heads to preserve stable dense information flow, while the remaining dense heads are replaced by a larger number of MoSA heads.
Selecting $k$ tokens from a sequence of length $T$ reduces the per-head attention cost from $O(T^2)$ to $O(k^2+T)$, which lets us trade each removed dense head for multiple sparse heads at matched FLOPs.
In this controlled IsoFLOP setting, across dense baselines from 28M to 516M parameters, MoSA is the only tested sparse attention variant that consistently improves over dense attention, reaching up to 27% lower perplexity at the same compute budget.
These results show that expert-choice, content-based sparse attention can outperform dense attention in the evaluated fixed-length, non-autoregressive training setup.
We also report perplexity-matched resource measurements showing lower wall-clock time, training memory, and stored key-value pairs in our implementation, which demonstrates the potential benefits of MoSA for autoregressive models if the non-autoregressive router is adapted to causal decoding.
URL: https://openreview.net/forum?id=HUpBs4TZkS
---
New submissions
===============
Title: What Survives Eviction? Information Recall Locality in Transformer KV Caches
Abstract: During autoregressive generation, transformer-based language models rely on the KV cache as their record of prior context.
We study information recall locality in LLMs. Specifically, we ask \textit{whether transformer-based language models can recall the information that token(s) carried after their original KV cache entries are removed}.
We study this question on a suite of association recall tasks on which we measure the recall of factual and sentiment information (e.g. ``doctor Alice'' vs. ``dingy bowl''), under various KV cache eviction scenarios.
Our evaluation of eight instruction-tuned models (0.6B--8B parameters) leads to three observed patterns:
(1) When all token-caches of a target association are evicted, \textit{recall drops to near zero} across all models, indistinguishable from settings where the association was never present.
(2)~When only one side of an associated pair is evicted (the entity or the attribute), partial recovery occurs if the surviving side appeared \emph{after} the evicted one in the original context.
(3)~For sentiment, re-mentioning the target entity later in the context partially
recovers the evicted information, \textit{whereas} factual recall benefits negligibly (around 1pp) from such re-mention.
These results show that recall from the KV cache is strongly local for factual information, while sentiment information can be distributed more diffusely across subsequent token caches.
URL: https://openreview.net/forum?id=eAMLpW74rV
---
Title: Depth-Consistent Graph Generation via Depth Regularization
Abstract: Many real-world graphs exhibit clear depth-aware or hierarchical organization. Most graph generation models struggle to capture this structure effectively. This research introduces DeCoGen, an auto-regressive graph generation framework that incorporates a differentiable depth consistency loss derived from depth-first search (DFS) timestamps to guide generation. This method exploits the connectivity patterns of depth-ordered graphs, encouraging nodes at depth $d$ to connect to nodes at a depth $d+1$ while penalizing depth reversals or insufficient separation between ancestor-descendant pairs, thereby promoting structurally realistic and hierarchically consistent graph generation. Depth constraints are applied selectively to discovery edges, enabling the model to operate on both hierarchical and non-hierarchical graphs. This auxiliary loss integrates with standard reconstruction objectives used by auto-regressive generators. The method requires only a lightweight auxiliary prediction head without redesigning the underlying auto-regressive generator and the depth regularizer operates in linear time and space, making it directly applicable to existing auto-regressive graph generators. Experiments were conducted across multiple graph domains, including citation networks, biological and molecular structures, and semantic auto-suggestion trees. Results show that DeCoGen matches or outperforms GraphGen and GraphRNN, highlighting the benefit of enforcing depth-aware structural constraints during generation.
URL: https://openreview.net/forum?id=tcX5XsDwlR
---
Title: Females Wear Makeup and Males Have Big Noses? Analyzing How Social Biases Manifest in Image Editing Models
Abstract: Text-to-image (T2I) models are widely adopted but are known to propagate human biases. In contrast to T2I generation models where biases have been extensively studied in prior work, biases in T2I editing models on the other hand, remain comparatively underexplored. Existing analyzes primarily quantify changes in gender, age, and race between input and edited images, providing limited insight into how these biases arise or how they relate to one another. In our work, we propose a bias evaluation framework that characterizes how demographic biases manifest through systematic changes in low-level attributes, which are fine-grained visual features such as facial hair and other appearance cues, and analyze relationships between different demographic biases. Our results show that demographic biases are consistently expressed through specific attribute changes and changes in one demographic dimension are systematically associated with changes in others. Furthermore, we demonstrate that modifying the editing prompt to discourage these bias-associated attributes can help to reduce biased behavior in image editing models, highlighting the potential of attribute-level analysis for understanding and reducing demographic biases.
URL: https://openreview.net/forum?id=rLi5hbUlOi
---
Title: Revisiting Attention Formulations for Molecular Transformers with Positional Encoding
Abstract: Molecular property prediction is a core task in machine learning for physics, with applications to interatomic potential learning and materials discovery. Recent transformer-based models achieve strong performance, but their gains often conflate two coupled choices: the design of SE(3)-aware positional encodings and how these encodings are incorporated into attention. Consequently, it remains unclear whether the empirical improvements stem from the positional encoding itself or from the attention formulation used to exploit it. Here, we present a controlled study of attention design in molecular transformers that fixes the positional encoding and only varies the attention formulation. Our results show that attention design alone has a significant effect on performance. Under a shared positional encoding, vector-valued positional encodings are most effective when used to modulate attention logits, while the best overall results are obtained when geometric information is introduced through a separate pathway that is adaptively gated by the main attention message. Our study isolates the effect of attention design, identifies key ingredients for accurate prediction of molecular properties, and offers practical guidelines for future research.
URL: https://openreview.net/forum?id=D1Tyti9BpG
---
Title: TaylorPODA: A Taylor Expansion-Based Method to Improve Post-Hoc Attributions for Opaque Models
Abstract: Post-hoc model-agnostic local attribution (LA) methods have been widely adopted to explain opaque AI models by quantifying feature-wise contributions. However, many existing methods rely on heuristic or only partially justified attribution mechanisms, while the quality of attribution itself is often shaped by downstream objectives without universally accepted standards. In this work, we propose Taylor exPansion-Originated aDaptive Attribution (TaylorPODA), a new post-hoc model-agnostic LA method grounded in the Taylor expansion framework. We first introduce a set of postulates, namely \textit{precision}, \textit{federation}, and \textit{zero-discrepancy}, which formalize principled requirements for explicitly and exhaustively attributing Taylor terms to the corresponding features. Based on these postulates, we analyze existing post-hoc model-agnostic LA methods and identify a fundamental tension between principled attribution and adaptation toward user-defined utilities. To address this challenge, TaylorPODA introduces a controllable allocation mechanism for Taylor interaction effects, enabling attribution results to adapt to downstream objectives while preserving the proposed postulates. Furthermore, although developed from a Taylor-expansion perspective, TaylorPODA also admits a Harsanyi-dividend interpretation, allowing the attribution mechanism to extend beyond model differentiability. Theoretical analysis demonstrates that TaylorPODA satisfies all the proposed postulates together with an additional \textit{adaptation} property. Empirical results across multiple datasets and both differentiable and non-differentiable models further show that TaylorPODA achieves consistently improved alignment with user-defined utilities while maintaining the communicability of the resulting explanations. Overall, this work provides a starting point toward more trustworthy XAI systems for the deployment of increasingly powerful yet opaque task models. The code is available at \url{https://anonymous.4open.science/r/TaylorPODA/}.
URL: https://openreview.net/forum?id=CwlgaPa7MJ
---
Title: Heavy Ball Dynamics with Hessian-Driven Damping for non-convex Optimization under the Łojasiewicz Condition
Abstract: In this paper, we examine the convergence properties of heavy ball dynamics with Hessian-driven damping in smooth non-convex optimization problems satisfying a Łojasiewicz condition. In this general setting, we provide a series of tight, worst-case optimal convergence rate guarantees as a function of the dynamics' friction coefficients and the Łojasiewicz exponent of the problem's objective function. Importantly, the linear rates that we obtain, improve on previous available rates and they suggest a different tuning of the dynamics' damping terms, even in the strongly convex regime. We complement our analysis with a range of stability estimates in the presence of perturbation errors and inexact gradient input, as well as an avoidance result showing that the dynamics under study avoid strict saddle points from almost every initial condition.
URL: https://openreview.net/forum?id=YYHRkoTyJa
---
Title: Bayesian Sequential Finding of Orthogonal Functions
Abstract: We propose a novel class of prior distributions for Bayesian inference on a sequence of orthogonal functions, which are frequently required in various statistical models such as functional principal component analysis (FPCA). Our approach constructs prior distributions sequentially by imposing orthogonality constraints adaptively through a hierarchical Bayesian formulation using conditionally normal distributions. The orthogonality is controlled via learnable hyperparameters, allowing for flexible trade-offs between exactness and smoothness. We illustrate properties of the proposed prior and show that it leads to nearly orthogonal posterior estimates with an efficient posterior computation algorithm. The proposed prior is employed in Bayesian FPCA to provide more interpretable principal functions and efficient low-rank representations than previous Bayesian approaches. Through simulation studies and analysis of human mobility data in Tokyo, we demonstrate the superior performance of our approach in inducing orthogonality and improving functional component estimation.
URL: https://openreview.net/forum?id=u05rHG8g5e
---
Title: Distance from Equilibrium Is Not the Same as Loss from Equilibrium: A Colonel Blotto Study of LLM Persona Behavior
Abstract: We formalize a small LLM-agent territorial-expansion game as a weighted Colonel Blotto game and ask how persona-driven LLM allocations relate to a numerically approximated equilibrium benchmark. We separate two notions that are easy to conflate: distance, a descriptive measure of how different a persona’s allocation shape is from the equilibrium’s mean allocation, and ex-post regret, the standard measure of how much worse a decision did than the best hindsight response. Across 300 independent real-API scenarios per persona on each of two model families (qwen-plus and deepseek-v3; every allocation in this paper is a real LLM decision, 9,300 calls total, 100% parseable), the two measures give opposite rankings on both backends: the persona farthest from the equilibrium distribution (Aggressive Developer) has the lowest regret (\(p<10^{-4}\) on both backends).
An 8-level concentration-gradient experiment, also replicated on both backends, then shows this dissociation is structural rather than a coincidence of three hand-written personas: as instructed concentration rises from uniform spreading to all-in, distance responds strongly (up to a 19-fold range, 0.070 to 1.319; Spearman \(\rho=0.672\) and \(0.644\), both \(p<10^{-280}\)) while regret against three equilibrium-sampling opponents stays flat (\(\rho=0.010\) and \(0.002\), \(p=0.62\) and \(0.92\)). A sweep over budget, battlefield count, player count, and value distribution finds the inversion in six of seven conditions; in the seventh (\(m=3\)), persona regret differences vanish entirely while distance still separates strongly—the dissociation’s limiting case, not a counterexample.
Behavioral analysis explains the mechanism: the equilibrium’s sampled play is itself highly concentrated (top-battlefield share 0.72–0.74, abandoning half the battlefields), so the persona that looks least like the equilibrium’s mean looks most like its actual play, lacking only its randomness—and all personas under-randomize 3–5× relative to equilibrium, the same direction of deviation documented for human Blotto subjects. We also report how we discovered the distance/regret distinction by first computing the wrong metric—comparing realized payoff to the equilibrium’s mean allocation, which produced a nonsensical negative “loss.” Behavioral resemblance to equilibrium and realized decision quality are empirically independent quantities in this game, and claims about the “rationality” of persona-driven LLM agents should specify which one is being measured.
URL: https://openreview.net/forum?id=xjzMQUKPoU
---
Title: Thinking Is Not Telling: Information Disclosure in User-Service LLM Agents
Abstract: User-engaged LLM agents increasingly operate in service scenarios where task success depends on coordination between the agent, the user, and a stateful environment. In such interactions, the agent often has access to task policies, tool results, and environment states that the user does not observe. This makes agent-user communication a central component of task completion. In this work, we study a communication-level failure mode of user-service agents: user-invisible reasoning does not necessarily translate into timely user-facing information disclosure.
We introduce information disclosure as a measurable communication mechanism, covering state corrections, available options, constraints, consequences, and proactively retrieved environment information. We use this perspective to analyze the widely used user-invisible "think" tool, which allows agents to reason internally before responding or acting. Across five models and three user-service scenarios, we first show that enforced thinking has unstable utility. We then analyze response scaling under different think efforts and find that more internal thinking does not reliably translate into richer user-facing communication. Using evidence-grounded probing and turn-level counterfactual intervention, we show that injecting golden decision-relevant information produces the clearest improvement in First Edit Correction Rate, outperforming verbosity and direct think-content controls. This indicates that missing disclosure can causally affect the agent's immediate downstream environment-edit behavior. We further find that failed trajectories most often lack state corrections and constraints, and that a lightweight disclosure-aware prompt improves overall performance while mitigating degradation from enforced thinking. These results suggest that the key challenge is not simply making agents think more, but ensuring that private reasoning and environment knowledge are externalized as timely, user-facing information disclosure.
URL: https://openreview.net/forum?id=xngXTG9Jyj
---
Title: Similarity Guided Adaptive Diffusion
Abstract: Denoising diffusion models have emerged as a leading paradigm for high-fidelity image synthesis; however, their iterative reverse sampling process incurs substantial computational overhead, resulting in high inference latency and infrastructure costs. While existing acceleration methods rely on resource-intensive model distillation or fixed, input-agnostic sampling schedules, we observe substantial temporal redundancy in intermediate denoising steps. Motivated by this observation, we propose an entirely training-free adaptive
inference framework based on temporal noise caching that dynamically exploits localized temporal redundancies. By monitoring the real-time update divergence between consecutive timesteps on-the-fly, our method intelligently reuses previously predicted noise estimates
and skips costly reverse network evaluations whenever the denoising trajectory enters a localized redundancy plateau. Extensive empirical evaluations on Stable Diffusion 1.5 across the MSCOCO 2017 and ImageNet-1K benchmarks validate the efficacy of our approach. Remarkably, our framework not only accelerates inference by up to 1.98× without retraining, but also achieves lower Fréchet Inception Distance (FID) than the original sampling process, suggesting that skipping redundant updates can improve sample quality.
URL: https://openreview.net/forum?id=g9Hdn8QRVT
---
Title: Mutual Information Preserving Neural Network Pruning
Abstract: Pruning has emerged as one of the primary approaches used to limit the resource requirements of large neural networks (NNs). Since the proposal of the lottery ticket hypothesis, researchers have focused either on pruning at initialization or after training. However, recent theoretical findings have shown that the sample efficiency of robust pruned models is proportional to the mutual information (MI) between the pruning masks and the model's training datasets, \textit{whether at initialization or after training}. In this paper, we introduce Mutual Information Preserving Pruning (MIPP), a structured activation-based pruning technique applicable before or after training. The core principle of MIPP is to select nodes in a way that conserves MI shared between the activations of adjacent layers, and consequently between the data and masks. Approaching the pruning problem in this manner means we can prove that there exists a function that can map the pruned upstream layer's activations to the downstream layer's, implying re-trainability. We demonstrate that MIPP consistently outperforms baselines, regardless of whether pruning is performed before or after training.
URL: https://openreview.net/forum?id=fk8AyTirA7
---
Title: Rethinking prototype projection in interpretable-by-design classifiers with LLM optimization
Abstract: Prototype neural networks are the most popular form of interpretable-by-design classifiers in machine learning.
Within this field, prototypes are typically learned as black-box vectors in the model's penultimate latent space, and then projected onto the nearest neighbor example from the training data for visualization and inference purposes.
This fosters interpretability because we can understand the logic behind predictions based on the similarity between the input instance and the nearest prototype in the network.
However, because these projected prototypes are real training instances, there are at least two major issues with this approach.
Firstly, because they do not perfectly match the learned black-box vectors, there is typically a change in the model's decision boundary and hence a performance drop-off.
Secondly, because they are real training instances, they are usually overly specific and full of spurious or irrelevant details, often making them difficult to interpret.
In this study, we address both of these problems by using large language models (LLMs) as a tool for optimization to generate better prototypes for projection.
Results show our method produces more concise explanations (i.e., the prototype length and number of concepts present) compared to baselines which project to nearest neighbors in the training data, and that when these same baselines induce a large drop in accuracy, this can be largely mitigated.
Overall, our approach suggests an alternative methodology for prototype projection which may help make interpretable-by-design classifiers more intelligible and accurate.
URL: https://openreview.net/forum?id=kH0UxAyGdF
---
Title: Identifiability in Noisy Label Learning: Theory, Limits, and a Practical Approximation
Abstract: Learning from noisy labels (LNL) is fundamental in deep learning, where a common strategy is to identify clean-label samples from imperfectly annotated data. However, the standard LNL setting with only one noisy label per instance is non-identifiable, meaning clean labels cannot be theoretically recovered without additional heuristics. We show that LNL becomes identifiable when at least $2C - 1$ i.i.d. noisy labels per instance are available, where $C$ is the number of classes. To meet this requirement without extra manual annotation, we propose a data-driven method that leverages nearest neighbours to automatically generate additional i.i.d. noisy labels. Our algorithm estimates local noisy-label distributions in feature space and samples from them, yielding a finite-sample approximation that empirically improves clean-label recovery, though without guaranteed identifiability in practice. These labels are then integrated into the Expectation-Maximisation algorithm to infer clean labels. Experiments on synthetic, web-controlled, and real-world benchmarks demonstrate accurate and robust clean-label estimation.
URL: https://openreview.net/forum?id=hGMd4nCKyK
---
Title: Explainable Visual Forgery Detection: A Survey
Abstract: The rapid growth of AI-driven image manipulation technologies poses critical challenges for verifying content authenticity. While many forgery detection systems achieve high accuracy, their black-box nature limits deployment in high-stakes domains that demand transparency and explainability. This survey presents the first comprehensive review of explainable forgery detection in images and videos, introducing a novel taxonomy structured around three dimensions: Forgery Localization (FL), which pinpoints manipulated regions; Forgery Attribution (FA), which identifies manipulation sources; and Forgery Judgment Basis (FJB), which clarifies decision reasoning. We systematically analyze 51 state-of-the-art methods across single-modal and multi-modal settings, examining architectural innovations and explainability mechanisms. Four feature-driven strategies (RGB, frequency-domain, noise-texture, and representation learning) are reviewed in detail, highlighting their complementary strengths. Benchmark datasets and evaluation protocols are also compared, and open challenges are identified, including the need for standardized explanation formats, uncertainty quantification, and broader dataset coverage. By establishing this taxonomy and synthesizing recent progress, this survey lays a foundation for developing transparent and trustworthy forgery detection systems, supporting real-world applications in forensic analysis, news verification, and regulatory compliance.
URL: https://openreview.net/forum?id=XPF3JQJ5x6
---
Title: Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale
Abstract: The rapid proliferation of Claude agent skills has raised the central question of how to effectively leverage, manage, and scale the agent skill ecosystem. In this paper, we propose \emph{AgentSkillOS}, the \textbf{first} principled framework for skill selection, orchestration, and ecosystem-level management. \emph{AgentSkillOS} comprises two stages: (i) \textbf{Manage Skills}, which organizes skills into a capability tree via node-level recursive categorization for efficient discovery; and (ii) \textbf{Solve Tasks}, which retrieves, orchestrates, and executes multiple skills through DAG-based pipelines. To evaluate the agent's ability to invoke skills, we construct a benchmark of 30 artifact-rich tasks across five categories: data computation, document creation, motion video, visual design, and web interaction. We assess the quality of task outputs using LLM-based pairwise evaluation, and the results are aggregated via a Bradley--Terry model to produce unified quality scores. Experiments across three skill ecosystem scales ($200$ to $200\text{K}$ skills) show that tree-based retrieval effectively approximates oracle skill selection, and that DAG-based orchestration substantially outperforms native flat invocation even when given the identical skill set. Our findings confirm that structured composition is the key to unlocking skill potential. All code and data are available at: https://anonymous.4open.science/r/AgentSkillOS-Anonymous-B154.
URL: https://openreview.net/forum?id=HlRye8dWCd
---
Title: Regularize or Localize: When Training-Time KV-Cache Ge- ometry Pays Under Quantization
Abstract: We study whether SIGReg—LeJEPA’s anti-collapse objective—can reshape representations during standard autoregressive language-model pretraining, and when the resulting geometry helps K/V-cache quantization. We train 110M-parameter models on 10B FineWeb tokens and report three findings. (1) At λ=0.01, SIGReg reduces hidden-state pairwise-cosine anisotropy by 38% across three paired seeds. Perplexity increases by less than 0.35% in every pair, with no consistent zero-shot loss. (2) This change does not propagate from
hidden states to the K/V cache. Applying SIGReg directly to K and V during continued training, however, reduces mean cache anisotropy by 94% across four checkpoints. A matched continuation without the K/V term leaves cache geometry nearly unchanged, and the frozen-
trunk retrofits we tested do not reproduce the effect. (3) Under untransformed symmetric group-free quantization, direct K/V regularization is the only training condition that prefers per-channel scaling in all three seeds, and under that same 3-bit per-channel scheme the
baseline incurs 4.3–7.9× the directly regularized model’s ∆NLL. Under the full simulated KIVI-style configuration (mixed arrangement, zero-points, grouped scales), however, all models reach near-parity, including when storage overhead is approximately matched. In this
110M regime, the training intervention helps when quantizer scales are coarse; the advantage vanishes under the tested combination of token-local grouping, mixed K/V scaling, and zero-points. To our knowledge this is the first training-time distributional regularization of
standard K/V-cache geometry evaluated against post-hoc cache quantization.
URL: https://openreview.net/forum?id=vPYlyHPomp
---
Title: A Theory of the Effect of Overparameterization on the Loss of Plasticity in Continual Learning
Abstract: The paradigm of continual learning has garnered significant attention due to its inherent characteristics that mirror real-world learning processes. Given the promising performance of deep neural networks in tackling complex tasks, several works have studied them when facing a sequence of tasks. These studies provided empirical evidence suggesting that neural networks lose their ability to learn new tasks as the sequence of tasks or the number of parameter updates increases. This phenomenon is called Loss of Plasticity. Motivated by the surprising effects of overparameterization in training and generalization, we aim to identify the impact of overparameterized models on the loss of plasticity in continual learning settings. We start by establishing a theoretical understanding of loss of plasticity for both overparameterized linear and non-linear models in continual regression tasks, verifying that overparameterization benefits the model in sustaining its plasticity. Founded on this theoretical understanding, we empirically analyze the effect of overparameterization in a broader class of problems, from rich continual supervised learning to continual reinforcement learning. The extensive results corroborate our theoretical findings, demonstrating improved plasticity of models.
URL: https://openreview.net/forum?id=IoDo2eQske
---
Title: Optimal Thresholding and Tilted Objectives for Reducing Predictive Multiplicity
Abstract: The existence of multiple equally accurate models for the same dataset, known as the Rashomon effect, has recently attracted the attention of the machine learning community. The multiplicity of models allows practitioners to select accurate models while also satisfying other objectives, such as fairness or interpretability. However, disagreements among models on individual samples, measured by ambiguity, can reduce the credibility of decisions. While ensemble and reconciliation methods mitigate ambiguity by aggregating multiple predictors, they do not address the source of instability within individual models. In this work, we theoretically study the ambiguity of models obtained from a randomized training procedure by relating it to the distributions of scores around the decision threshold and residuals, and, in particular, to the occurrence of large residuals. Based on our theoretical results, we present a simple yet effective approach for threshold selection that reduces ambiguity at a low cost to accuracy. We also present an adapted loss for binary classification that reduces ambiguity by controlling the tail of the residual distribution. Our experiments on five datasets confirm that ambiguity is governed by the tail behavior of residuals and decision density. By regulating these properties, we demonstrate that practitioners can obtain low-ambiguity classifiers when compared with traditional methods.
URL: https://openreview.net/forum?id=6yVCUeBasx
---
Title: Evaluating Tool-Augmented Skill Induction in Web-Based Agents
Abstract: Browser-based agents powered by Large Language Models (LLMs) traditionally rely on DOM interactions, which suffer from high latency and computational inefficiencies. In this work, we analyze the intersection of structured tooling via the Model Context Protocol (MCP), and agent skill architectures via the autonomous Agent Skill Induction (ASI) framework, to evaluate how each abstraction layer intersects with the agentic reasoning loop. Evaluating across a subset of $n=187$ tasks on the WebArena benchmark using Claude Sonnet 4.6, we demonstrate that MCP integration alone yields a significant capability leap over the DOM-structured baseline, increasing task success rates from 51.3\% to 60.4\% while simultaneously halving token consumption. However, applying ASI over this structured action space exposes a critical capability-efficiency trade-off; while ASI provides marginal gains under the DOM regime, the sequential MCP+ASI configuration triggers a 4.3 percentage point regression in raw success rate. Through a systematic failure analysis, we then demonstrate that 42.7\% of recorded task failures (under the MCP+ASI architecture) are artifacts of the intrinsic structural limitations present in WebArena evaluation, which fundamentally fails to capture true agentic capability. After isolating for genuine agentic reasoning failures, trajectory analysis traces the MCP+ASI regression to the abstraction layer. While DOM-level abstraction natively retains environmental observability, naive API-layer ``skill induction'' frequently discards raw application state, leading to semantic locking, contextual data destruction, and brittle control-flow hijacking resulting in subsequent task failures. Our empirical findings establish the structural boundaries of baseline macro-generation, further supporting the shift of the architectural SOTA (State of the Art) toward more dynamic, externalized memory structures. Our work demonstrates that delegating core logic to static macro ``skills'' prematurely truncates the agent's reasoning footprint, introducing a regression in agentic performance.
URL: https://openreview.net/forum?id=5pSfQgr4JJ
---
Title: Soft Probabilistic Discrete Minimax Classifier
Abstract: Class imbalance or prior probability shift can significantly degrade classifier performance by inducing large disparities among class-conditional risks, especially in non-separable settings where the global risk strongly depends on class priors. We propose the Soft Probabilistic Discrete Minimax Classifier (SPDMC), a deterministic profile-based framework that balances empirical class-conditional risks via prior optimization. Building on soft probabilistic discrete Bayesian classification, the method combines profile-based posterior approximation with an iterative risk-driven update of class priors, while remaining interpretable, efficient, and compatible with arbitrary loss matrices. In the binary case, the empirical risk difference varies monotonically with the prior, enabling a globally optimal bisection procedure; in the multiclass case, the prior simplex admits a finite partition with piecewise-constant risks, motivating a projected update algorithm. Experiments on imbalanced binary and multiclass datasets show that the SPDMC achieves superior risk balance and improved robustness under prior shift by reducing class-wise risk variance.
URL: https://openreview.net/forum?id=YK3CRhAKOH
---
Title: WaveLoc: Wavelet-Augmented Hybrid Vision Encoders for AI-Generated Image Forgery Localization
Abstract: Advances in generative AI have made it increasingly difficult to determine whether regions of an image are authentic or artificially manipulated, raising serious concerns for media integrity and digital forensics. Modern diffusion-based inpainting models can produce manipulated regions that are increasingly difficult to distinguish from genuine content, exposing the limits of methods that rely on visible artifacts or hand-crafted cues. Existing approaches either fail to generalize across generators or resort to large-scale foundation models whose parameter counts limit practical use. A gap shared by these methods is the limited use of frequency-domain analysis within a compact architecture that combines complementary spatial and global representations. We propose WaveLoc, a hybrid multi-encoder architecture that integrates frequency-aware feature learning into two complementary encoders: a convolutional encoder that captures fine-grained local structure and a Vision Transformer that models global context. Within each encoder, we apply WTConv - Wavelet Transform Convolution (Finder et al., 2024) at the stage where it is most effective - the first residual block of the CNN and the reshaped feature map of the ViT - so that wavelet decomposition operates on learned representations rather than raw inputs. Our ablations show that these gains are driven primarily by the Vision Transformer branch: applying WTConv to its reshaped feature map improves localization more than applying it within the CNN, indicating that learned global transformer representations retain frequency-domain structure that wavelet analysis can explicitly exploit. The two streams are fused in a decoder and refined to full resolution. Evaluated on the OpenSDI benchmark (Wang et al., 2025) - which was introduced alongside the MaskCLIP baseline method - WaveLoc attains an average F1 comparable to this much larger foundation-model baseline across five leading diffusion generators while using roughly 6$\times$ fewer parameters. Among the more recent generators, it achieves the strongest F1 on SDXL and FLUX.1 and closely matches the best F1 on SD3, indicating that targeted frequency-aware design can be a competitive alternative to scaling alone.
URL: https://openreview.net/forum?id=c2fPsf4r9D
---
Title: Preemptive Detection and Steering of LLM Misalignment via Latent Reachability
Abstract: Large language models (LLMs) are now ubiquitous in everyday tools, raising urgent safety concerns about their tendency to generate harmful content. The dominant safety approach -- reinforcement learning from human feedback (RLHF) -- effectively shapes model behavior during training but offers no safeguards at inference time, where unsafe continuations may still arise. We propose BRT-Align, a reachability-inspired framework that brings control-theoretic safety tools to LLM inference. BRT-Align models autoregressive generation as a dynamical system in latent space and learns a safety value function via backward reachability, estimating future unsafe evolution under a learned latent-space abstraction. This enables two complementary mechanisms: (1) a \textit{runtime monitor} that forecasts unsafe completions several tokens in advance, and (2) a selective \textit{steering filter} that perturbs latent states only when the learned safety value indicates elevated risk, redirecting generation away from unsafe regions. Experiments across multiple LLMs and toxicity benchmarks show that BRT-Align is substantially more balanced than prior latent-control monitors and, unlike reactive classifier references, detects unsafe trajectories several tokens before unsafe text appears, enabling preemptive intervention. Moreover, for LLM safety alignment, BRT-Align substantially reduces unsafe generations while preserving sentence diversity and usable coherence. Qualitative results further highlight that BRT-Align often produces responses that are less violent, less profane, and less offensive. Together, these findings show that a reachability-inspired formulation provides a practical foundation for preemptive inference-time LLM safety.
URL: https://openreview.net/forum?id=XQzALvE9li
---
Title: UniDoc: Towards Unified Thinking in Documents
Abstract: Answering questions over long, visually rich documents demands both cross-page struc-
tural reasoning and fine-grained visual understanding. Existing approaches sacrifice one for
the other: retrieval-augmented methods fragment documents and lose inter-page structure;
monolithic vision-language models preserve structure but scale poorly beyond a few pages.
This impasse stems from a false premise: treating the macroscopic task of evidence retrieval
and the microscopic task of detailed reasoning identically. Instead, by decoupling their per-
ceptual resolutions instead of their architectures, a single model can achieve the structural
holism of monolithic processing alongside the efficiency of targeted retrieval. UniDoc unifies
both capabilities in a single model through two resolution-adaptive stages: a Fast Read-
ing stage processes all pages at low resolution to structurally identify evidence pages, and
a Focused Thinking stage re-processes only those pages at high resolution for multimodal
inference. To train this pipeline, we construct 30K multi-hop reasoning instances with stage-
level supervision. On MMLongBench-Doc, UniDoc improves Qwen2.5-VL-7B by 11.8 points
and Qwen3-VL-8B by 7.6 points, while significantly reducing token cost relative to uniform
high-resolution processing. Error analysis identifies page retrieval and fine-grained visual
reasoning as the remaining bottlenecks, delineating concrete targets for future wor
URL: https://openreview.net/forum?id=c4HyrJIiEa
---
Title: Variance-sensitive Thompson sampling for generalised linear bandits, revisited
Abstract: We prove a variance-sensitive regret bound for Thompson sampling in stochastic generalised linear bandits. The argument assumes a warm-up, after which the regret is controlled through using the Gaussian Poincar\'e inequality. This bypasses the point at which previous optimism-based analyses break down. Removing the warm-up while retaining the same variance-sensitive scaling remains open, and appears nontrivial.
URL: https://openreview.net/forum?id=5yNZjXKgJI
---
Title: When Should the Agent Speak? A Survey of Intervention Timing for Always-On AI Assistants
Abstract: Capable actors have arrived: agents complete multi-step tasks against verifiable goals, and egocentric perception resolves user intent with high accuracy. What remains unmodeled is the decision of when to act unprompted. We survey intervention timing for always-on assistants (smart glasses, MR headsets, ambient copilots) around a single decision rule, intervene iff the expected benefit of acting exceeds the expected cost of interrupting, and organize the literature into five layers: signals, decision, action, memory, and evaluation. We reconnect two lineages that currently do not cite each other: the 1999-2017 interruptibility literature, which formalized interruption cost rigorously but had no capable actor, and the 2024-2026 proactive-agent wave, which has actors but rediscovers the cost term only in fragments (false-alarm pricing, cognitive load, social violation, compute) that no work unifies. We argue that evaluation is the gating layer. Until intervention quality is measurable, a reinforcement-learning reward for proactivity cannot be defined. We therefore propose the design of a benchmark for open-world intervention timing with an explicit cost term. Six open problems close the survey.
URL: https://openreview.net/forum?id=b0yKEdAXEr
---
Title: OSHO-CCA: Orthogonal and Scalable High-Order Canonical Correlation Analysis
Abstract: Canonical Correlation Analysis (CCA) is a classical technique for learning shared repre-
sentations from two views of data by maximizing the correlation between the resulting
representations. Existing extensions to more than two views either maximize pairwise cor-
relations, sacrificing higher-order structure, or model high-order interactions at the expense
of orthogonality and scalability. Focusing on the linear multiview CCA setting, we propose
OSHO-CCA, a novel linear method for Orthogonal and Scalable High-Order CCA that
jointly addresses all three desiderata: (1) it captures high-order dependencies across views,
(2) enforces orthogonality among projected features to ensure decorrelated embeddings, and
(3) scales efficiently with the number of views. We further introduce a new evaluation metric
for Total Canonical Correlation (TCC) that generalizes traditional two-view CCA metrics to
the multiview setting. Experiments on real and synthetic datasets demonstrate that OSHO-
CCAachieves strong correlation maximization and competitive downstream classification
performance relative to existing methods, while maintaining scalability and orthogonality
even in challenging multiview scenarios
URL: https://openreview.net/forum?id=dS5vDXjE1D
---
Title: Why Does Train–Validation Separation Emerge? Update-Pressure Density Dynamics in Pretrained Backbones
Abstract: Train--validation separation occurs when training performance continues to improve while validation performance improves more slowly, saturates, or decreases. We study why this separation emerges. Our experimental protocol uses pretrained-backbone fine-tuning to control initialization variability: because optimization begins from a structured source-learned state rather than a diffuse random representation, the initialization condition is held fixed, so gap formation can be analyzed through subsequent changes in support use. This controlled starting point lets us analyze the gap as a transfer-learning phenomenon in which update pressure shifts from broadly reusable support toward narrower downstream patterns that fit the observed training examples but transfer less efficiently to validation examples. To make this support-narrowing mechanism observable, we fix a set of training examples as probes. At each record time, a fixed backbone monitor returns a nonnegative update-pressure vector for each probe over task-adjacent interface coordinates. After $\ell_1$ normalization, the primary observable $D_{\mathrm{den},t}$ is the average squared dispersion of these probe pressure distributions around their cross-probe mean. Low $D_{\mathrm{den},t}$ means that probes place pressure on similar monitored coordinates; high $D_{\mathrm{den},t}$ means that their pressure allocations are more heterogeneous. We first test the mechanism in a controlled five-level hierarchy simulation with a pretrained ResMLP-style backbone and multiple private-support mixture settings, where support levels are constructed and train/validation masked-loss transfer can be tracked directly. We then apply the same density-dispersion functional to four pretrained NLP backbones over a common six-dataset roster. Across RoBERTa-base, Qwen3-1.7B, DeepSeek-R1-Distill-Qwen-1.5B, and DeBERTa-base, the association between $D_{\mathrm{den},t}$ and the accuracy gap is consistently positive and strong: all 120/120 reported real-data runs have positive run-level direction, and scalar $D_{\mathrm{den},t}$ LOWESS regressions give strong gap-level fits. Together, the controlled support manipulation and the real-data backbone study provide a theory-guided, directly computable, and falsifiable mechanism account of train--validation separation, tested through a controlled-initialization fine-tuning protocol.
URL: https://openreview.net/forum?id=O4m5eAJ9Z1
---
Title: LLM-as-Judge Data Qualification: Calibration Collapse and Bootstrap Bounds
Abstract: LLM-as-Judge pipelines now generate the preference data behind most post-training alignment methods, including Constitutional AI, RLAIF, and self-play fine-tuning, all of which route the training signal through a model evaluating its own outputs. When that model is simultaneously being fine-tuned on those evaluations, self-preference biases compound across iterations. Whether the resulting loop converges or collapses had no prior formal answer. We model the judge-student loop as a discrete-time Markov chain and prove a sharp stability threshold: the pipeline remains well-behaved as long as the judge's self-preference bias stays below a critical level that depends on the training temperature and how far the model is allowed to drift from its reference version. Above that threshold, the loop collapses into a degenerate state where the judge can no longer tell good outputs from bad ones, a state that is simultaneously a stable equilibrium and a textbook case of Goodhart's law, where optimizing a proxy measure destroys its value as a signal. We bound the key stability constant using the singular values of the LoRA fine-tuning matrices, giving certified safe and collapse zones without needing to know the model's true sensitivity in advance. We also show the result holds up under realistic model misspecification, with an explicit formula for how long a near-collapse state can persist before fully collapsing. We further derive bootstrap-based confidence intervals for a label-free proxy of calibration quality, and show these intervals remain valid even when true human labels are replaced by another AI judge. Finally, we provide a sequential monitoring method that can detect the onset of collapse in near-optimal time. The framework yields concrete, testable predictions for two widely used open models, Llama-3-8B and Mistral-7B, across a range of training temperatures.
URL: https://openreview.net/forum?id=klAuJebJaX
---
Title: From Prompts to Perception: Auditing Stereotypes in Multimodal AI
Abstract: This work presents a model-agnostic framework for auditing stereotype-related associations across end-to-end T2I-to-MLLM pipelines. Across four T2I models and five MLLMs, demographic conditioning produced systematic shifts in generated visual traits, which subsequently influenced predictions of occupations and interests. We observed nationality-specific clustering in the embedding space, alignment of gender-neutral prompts with male representations, strong concentration on a limited set of predicted labels, and recurring demographic patterns in both closed- and open-vocabulary settings.
To quantify these behaviors, we introduced TDS and WTD for measuring trait-distribution shifts, LDI and OM for summarizing label dominance and cross-group overlap, and MCS for capturing corruption-induced semantic changes. The main findings remained stable across alternative divergence measures. The human study and controlled trait-recovery experiments provided additional validation of the trait-labeling process, while also highlighting the subjectivity of attributes such as skin tone and face shape. The corruption analysis further showed that modest changes in image quality can alter semantic predictions and the recovery of prompt-specified nationality, including occasional shifts to socially sensitive labels. Similar patterns on the UTKFace dataset suggest that these behaviors are not limited to generated images.
Overall, the proposed framework provides a practical and extensible approach for identifying where stereotype-related associations emerge, how they vary across demographic groups and models, and how stable they remain under image degradation. The metrics can support model comparison, post-deployment monitoring, and future mitigation studies, for example, by identifying excessive label concentration or unstable socially consequential predictions. We hope this work encourages broader and more systematic evaluation of multimodal systems across additional demographic groups, attributes, languages, and real-world conditions.
URL: https://openreview.net/forum?id=yPGqiQjSjj
---
Title: Machine Learning Research Has Outpaced Its Communication Norms
Abstract: Machine learning research has grown faster than its communication norms. We measure this gap across NeurIPS (30,595 papers, 1987 to 2025), ICLR (16,807 papers, 2013 to 2026), and ICML (15,226 papers, 2008 to 2025), against baselines of 2.8 million arXiv papers and 24.3 million PubMed abstracts. We apply five metric families: classical readability scores, a writing style suite, acronym density and reuse, LLM judge ratings, and citation counts. Mean Flesch Reading Ease on NeurIPS abstracts falls from approximately 24 in 1987 to approximately 8 in 2025, with ICLR and ICML on the same downward path. Readability is declining across science, but not equally. Between 2015 and 2025 the three conferences lose 49 to 62 percent of their Flesch Reading Ease, against 34 percent for the non-ML arXiv control and 27 percent for PubMed, and by 2025 every conference sits 4.7 to 10.0 points below both. Sensational language on NeurIPS rises by approximately 80 percent between 2015 and 2025, again outpacing the non-ML control. Title acronym density has climbed at all three conferences. In 2019 they ranged from 1.6 to 2.8 per 100 words, bracketing the non-ML control at 2.2. By 2025 they reach 3.5 to 4.2 against 2.8 for that control. Roughly 88 to 90 percent of ML acronyms appear fewer than ten times, against 79 percent science-wide, so most never become shared vocabulary. LLM judges do not detect this decline. On NeurIPS and ICML they rate recent abstracts as easier to read, the opposite of every classical metric, underscoring that these scores target human readers rather than automated ones. More readable papers tend to receive more citations within their venue and year at all three conferences, where the top cited decile reads easier than the bottom decile on twelve of fifteen readability metrics, so the readability decline plausibly compounds across the literature. We propose seven measurable writing standards that any of these venues could pilot to improve readability for human readers.
URL: https://openreview.net/forum?id=RoBtrexUvK
---
Title: MASED: Multi-scale Adaptive Structural Evidence Decoding for Mitigating Object Hallucination in Large Vision-Language Models
Abstract: Large Vision-Language Models (LVLMs) have made substantial advances in multimodal understanding and reasoning, but object hallucination remains a major challenge. One underlying issue is that models may assign high confidence to object predictions based on local visual cues without sufficient cross-patch structural support, allowing unreliable visual evidence to influence decoding. We use multi-scale patch shuffling to probe this issue. By disrupting cross-patch relationships at different scales while largely preserving local content, we find that some predictions remain order-consistent along the structural perturbation axis, whereas others become unstable. Based on this observation, we propose Multi-scale Adaptive Structural Evidence Decoding (MASED), a training-free method for gate-controlled token-distribution calibration at inference time. A Jensen--Shannon (JS) evidence gate determines whether the current prediction shows reliable structural reliance, thereby routing decoding to either multi-scale structure-disrupted views or a Gaussian-noised detail-disrupted view. MASED then performs complementary or contrastive calibration according to the distributional response of the selected visual branches. Experiments on POPE, MME, MMHal-Bench, and CHAIR demonstrate that MASED reduces object hallucination while preserving the general multimodal capabilities of LVLMs.
URL: https://openreview.net/forum?id=z1fai5wM5I
---
Title: Combining Stochastic and Deterministic Velocities for Probabilistic Time-Series Prediction
Abstract: Accurate uncertainty quantification is critical for many time series forecasting applications, and recent studies have increasingly adopted generative models to capture predictive uncertainty. Existing approaches often combine deterministic forecasting with generative modeling to improve predictive performance. However, their joint training is typically unstable because the deterministic model is embedded within the generative process, resulting in deep computational graphs and changes to the probability transport path. In this work, we propose DV-Flow, a unified flow-matching formulation that incorporates the deterministic predictor as a parallel velocity component rather than the endpoint of the transport path. This design enables stable end-to-end optimization while subsuming several existing formulations as special cases, allowing direct comparisons among them. Extensive experiments on six real-world datasets demonstrate that DV-Flow consistently outperforms strong baselines in both point forecasting accuracy and uncertainty calibration.
URL: https://openreview.net/forum?id=dDEDwFnnoA
---
Title: A Practical Investigation of Training-free Relaxed Speculative Decoding
Abstract: Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM’s sampling distribution. Recent work argues that relaxing this strict guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains. We practically investigate training-free relaxed speculative decoding techniques, unify existing approaches within a shared framework, benchmark them on contemporary settings, and distil takeaways and empirical findings for practitioners. Important takeaways include: relaxation can require considerable capability evaluation unlike lossless speculative decoding, and many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters. Code will be available upon publication.
URL: https://openreview.net/forum?id=jeN0PKCYZE
---
Title: Learning under Random Graph Shifts: A Generalization Theory in Truncated-Spectrum vRKHS
Abstract: This paper develops a theory of graph classification under domain shift through a random-graph generative lens, where we consider intra-class graphs sharing the same random graph model (RGM) and the domain shift induced by changes in RGM components. While classic domain adaptation (DA) theories have well-underpinned existing techniques to handle graph distribution shift, the information of graph samples, which are itself structured objects, is less explored. The non-Euclidean nature of graphs and specialized architectures for graph learning further complicate a fine-grained analysis of graph distribution shifts. In this paper, we propose a theory that assumes RGM as the data generative process, exploiting its connection to hypothesis complexity in function space perspective for such fine-grained analysis. Building on a vector-valued reproducing kernel Hilbert space (vRKHS) formulation, we derive a generalization bound whose shift penalty admits a factorization into (i) a domain discrepancy term, (ii) a spectral geometry term summarized by the accessible truncated spectrum, and (iii) an amplitude term that aggregates convergence and construction-stability effects. We empirically verify the insights on these terms in both real data and simulations.
URL: https://openreview.net/forum?id=1OHULCOsh1
---
Title: XPlore: Full-Space Counterfactual Explanations for Graph Neural Networks
Abstract: Graph Neural Networks (GNNs) are increasingly adopted across domains such as molecular biology and social network analysis, yet their black-box nature hinders interpretability and trust. This is especially problematic in high-stakes applications, such as predicting molecule toxicity, drug discovery, or guiding financial fraud detections, where transparent explanations are essential. Counterfactual explanations offer a transparent lens into GNNs' behavior. In this work, we introduce XPlore, a novel technique that significantly broadens the counterfactual search space. It consists of gradient-guided perturbations to adjacency and node feature matrices. Unlike most prior perturbation methods, which focus solely on edge deletions, our approach belongs to the growing class of techniques that optimize edge insertions and node-feature perturbations, here jointly performed under a unified gradient-based framework, enabling a richer and more nuanced exploration of counterfactuals. To quantify both structural and semantic fidelity, we introduce a cosine similarity metric for learned graph embeddings that addresses a key limitation of traditional distance-based metrics, and demonstrate that XPlore produces more coherent and minimal counterfactuals. Empirical results on 13 real-world and 5 synthetic benchmarks show up to +56.3% improvement in validity and +52.8% in fidelity over state-of-the-art baselines, while retaining competitive runtime.
URL: https://openreview.net/forum?id=8pamMUOb6n
---
Title: A Scalable Hybrid Approach for Bayesian $L_q$ Sparse Regression
Abstract: Bayesian sparse regression provides built-in uncertainty quantification but faces severe computational and mathematical challenges in high dimensions, particularly with non-convex priors. We introduce a scalable hybrid stochastic pseudo-sampler for $L_q$ penalized regression ($q \in (0, 1]$). Our method updates the high-dimensional coefficient vector via a tempered proximal stochastic gradient Langevin dynamics (Prox-SGLD), paired with an adaptive, validation-driven Metropolis-Hastings update for the hyperparameters. We demonstrate that rigorous implementation requires deriving the prior's log-normalization constant alongside a dimension-scaled hyperprior to mathematically prevent runaway over-shrinkage. Crucially, standard inference for continuous shrinkage faces a structural limitation such that exact block updates scale cubically ($\mathcal{O}(p^3)$), while computationally cheaper single-site samplers suffer from coordinate-wise locking and predictive failure. Our Prox-SGLD framework circumvents this dichotomy, updating the full coefficient vector simultaneously via first-order gradients to scale efficiently at $\mathcal{O}(np)$. Evaluated across simulations and clinical data (NHANES), the flexible $L_q$ family allows our sampler to adapt directly to the underlying data topology. Non-convex fractional penalties ($q \le 1/2$) outperform benchmark priors (Horseshoe, Bayesian Bridge, and Spike-and-Slab) in exact sparse signal recovery, whereas the convex Bayesian Lasso ($q=1$) proves predictively superior for dense, poly-factorial real-world traits.
URL: https://openreview.net/forum?id=1Meb7laNMU
---
Title: Threshold-Cascaded Lexicographic Rewards for Multi-Constrained RL in Real-Time Bidding
Abstract: Multi-objective reinforcement learning under heterogeneous constraints is a recurring challenge in real-time bidding (RTB), where an agent must simultaneously satisfy budget pacing, margin, click-through rate, and other operational targets. The dominant approach to constrained MDPs (the Lagrangian dual method) generates persistent dual-variable oscillations under periodic non-stationarity, an instability that is structurally worst when the environment varies slowly relative to the dual learning rate, the regime characteristic of intra-day RTB.
We propose Threshold-Cascaded Lexicographic (TCL) rewards: a reward-shaping framework that encodes a soft priority ordering over multiple constraints via a product of sigmoid gates, with no dual variable. We establish four theoretical results. First, the Lagrangian dual amplitude scales as $\mathcal{O}(\alpha/\omega)$ (unbounded as the variation frequency $\omega \to 0$), while TCL's gate signal carries no integrator and is asymptotically independent of $\omega$. Second, as the cascade gain grows, accumulation points of TCL-optimal policies solve a threshold-cascaded lexicographic constrained MDP, with approximation error decaying exponentially in the gain and in the policy's margin from the satisfaction thresholds. Third, the TCL policy gradient naturally suppresses lower-priority objectives until higher-priority constraints are met, with a Lipschitz constant that grows quadratically in the gain, motivating a gain-annealing schedule. Fourth, when the base reward is uniformly negative (loss-budget regime), the shaped variant exhibits a reward inversion above an explicit critical gain, which we characterize formally and eliminate via a constant reward shift.
We validate TCL on the AdCraft RTB simulator and on a constrained pendulum benchmark. Formulation (A) (the standalone additive cascade) achieves a primary-constraint satisfaction rate (CSR) of $1.000\pm0.000$ on all AdCraft loss-budget scenarios and $0.965\pm0.020$ on the constrained pendulum. In slowly non-stationary (slow-drift) scenarios, Formulation (B) (the multiplicatively shaped variant) achieves the highest CSR ($0.928\pm0.172$, 10 seeds) among all methods, while Lagrangian, fixed-weight, and hierarchical potential-based shaping baselines collapse or remain seed-dependent (CSR $\leq 0.531$). The critical gain $\beta^\star$ predicted by our closed-form characterization and the threshold-sensitivity asymmetry identified in (C4) are confirmed quantitatively; the minimal reward shift eliminates the loss-budget failure mode in all tested configurations.
URL: https://openreview.net/forum?id=BtLMJhSkIt
---
Title: Eco-Fast SafetyAnalyst: A Reproducibility Study and Computational Scaling Analysis of Interpretable Prompt-Safety Moderation
Abstract: We reproduce SafetyAnalyst, an interpretable prompt-safety moderation pipeline that classifies prompts by generating structured harm–benefit trees and aggregating them into a harmfulness score $H$, thresholded at $H > 0$. We fine-tune HarmReporter and BenefitReporter student models on the released supervision data, fit aggregation weights on a balanced alignment set, and evaluate on three disjoint balanced WildJailbreak test sets, achieving F1 88.0, AUPRC 94.5, and AUROC 95.0, closely matching the original student-model results. Generated trees follow the expected schema but are denser than reported, indicating a structural distribution gap. We further find that the authors' released aggregation weights collapse to F1 58.4\% on our independently fine-tuned trees; this is partly a parameter-stripping bug in their evaluation script (corrected: 79.7\%) and partly a genuine generalization gap closed only by recalibration (88.0\%), showing recalibration is a practical necessity rather than an optional step. We study three extensions: (1) vectorized aggregation fitting, preserving the scoring rule while reducing optimizer overhead; (2) Top-$K$ stakeholder pruning, cutting fine-tuning emissions from 12.61 to 6.24 kg CO$_2$ while keeping F1 at 89.5\%. An ablation shows heuristic selection beats random selection by up to 4.5\% absolute F1, confirming the ranking itself, not just shorter trees, preserves performance; and (3) structured JSON decoding, which guarantees syntactic correctness but incurs a 58.6x latency slowdown and 65.7x throughput reduction versus standard decoding with retry loops. We conclude that structured constraints are too slow for real-time safety moderation, and standard decoding with repair loops remains the more practical paradigm.
URL: https://openreview.net/forum?id=u5yAuzTFpg
---
Title: On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode
Abstract: Factual hallucinations in language models are often plausible substitutions: the model gives the wrong city, the wrong author, or another entity of the correct semantic type. We study this failure at the first answer token by separating \emph{read} from \emph{write}. Read asks whether the gold answer token can be decoded from intermediate residual states, using decoys from the same relation to control for type priming. Write asks whether the final readout ranks that token first.
Across models, many failed generations remain readable even under a strict decoy criterion, yet the gold token is not selected. We explain this with a selection margin account. In many models, the main causal factor is weak support along the answer direction at the final readout: when we set this support to the level observed in successful generations, first token selection often changes. This intervention is not always sufficient. A baseline term, averaged over contexts and linked to token frequency, explains which plausible alternative is favored when answer support is weak, but removing frequency information alone rarely recovers the gold token. Component attribution and patching between paraphrases of the same fact show that a stable set of late components produces much of the margin.
These results isolate one cause of factual error: the model can carry decodable evidence for the correct answer, and generation can still begin with a frequent alternative of the same type, because the support for the answer is too small relative to the full margin at the final readout.
URL: https://openreview.net/forum?id=U7zU7eDjel
---
Title: Linear Convergence of FedAvg(GD) Under Constrained PL- Type Assumptions: A Single Hidden Layer Neural Network Analysis
Abstract: We study the optimization behavior of the widely used FedAvg algorithm for solving Federated Learning problems with deterministic local gradient descent updates. Existing linear-convergence analyses of FedAvg(GD) often rely on global structural assumptions such as strong convexity, global Polyak–\L ojasiewicz (PL) inequality, interpolation, bounded gradients, or bounded client heterogeneity. These assumptions can be difficult to verify for overparameterized neural networks and may not reflect the local geometry explored by the algorithm. We introduce a constrained PL-type framework in which gradient-dominance conditions are required only in a neighborhood of the initialization, while simultaneously controlling both client-side local trajectories and the averaged global iterates generated by FedAvg(GD). Under these localized conditions, we prove that FedAvg converges linearly in objective value to zero loss, that all relevant iterates remain inside the prescribed neighborhood, and that the iterates approach the zero-loss set. Moreover, the existence of a zero-loss solution in the localized region is obtained as a consequence of the assumptions and convergence analysis, rather than imposed separately as an interpolation assumption. We further specialize the framework to single-hidden-layer neural networks with squared loss. In this setting, the constrained PL-type conditions reduce to spectral conditions on the client-wise and aggregated Jacobian Gram matrices at initialization. The dimensional requirement $md>nK$, equivalently, $m>nK/d$, is necessary for the aggregated initialization Jacobian to be full row rank, while the sufficient convergence conditions are stated through explicit lower bounds on $\lambda_{\min}(\bm H_{k,0}\bm H_{k,0}^{\top})$ and $\lambda_{\min}(\bm H_{0}\bm H_{0}^{\top})$. Empirical results on MNIST, Fashion-MNIST, and CIFAR-10 support the predicted Jacobian eigenvalue scaling and illustrate stable FedAvg optimization behavior under heterogeneous client partitions.
URL: https://openreview.net/forum?id=xtIofjUiRF
---
Title: Forgetting in Capacity-Stressed Selective SSMs is Recall-Capability Loss
Abstract: Continual-learning methods for selective state-space models (Mamba) protect past tasks by
projecting gradients of the SSM input matrices out of a low-rank “protected” subspace. We
show this family plateaus, and explain why. Building a Courant–Fischer-optimal projector
from the scan-output Jacobian, we obtain a clean dissociation. On real gradients this projector
controls first-order output drift about an order of magnitude better than the standard
per-parameter projector (geometric-mean 11.5–19× across timestep aggregations), yet at
matched rank it forgets no less. A sequence of controlled probes (causal parameter transplant,
an operational memory-curve, and a paired collapse-probability comparison) shows
that forgetting is not governed by first-order drift; it is contraction of the model’s effective
recall capability: this capability is driven redundantly across multiple parameter channels,
only one of which gradient projection protects. The mechanism prescribes the remedy: exercise
the lost capability directly via a small budget of generic, synthetic long-range recall
examples (no old-task data). At 1–4% training overhead this prevents forgetting in these
settings and, in paired comparisons at this scale, outperforms the tested projection baselines
whose plateau the mechanism explains. The result replicates across two from-scratch model
scales (d=128, 256) and on a second, mechanistically distinct task family. We delineate the
scope of the result: the phenomenon requires the model to be capacity-stressed relative
to the task (so a pretrained 130M, which is over-capacity for MQAR, does not forget it),
and tasks below the architecture’s learnable horizon do not exhibit it. We also include a
methodology note on the measurement pitfalls that make the mechanism look inconsistent
until the metrics are corrected.
URL: https://openreview.net/forum?id=P94Yy9BdGj
---
Title: TOGA: Train-Once-Get-All Compression Framework for Efficient Large Language Model Inference
Abstract: Recently, the deployment efficiency of Large Language Models (LLMs) has become a critical concern in practical applications. While post-training quantization (PTQ) and structured pruning are established techniques for reducing memory footprint and inference latency, most existing PTQ approaches optimize quantization errors on a per-layer basis, overlooking how errors accumulate and propagate through the network and often resulting in suboptimal solutions. Traditional pipelines also tend to apply pruning and quantization in isolation or sequentially, further compounding this sub-optimality. We introduce a novel end-to-end framework that addresses these limitations in two key ways. First, we propose a mixed-precision PTQ strategy that directly minimizes global error propagation across the entire model, rather than isolating layer-wise errors. Building on this, we develop a joint optimization approach that simultaneously learns structural pruning decisions and mixed-precision quantization policies within a unified search space. Extensive experiments show that, at ultra-low precisions (e.g., 3 bits), our quantization method reduces WikiText perplexity by up to 21\% compared to state-of-the-art (SoTA) weight-activation quantization baselines. Against leading weight-only quantization methods, it achieves up to 59\% and 85\% lower perplexity on WikiText and C4, respectively. Compared to SoTA joint pruning-and-quantization techniques, our proposed method delivers superior perplexity and reasoning performance at ultra-low bits. Moreover, under mainstream precision configurations (e.g., 4-bit/8-bit), our compressed models remain highly competitive on WikiText perplexity and zero-shot reasoning accuracy, while achieving up to $3.23\times$ faster prefilling and $2.55\times$ faster decoding over FP16, and up to 64\% faster prefilling than models compressed by SoTA methods.
URL: https://openreview.net/forum?id=NZKkD2OK8R
---
Title: Is the Geometry Doing the Work? An Operating-Point Audit of Hierarchy in Hyperbolic Vision–Language Models
Abstract: Hyperbolic vision--language models are designed to encode abstraction geometrically: general concepts near the origin, specific ones farther out, and entailment cones representing directed order. We ask whether trained MERU, HyCoCLIP, and PHyCLIP models actually use these mechanisms. We audit seven released checkpoints and matched from-scratch interventions, using diagnostics designed to distinguish active hyperbolic geometry from angular structure and supervision-related effects. All audited converged checkpoints remain near-Euclidean in the dimensionless radius $u=\sqrt{c}\rho$, which measures how strongly embeddings experience hyperbolic geometry: the largest observed image-side value is $0.37$---well below $u\approx0.84$, where local metric distortion reaches $10\%$. Releasing the curvature floor changes curvature and norms but not this regime, with mixed, generally modest downstream shifts. Trained entailment cones are saturated or nearly saturated, so low violation rates can arise from trivially wide cones rather than learned order. A preregistered semantic-traversal analysis detects weak order when retrieval is restricted to the correct branch, but no operative readout over the full hierarchy. Shuffle-controlled tests detect no pair-specific radial ordering in released checkpoints, and no positive result is consistent across all three matched ViT-B seeds; remaining probes find at most small, conditional radial signals that do not establish a usable hierarchy beyond angular structure. We trace this behavior to a low-curvature shortcut: lowering curvature widens entailment cones and suppresses violations without learning order. In the probed training trajectories, gradient decomposition identifies entailment as the dominant curvature-lowering pressure during collapse. Yet curvature continues to contract when entailment is removed, so the shortcut is not the sole cause. Under our diagnostics, the audited formulations do not demonstrate an operative radial or cone-based hierarchy. We distill the audit into a five-number geometry report for evaluating future hierarchy claims.
URL: https://openreview.net/forum?id=Rj5ewBz62j
---
Title: Stochastic Overestimated Regularized Newton
Abstract: Modern optimization often involves degenerate, low-rank landscapes with non-isolated minima. We propose Stochastic Overestimated Regularized Newton (SORN), a Newton-type method that regularizes a stochastic overestimation of the Hessian to ensure stability in rank-deficient directions. Globally, SORN preserves an accelerated $O(n^{-2})$ global convergence rate when the expected optimality gap is large, transitioning to $O(n^{-1})$ as it approaches the noise floor of the stochastic approximation. Locally, we establish that SORN converges superlinearly for exact overestimation or linearly for inexact cases toward the set of non-isolated minima under the local Polyak-Lojasiewicz condition. We provide a practical variant using randomly pivoted Cholesky for rank-$k$ Hessian approximation, resulting in a low per-iteration cost of $O(\textup{dim} \cdot k^2)$. For entropic optimal transport with effectively sparse marginals, we show that the Hessian spectrum is as concentrated as the marginal mass, so a small rank parameter approximates the Hessian well and yields an improved linear convergence rate. Such marginals arise naturally in knowledge distillation between neural-network classifiers and in high-dimensional RNA-sequencing data, where SORN substantially outperforms the standard Sinkhorn algorithm and first-order methods in both precision and computational efficiency.
URL: https://openreview.net/forum?id=URxDMgi1Qd
---
Title: A Tale of Two Readouts: Unmasking Data Attribution through a Unified Kernel Framework
Abstract: Understanding which training examples drive a model's predictions is central to interpretability and data quality in deep learning. Exact leave-one-out retraining is prohibitive, and Hessian-based influence functions require convexity assumptions that do not hold in overparameterized networks. We present NTK-JL, a retrain-free training-data attribution framework that linearises a fine-tuned model via the empirical Neural Tangent Kernel and solves a kernel ridge regression in a compressed sketch space, obtained by random parameter subsampling followed by a Johnson--Lindenstrauss (JL) projection. For a model with $P$ parameters and training set size $n$, projecting down to dimensionality $s$ reduces gradient storage from $\mathcal{O}(nP)$ to $\mathcal{O}(ns)$ ($s \ll P$) while preserving inner products. Our central observation is that this unified empirical-NTK framework admits two readouts that recover two independently developed attribution families: the \emph{decomposition} readout reduces to Representer Point Selection (RPS) under a frozen backbone, confirmed empirically with Spearman $\rho = 0.97$ on frozen ViT-B/16, while the \emph{datamodel} readout is, under TRAK's own projected-gradient representation and Gauss--Newton curvature approximation, algebraically identical to single-model TRAK. Using the Linear Datamodeling Score (LDS), we show the two readouts answer different questions: the decomposition readout reconstructs the model's current prediction but has near-zero counterfactual fidelity, whereas the datamodel readout attains TRAK-level fidelity (LDS 0.63 on SST-2, 0.75 on QNLI). Proponent and opponent deletion experiments on DistilBERT/SST-2 and RoBERTa-base/QNLI further show that NTK-JL's fine-tuned gradients localise task-specific signal that frozen features used by RPS miss. Finally, the same framework yields self-influence scores at no additional cost, surfacing probable annotation errors in QNLI (2.8x enrichment over random) and phrase-level labeling ambiguity in SST-2 (3.2x enrichment), providing actionable data-quality signals.
URL: https://openreview.net/forum?id=UP6mrdDRge
---
Title: Communicating about Space: Language-Mediated Spatial Integration Across Partial Views
Abstract: Humans build shared spatial understanding by communicating partial, viewpoint-dependent observations. We ask whether Multimodal Large Language Models (MLLMs) can do the same, aligning distinct egocentric views through dialogue to form a coherent, allocentric mental model of a shared environment. To study this systematically, we introduce Cosmic, a benchmark for Collaborative Spatial Communication. In this setting, two static MLLM agents observe a 3D indoor environment from different viewpoints and exchange natural-language messages to solve spatial queries. Cosmic contains 899 diverse scenes and 1{,}250 question--answer pairs spanning five tasks. We find a capability hierarchy, MLLMs are most reliable at identifying shared anchor objects across views, perform worse on relational reasoning, and largely fail at building globally consistent maps, performing near chance, even for frontier models. Moreover, we find thinking capability yields gains in anchor grounding, but is insufficient for higher-level spatial communication. To contextualize model behavior, we collect 250 human--human dialogues. Humans achieve 95% aggregate accuracy, while the best model, Gemini-3-Pro-Thinking, reaches 72%, leaving substantial room for improvement. Moreover, human conversations grow more precise as partners align on a shared spatial understanding, whereas MLLMs keep exploring without converging, suggesting limited capacity to form and sustain a robust shared mental model throughout the dialogue.
URL: https://openreview.net/forum?id=G7OTeKt1eJ
---
Title: Item Clustering Inflates Statistical Significance in LLM Benchmark Comparisons
Abstract: Leaderboards compare large language models by aggregate benchmark accuracy, and gaps of one or two percentage points are routinely reported as improvements. Whether such a gap is real is a question of statistical inference --- yet the significance tests used in practice, including McNemar's test, treat a benchmark as an independent sample of items. Benchmark items, however, are often dependent by construction: like a survey, a benchmark groups items by subject, source, passage, or template, and items within a group tend to be answered together. Such clustering shrinks the effective sample size below the nominal item count, so a benchmark's \emph{size} is not the same as its \emph{information}. We quantify this empirically using the per-instance evaluation logs of the Open LLM Leaderboard, re-analysing 28 pairwise comparisons among eight open-weight models on MMLU-Pro, with no model inference required. The empirical design effect is large --- median 3.5, up to 11 --- leaving effective sample sizes 2--4$\times$ smaller than nominal. As a result, all 28 comparisons are significant under the standard independent-items test, but 3--4 become non-significant once clustering is accounted for (using a wild cluster bootstrap suited to the small number of clusters), including one whose naive $p$-value is $5\times10^{-6}$; the conclusions hold across two different clustering definitions, and the effect replicates on BBH and MATH. The same correction implies that detecting a one-point improvement needs roughly three times more evaluation items than independent-items calculations suggest, so MMLU-Pro's 12,032 items are too few for the gaps it is used to adjudicate. We provide a drop-in procedure that practitioners can apply to existing logs, and recommend reporting design-aware uncertainty and effective sample sizes as standard.
URL: https://openreview.net/forum?id=HaKpzjoXxf
---
Title: Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
Abstract: Reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) are important techniques to align large language models (LLM) with human preference. However, the quality of RLHF and DPO training is seriously compromised by corrupted preference, reward overoptimization, and bias towards verbosity. To our knowledge, most existing works tackle only one of these important issues, and the few other works require much computation to estimate multiple reward models and lack theoretical guarantee of generalization ability. In this work, we propose RLHF-COVand DPO-COV algorithms that can simultaneously mitigate these three issues, in both offline and online settings. This ability is theoretically demonstrated by obtaining length-regularized generalization error rates for our DPO-COV algorithms trained on corrupted data, which match the best-known rates for simpler cases with clean data and without length regularization. Moreover, our DPO-COV algorithm is simple to implement without reward estimation, and is proved to be equivalent to our RLHF-COV algorithm, which directly implies the equivalence between the vanilla RLHF and DPO algorithms. Experiments demonstrate the effectiveness of our DPO-COV algorithms under both offline and online settings.
URL: https://openreview.net/forum?id=SnrvAFH5J0
---
Title: PROACT: A Continual Learning Defense Against Forgetting Attacks
Abstract: Continual learning models are trained incrementally on a stream of non-stationary data. In such a setting, models are continually exposed to data poisoning attacks, and ensuring their reliability and robustness becomes particularly challenging. Indeed, recent work has shown that continual learners are susceptible to forgetting attacks, poisoning attacks designed to induce forgetting of previously learned tasks. This paper introduces PROjection and Activation Constrained Training (PROACT), the first continual learning method designed to be robust against forgetting attacks. PROACT combines three key components: model inversion to reconstruct proxy data from past tasks, gradient projection to remove interference and selectively activate weight regularization, and representation-level regularization to prevent representation drift. Extensive experiments across multiple weight-regularization methods demonstrate that PROACT substantially reduces poisoning-induced forgetting, even in the most challenging attack scenarios.
URL: https://openreview.net/forum?id=KyzMlMtQX6
---